Skip to content

taquba-workflow-v0.5.0

Choose a tag to compare

@micllam micllam released this 28 May 12:57
· 71 commits to master since this release
c444f84

Added

  • Memo: per-step durable key-value store for memoizing within-step
    side effects, backed by object storage. Bound to a specific
    (run_id, step_number); get(key) / put(key, value) take only
    the user key. Strictly per-step; the durable channel between steps
    is StepOutcome::Continue's payload, not memo.
  • MemoStore: the backing store Memo views are derived from
    (Arc<dyn ObjectStore> + path prefix). Used internally by the
    runtime builder; users construct one directly mainly in tests.
  • Step::memo: every step receives a Memo scoped to its own
    (run_id, step_number). Runners use it to cache results of
    expensive within-step side effects (LLM calls, paid APIs) so
    at-least-once retries don't re-pay for work the prior attempt
    already did.
  • WorkflowRuntimeBuilder::memo_prefix: configures the object-store
    prefix Step::memo entries live under. Defaults to "workflow-memo";
    set a distinct prefix when multiple runtimes share one store.
  • Error::Store(taquba::object_store::Error): surfaced from memo
    read/write failures. Classified as transient by is_permanent.
  • WorkflowRuntimeBuilder::memo_retention(Duration): opts the runtime
    into writing a terminal marker via MemoStore::write_terminal_marker
    on every terminal state (Succeeded, Failed, Cancelled). Markers
    outlive the run record and provide the input a memo-retention sweep
    consumes to decide when a run's memo entries are eligible for
    deletion. Without this setter no marker is written and memo entries
    are retained indefinitely (appropriate for short-lived runs or
    external cleanup).
  • Memo-retention sweeper: when memo_retention is set,
    WorkflowRuntime::run spawns a background task that periodically
    scans terminal markers and, for each marker older than the
    configured window, deletes the run's memo entries and then the
    marker itself. The first sweep fires on startup so a fresh process
    catches markers left behind by an earlier one. The sweeper shuts
    down with the caller-supplied shutdown future.
  • WorkflowRuntime now reads every timestamp it writes
    (DurableRunRecord::submitted_at_ms, the ContinueAfter run_at,
    and the terminal-marker timestamp) through a taquba::Clock. By
    default the runtime shares the clock its Queue was opened with
    (via Queue::clock), so passing a MockClock to OpenOptions
    virtualises time for the queue and the workflow runtime together.
  • WorkflowRuntimeBuilder::clock(Arc<dyn Clock>) overrides the
    defaulted-from-queue clock when a test or specialised setup needs a
    separate time source.

Changed

  • Breaking: WorkflowRuntime::builder now takes an additional
    required object_store: Arc<dyn ObjectStore> argument between the
    queue and the runner. The store backs Step::memo and need not be
    the same store the queue was opened with, though sharing one (just
    cloning the Arc) is the common case. Existing call sites must add
    the store argument:

    // Before:
    let runtime = WorkflowRuntime::builder(queue, runner, hook).build();
    // After:
    let runtime = WorkflowRuntime::builder(queue, store, runner, hook).build();