taquba-bulk-v0.1.0
·
65 commits
to master
since this release
Initial release. Per-batch orchestrator that runs one pipeline over many
inputs in a single process on top of taquba-workflow.
Added
Pipeline: the per-item contract (typedInput/Output, anError
that converts into aStepError, and an asyncrun). Each input item
becomes onetaquba-workflowrun whose single step invokesrun; the
pipeline's own logical steps live insiderunasBulkCtx::memoized
calls.BulkCtx<T>: per-item execution context. Carries the typedinput,
run_id, and submitterheaders; exposesmemoized(durable per-step
result caching so an at-least-once retry replays cached results instead of
repeating a paid call),record_cost, andcancel_token.CostReport: generic named-metric accumulator (token counts, paid-API
units, compute-seconds, dollars). Interior-mutable while a step runs and
serializable for the per-item envelope and the batch rollup.Bulk/BulkBuilder: the runner. Submits N runs, drives the worker pool,
streams output as items complete, and aggregates progress and cost.
Builder options:output,key_fn,headers,max_concurrent,
poll_interval,queue_name,memo_prefix,fail_threshold.run
executes to completion;run_with_shutdowndrains in-flight items on a
shutdown signal (e.g. spot preemption).ProgressSnapshot: point-in-time counts, rate, estimated time remaining,
and cost rollup, returned byBulk::progress.BulkReport: final counts, elapsed time, cost rollup, and
failed_run_ids(re-submitting those ids resumes from cached memo state).OutputSinkwithJsonlSink(one JSON record per line) andNullSink
(discards records, for side-effecting pipelines);read_jsonlfor
line-delimited JSON input.Error/Result: crate error type, including
Error::FailureThresholdExceededwhen the share of failed items crosses
the configured threshold.- Re-exports
StepErrorandStepErrorKindfromtaquba-workflowfor the
Pipeline::Errortype.