server-zd v0.1.23

Breaking Changes

• Each run now chooses at start whether it offloads its history to object storage, with the new FlowConfig.offload_history (field 9, optional bool). Unset means true, in serve, dev, and local.
  • An offloaded run seals its history into segments, uploads them under PREFIX/history/, deletes uploaded history locally after the grace period, and always deletes the objects of segments the window passed (about one hour after expiry).
  • A run with offload_history: false keeps its history on local disk and never writes a history object. The window still prunes old events and deletes the superseded values it passed.
  • Set it at StartFlow (flow_config_override) or in a SubFlow override; a child inherits its parent's choice unless its override sets one. UpdateFlowConfig rejects a request that sets it with INVALID_ARGUMENT.
  • Trade-off: keep offload for most and all long-running runs; turn it off for many short runs, which otherwise add an object write at close and a delete at retention per run.
  • A run without offload keeps every superseded value its window covers on local disk, with no byte cap (the removed history.maxRetainedSupersededBytes does not apply to it). For a run that rewrites large values, keep offload or set a smaller history_window_events.
• Runs that started before the upgrade keep the offload mode their partition recorded at the upgrade, for their whole life:
  • Offload off at the upgrade (serve unless history.offloadEnabled was on): their new history stays on local disk; nothing migrates. A partition that turned offload on and then off again is off too: no new sealing, migration stops; already sealed segments may still finish uploading, and uploaded history stays readable.
  • Offload on at the upgrade (0.1.22 dev/local by default, or serve with the switch on): offloaded runs stay offloaded and keep uploading. A run that recorded history before offload was turned on migrates lazily: each history-recording command seals one more segment, which uploads in the background; an idle run keeps its older history locally until it gets commands; closed runs never migrate and keep local history until retention.
  • A dev/local data directory upgraded from 0.1.21 directly to 0.1.23 never had offload, so its runs stay local.
  • They keep their recorded superseded byte cap and follow their delete switch.
• history.offloadEnabled is removed. true is accepted with a startup warning and no effect; false fails startup and asks you to remove the key and set offload_history: false on the Flows that should keep history locally.
• history.deleteExpiredObjects and history.maxRetainedSupersededBytes are removed: accepted, no effect, startup warning.
• FlowConfig.delete_expired_history_objects (field 8) is deprecated: runs that start with 0.1.23 or later ignore it (StartFlow drops it). Runs that started earlier still follow it, and UpdateFlowConfig can still change it for them, for example to delete the expired objects that a 0.1.22 run with the switch off keeps until its retention ends. SDK 1.40.0 marks it deprecated and stops sending it, so use an earlier SDK or a direct gRPC request for that.
• Default history.windowEvents is mode-specific: 50,000 for serve, 5,000 for dev and local (was 10,000). Runs record their window at start, so existing runs keep theirs. SubFlow children now record the server's history.windowEvents and history.pruneChunkSize instead of the built-in 10,000 and 1,000. history.maxWindowEvents stays 100,000.
• limits.maxFlowStateBytes: runs started since 0.1.23 no longer count the bookkeeping rows of superseded values (they wait for the window or an upload, like history). Before, an offloaded run that rewrote values under a small budget could stall before its first segment sealed. Older runs keep their accounting.
• LoadBlobs: the 1,000-entry limit now counts entries whose run offloads history (before: any entry once a named partition offloaded). Entries of runs without offload stay unlimited.
• Metric dexzd_history_offload_mode_skipped_partitions_total is removed: partitions no longer record an offload mode at startup.
• Upgrade:
  • Stop the standby and the active process, upgrade both binaries, then start the active process first and the standby after it. Apply results change, so mixed versions diverge.
  • New runs offload by default after the upgrade. Do not run an earlier release on the storage prefix or data directory afterwards: history that new runs removed from local disk needs object storage reads. Roll forward, or restore a pre-upgrade backup into a new prefix.
  • The node needs read, write, and delete access to PREFIX/history/ (covered by the prefix-confined IAM policy), and no bucket lifecycle rule may expire objects under PREFIX/history/.
  • serve now creates the history read cache directory <localStorage.directory>-history-read-cache (1 GiB by default); point history.readCacheDirectory at a writable volume if the store volume is mounted at localStorage.directory.
  • A purge of an offloaded run waits one minute for in-flight history uploads before deleting its objects.
• limits.maxFlowStateBytes no longer counts Steps that ended normally: their StepExecution rows are now deleted when they end (#1429), so a turn that ends its Step may be admitted while it adds less than the freed row.
• history.uploadConcurrency is now the cap of an adaptive upload limit (default 256, was a fixed 16). The limit starts at the new history.uploadMinConcurrency (16) and grows while uploads wait. To keep a fixed value N, set both to N. A new history.uploadMaxInFlightBytes (128 MiB) bounds the bytes being encoded and uploaded at once (#1426).

server-zd v0.1.23

Long-lived runs no longer slow down as they age

A run that executed many Steps got slower with every Step: at 50k Steps one command spent about 73 ms in apply, and ASYNC throughput fell from about 3,000 to 54 Steps/s. Its local state also grew without bound. Fixed by:

• Maintaining the active Step types for search incrementally instead of scanning every Step row on each command; nothing is maintained when active Step search is disabled (#1425).
• Starting history window pruning and value garbage pruning from persisted cursors instead of rescanning deleted rows (#1427).
• Deleting the StepExecution rows of Steps that ended normally (#1429), and sweeping the rows older runs kept, at most 256 per command (#1430). A 40k-Step run now keeps 4 Step rows instead of 40,002, and its checkpoints stay small.
• Indexing the speculative apply overlay by key (#1422).
• The debug time-travel reconstruction check ignores the deleted rows when replaying older logs (#1431).

With these changes, eight concurrent runs of 40k Steps each stay flat at about 4,500–7,500 Steps/s in local tests.

History uploads

• History upload concurrency adapts to the backlog, halves on retryable storage failures such as S3 SlowDown, and respects an in-flight byte budget (#1426).
• dexzd_history_upload_backlog_segments now reports the live backlog; before, it was only refreshed at startup and every 60 seconds. New gauges: dexzd_history_upload_concurrency_limit, dexzd_history_upload_in_flight_bytes (#1426).

Per-run history offload

See Breaking Changes: FlowConfig.offload_history chooses per run whether history is offloaded; the server-level switches are removed (#1423).

Added

• SDKs 1.40.0 (Go, Python, Java, TypeScript, Rust) expose FlowConfig.offload_history (OffloadHistory, offload_history, offloadHistory). dexcli flow start --config accepts offloadHistory.