Repository navigation
- @streamsy/core@0.12.0
- @streamsy/serve@0.12.0
- @streamsy/projection@0.12.0
- @streamsy/storage@0.12.0
Follow streams over hibernatable WebSockets on Cloudflare (ccfba50)
A Durable Object made with StreamsyObject.make can now serve live=websocket, a Streamsy extension to Durable Streams HTTP. Set live: { websocket: true }, or live: { websocket: { prefix: "/streams" } } when the routes have a prefix. The object accepts the socket with the WebSocket Hibernation API, keeps each socket's offset in its attachment, and pushes the new messages in the event that committed them. An object whose readers all hold sockets can hibernate between commits.
Each data frame holds the exact body of a catch-up GET, and a control frame with the SSE control fields follows it. The option is off by default, and an object without it behaves as before.
Http.read and Http.app now answer live=websocket with 426 Upgrade Required on every host that does not take the upgrade, so a client falls back to long-poll. @streamsy/core/http exports the frame helpers and close codes. See Follow streams over WebSockets.
Observe, retry and trigger attached projections (240b8a1)
Projection.Registry is a new in-process view of the runners that Projection.attach and ProjectionWakeup.attach start. Use it to check runner health, inspect failed bindings, request runs and wait until a projection has read your write. Trigger streams request projection runs from a wake stream; accepted streams publish how far a projection has read each input.
Provide Projection.Registry.layer below your attach layers so every attachment sees the same registry. On a Durable Object, put it in the object's layer next to CheckpointStore.layer. Each runner registers under its projection id and unregisters when its attach scope closes. Two attachments with the same id in one registry fail the layer build. See Observe attached runners.
The registry has these members:
statusreports each runner'sstate, consecutive passfailures,retryAt,active,queuedandlastPassAt. Both modes include amarkssummary with disjointfresh,sleeping,retryDue,backingOffandstoppedcounts, plusnextDueAt. The runner'sfailure,stoppedcount andtriggersexpose pass, binding and trigger-stream stops.statusfails withStorageFaultif it cannot read the mark counts.changesemits the status list on subscription and after runner changes. A slow subscriber keeps at most one pending notification. The stream logs and skips failed snapshots and stays open.schedule(binding)makes that binding due for the next pass, overriding its backoff or stop. It returns before processing finishes and fails withInvalidBindingwhen encoded params do not round-trip through the attached declaration.retry(target)makes failed and stopped bindings due now and returns the number of changed marks. A repeat before execution returns zero. Pass an id or declaration to select its failed bindings and resume stopped trigger streams, or a binding to select one mark. Resuming a trigger stream or failed pass adds nothing to the count.failed(target, { limit })lists failed bindings, stopped first, then by deadline. Entries include encodedparams,attempts,dueAt(Nonewhile stopped),stoppedand the stored failure's time, optional reason and message. The limit defaults to 100 and accepts integers from 1 to 1,000. Durable failures remain readable after eviction.eventsemits future binding observations.Rancarries status, unit count and input offsets.Failedcarries attempts, failure, stopped state and a retry deadline in either mode (Nonefor a stop).Droppedreports a source removed before the binding's first run. Events have no replay and buffer without a limit, so close subscriptions when finished.awaitOffset(binding, input, offset)subscribes before reading the checkpoint, then waits for the requested input offset. A matching failed run prompts a checkpoint reread for units already committed. A different attached version or generation fails withProjectionFaultreasonidentity-mismatch. AddEffect.timeoutto bound the wait.
Requests resolve by projection id and fail with NotAttached when it is missing. schedule and retry end the runner's current wait and permit one pass, including after a finite pass policy is exhausted. They preserve that policy: another failed pass after exhaustion returns straight to failed, with no automatic delay. On an object, both methods reconcile the host alarm before returning. Projection.Registry.isFor(binding) matches events by id and encoded params; RunnerEvent exports constructors, $is and $match.
Pass triggers: [FolderWake] to request source bindings from a fixed-address wake stream; the process host also supports remote triggers, while the alarm host accepts only its own storage owner's triggers. See Trigger streams for ownership, cursor recovery and validation rules.
Set accepted to a wake class on a declaration to publish the offset of each input that moved in a unit. State tables use StateTable.define(State, { accepted: TotalsAccepted }). Accepted offsets require Commit.transaction or Commit.producer, including on folds. Transactional rows commit with the checkpoint; producer rows can be visible before checkpoint settlement and deduplicate on retry. Streamsy.awaitWake(Wake, params, { stream, offset }) waits over local or remote storage. It reads from the start unless a fourth argument supplies { from: savedWakeCursor }; only rows after that cursor can satisfy the wait. See Accepted stream.
In @streamsy/serve, Wakeup.bind(host) receives reconcile, retry and retry state effects that work outside requests. An optional retryable(cause) returning false stops automatic run retries without stepping the schedule. A non-retryable StorageFault from next does the same. changed runs after a run or failed deadline read updates retry state. ProjectionWakeup supplies these hooks for the registry. A cold start makes stopped durable bindings due once; a still-broken binding stops again. Stopped marks alone arm no alarm. See Projections on alarms.
Changed behaviour (240b8a1)
Breaking change: Attached runners call the passRetry schedule for timing only, once per retryable failed pass with no failure input. A source write or trigger row records work without shortening backoff or ending a stop. In both durability modes, a retryable failed binding waits alone, by default from 1 s doubling to 1 hour; a finite pass schedule no longer bounds non-durable binding retries. See binding retry for the declaration's policy. The runner stops the binding, trigger stream or pass that hits a known non-retryable fault and reports the cause. registry.retry or the next runner start resumes it; registry.schedule also resumes a selected binding. Handler errors and defects retry on attached runners. Projection.follow now retries token-conflict and output-busy on its schedule, while other handler errors and defects still fail its stream. Structurally typed wake outputs must supply paramCodecs, as Streamsy.wake classes already do; structural Attachable values must include version and generation.
Breaking change: Opening SQL storage upgrades the schema to version 3, including each existing Durable Object on its next acquisition. Projection marks gain stopped, failed_at, failure_reason, failure_message and nullable wake_at; existing sequence numbers, attempts and due times remain intact. There is no downgrade from schema version 3: a release that supports only version 2 refuses the database, so returning to it requires restoring a pre-upgrade backup. Custom durable-mark adapters must implement the updated internal ProjectionMarksApi: summary, stopped-aware reads and retry, fail in place of defer, plus resume, bounded failed and sleep; the marks-level deferred method is removed. clear resets failure state when a newer sequence keeps the mark. The storage contract checks these rules. See the marks contract and SQL migration rules.
Request a process wake with its checkpoint (f1d2c08)
Return Projection.wakeAt(time) from a Projection.process handler to record its
next wake, using epoch milliseconds or an Effect DateTime. Returning nothing
clears it. Use Commit.atLeastOnce or Commit.transaction, including compensation
wrappers, so the request commits with the checkpoint.
A due unit carries the recorded time as unit.due, and its key includes that time
for idempotent retries. Without new input it has empty slices and ranges.
Progress.wakeAt reports the next recorded deadline. run, pass and follow
handle due timers when they run; each cycle makes at most one due unit without
input. Attached process and Durable Object hosts wake at the deadline. A process
wait rechecks wall time every 30 seconds while a deadline exists, including after
host suspend. Backoff delays a wake; stops keep it until retry or restart. With
durable: false, restart or eviction loses the wake until the next input. Return
a time after unit.due to avoid repeatedly running an immediately due timer.
RunnerStatus.marks.sleeping counts sleeping bindings separately from pending and
failed work. RunnerEvent.Ran.wakeAt reports the next recorded wake. Schema
version 3 adds nullable wake_at to projection marks on every SQL host.
See process timers.
Run external deliveries as projections (461a778)
Breaking change: @streamsy/outbox is no longer released with Streamsy and
gets no further versions. OutboxWakeup is removed from
@streamsy/serve/cloudflare and @streamsy/serve/alchemy.
Use a projection with Commit.atLeastOnce and unit.key as the idempotency key,
the declaration's retry policy for a retry budget and onExhausted, and
Projection.wakeAt for a deferred retry or a snooze. On Cloudflare, attach it with
ProjectionWakeup.attach. See Effects.
Set a retry policy for each projection declaration (c0f8f70)
All five projection constructors accept retry with a timing schedule, a failed-run
budget maxAttempts, and an onExhausted(unit, failure) hook. Both attach hosts stop
exhausted bindings and list them in registry.failed with exhausted: true.
registry.retry permits another run. New input during backoff preserves the budget
and stored deadline; success clears the failure count. The default remains unlimited
retry with delays doubling from 1 s to 1 hour. Use schedules based on step count;
schedules depending on input or elapsed time are unsupported.
Hooks run at least once before stopping; use unit.key to dedupe. When the exhausting
failure occurs before a unit forms, the hook is skipped and logged. Hook failures are
logged and the binding stops anyway. Non-retryable faults stop without the hook.
ProcessContext.attempt starts at 1 for each binding and increases after failures;
direct Projection.run and Projection.follow always supply 1.
Breaking change: AttachOptions.retry is now passRetry. Rename the option on
Projection.attach and ProjectionWakeup.attach. Its meaning is unchanged: it paces a
failed drain pass. retry now names the binding retry policy on a declaration, and an
attach call that still passes retry fails with a RangeError.
See binding retry.
Run projection bindings in parallel (b5061fc)
Pass concurrency to Projection.attach or ProjectionWakeup.attach to run several
bindings at once. The default is 1; each binding still runs its units in order,
and a failed binding backs off alone. The option must be a positive safe integer.
SQLite serializes Commit.transaction handlers; Postgres and MySQL run them up to
the pool size. Commit.atLeastOnce handlers run in parallel on every host. Keep
concurrency below the pool size, use the provided SqlClient in the handler's fiber,
and append to shared outputs last. See binding concurrency.
Read the offset of each local projection entry (d14dd51)
Slice.offsets now gives the storage offset of every item on local inputs, and
Entry.offset gives the corresponding offset in entry handlers. Reading from
entry.offset resumes after that entry. Pinned retries retain the same offsets.
Non-empty HTTP inputs have offsets undefined and entries omit offset, because
the protocol supplies page boundaries only. Every empty slice has offsets: [],
including skipped inputs and timer-only units.
See slices and entries.
Schedule a projection family and rescan its trigger history (3421973)
registry.schedule now accepts an array of bindings of one projection, validates
them all before writing marks, coalesces duplicates and wakes the runner once.
Empty arrays do nothing; mixed projection ids fail with RangeError. A storage
failure in a later chunk can leave earlier marks written; repeating the request is safe.
After a fold fix, call registry.rescan(projectionOrId, WakeClassOrStreamId) to reset
one trigger cursor to the start, clear its stop and wake the runner. The reset is
stored in both durability modes. A first attach already reads from the start, so a
new family needs no rescan. Both requests work on the process and alarm hosts.
See runner requests.
Follow wake streams on another owner (e3f9109)
Projection.attach can now use a wake stream on another owner in triggers.
Provide its stream class through the remote or other owner's protocol layer.
A scoped live-read follower wakes the runner on new rows. Each remote page's
cursor commits locally with its durable pending marks, so restart resumes from
that page boundary.
Remote read failures use passRetry after local bindings run; local commits
during pass backoff wait until it ends (up to 30 seconds by default). Invalid
trigger data stops only that stream. ProjectionWakeup.attach rejects remote
triggers with a RangeError naming the stream; it accepts only triggers on its
own storage owner. See Trigger streams.