[measurement] Storm repro with sequential replays

.changeset/event-count-guard.md

@@ -0,0 +1,9 @@
+---
+'workflow': minor
+'@workflow/core': minor
+'@workflow/world-vercel': minor
+'@workflow/world': minor
+'@workflow/errors': minor
+---
+
+Strengthen the event-creation precondition guard: replay-context writes now also send the number of loaded events, so a snapshot that is missing an event is rejected instead of committing a divergent event log, and a rejection restarts the replay in-process (consuming the events a world may attach to the rejection) rather than re-committing the rejected payload or re-invoking over the queue. `@workflow/world-local` and `@workflow/world-postgres` do not implement the check.

.changeset/step-delivery-ordering.md

@@ -0,0 +1,5 @@
+---
+'@workflow/core': patch
+---
+
+Deliver step results, wait completions, hook payloads and aborts in strict event-log order relative to one another, preventing replay divergence (`CORRUPTED_EVENT_LOG`) when a step completion is adjacent in the log to a `wait_completed`, `hook_received` or abort that a concurrent branch is awaiting.

docs/content/docs/v5/api-reference/workflow-errors/precondition-failed-error.mdx

@@ -8,14 +8,16 @@ related:
   - /docs/api-reference/workflow-errors/entity-conflict-error
 ---
 
-`PreconditionFailedError` is thrown by world implementations when an event creation is rejected because the client's event-log snapshot is stale — a newer out-of-band event (such as a received hook or a completed step) was recorded after the snapshot the client replayed from. It corresponds to HTTP 412 Precondition Failed semantics.
+`PreconditionFailedError` is thrown by world implementations when an event creation is rejected because the client's event-log snapshot is stale — either a newer out-of-band event (such as a received hook or a completed step) was recorded after the snapshot the client replayed from, or the snapshot is missing an event recorded at or before it. It corresponds to HTTP 412 Precondition Failed semantics.
 
-This only occurs when the optimistic-concurrency guard is enabled via `WORKFLOW_PRECONDITION_GUARD=1` (see [Runtime Tuning](/docs/configuration/runtime-tuning)); event creations that carry no snapshot are never rejected with this error.
+This only occurs while the optimistic-concurrency guard is enabled (`WORKFLOW_PRECONDITION_GUARD`, on by default — see [Runtime Tuning](/docs/configuration/runtime-tuning)); event creations that carry no snapshot are never rejected with this error.
 
 <Callout>
-The Workflow runtime handles this error automatically: it reloads the event log and retries, ultimately re-enqueueing the run for a fresh replay if it cannot catch up. You will only encounter it when interacting with world storage APIs directly.
+The Workflow runtime handles this error automatically: it restarts the replay in the same invocation from a corrected event log, and re-invokes the run for a fresh replay only once its in-process restart budget is spent. It never retries the rejected creation as-is, because a replay working from a corrected log derives different events. You will only encounter it when interacting with world storage APIs directly.
 </Callout>
 
+A world may attach the events the client was missing to the rejection as `details`, which lets the runtime restart without re-reading the event log. Doing so is optional, and the runtime falls back to a full reload when the details are absent or unusable.
+
 ```typescript lineNumbers
 import { PreconditionFailedError } from "workflow/errors"
 declare const world: { events: { create(...args: any[]): Promise<any> } }; // @setup
@@ -40,6 +42,8 @@ definition={`
 interface PreconditionFailedError {
   /** Delay in seconds before the operation should be retried. Present when the server sends a Retry-After header. */
   retryAfter?: number;
+  /** Optional rejection payload. A world may put the events the client was missing here, as \`{ events, cursor }\`, so the runtime can restart its replay without re-reading the event log. */
+  details?: unknown;
   /** The error message. */
   message: string;
 }

docs/content/docs/v5/configuration/runtime-tuning.mdx

@@ -41,13 +41,19 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
 ### `WORKFLOW_PRECONDITION_GUARD`
 
 - Default: enabled
-- An optimistic-concurrency guard for event creation: replay-context event creations send a `stateUpdatedAt` snapshot timestamp, and a backend that supports the guard rejects a creation with 412 ([`PreconditionFailedError`](/docs/api-reference/workflow-errors/precondition-failed-error)) when a newer out-of-band event (a received hook or a completed step) was recorded after that snapshot.
-- On rejection the runtime reloads the event log and retries, falling back to a queue re-invocation with a fresh replay if it cannot catch up.
+- An optimistic-concurrency guard for event creation: replay-context event creations describe the snapshot they replayed from — its latest event timestamp (`stateUpdatedAt`), the number of events it contains (`stateEventCount`), and its event-log cursor (`stateCursor`) — and a backend that supports the guard rejects a creation with 412 ([`PreconditionFailedError`](/docs/api-reference/workflow-errors/precondition-failed-error)) when a newer out-of-band event (a received hook or a completed step) was recorded after that snapshot, or when the snapshot is missing an event recorded at or before it.
+- On rejection the runtime restarts the replay in the same invocation from a corrected event log, and falls back to a re-invocation with a fresh replay once the restart budget is spent. The rejected write is never retried as-is: a replay working from a corrected log derives different events, so only a fresh replay may write again.
 - When enabled — and the World declares that it enforces the guard (`capabilities.preconditionGuard`; the Vercel World does) — the runtime also keeps the per-step event-log delta optimization (consuming the delta returned by a step's terminal write instead of issuing an extra `events.list` per step) active while the run has an open hook. Without an enforced guard, an open hook disables it.
 - While a hook is open on a guard-enforcing deployment, inline steps take the await-then-run path even when optimistic inline start is enabled: the step's `step_started` claim carries the snapshot and is awaited before the body runs, so a claim the backend rejects as stale never executes user code.
 - Backends that do not support the guard ignore the snapshot; they must not declare the capability, so guard-dependent optimizations stay off against them even when the flag is set.
 - Set `0` to disable.
 
+### `WORKFLOW_PRECONDITION_MAX_INPROCESS_RESTARTS`
+
+- Default: `3`
+- Replays a single invocation restarts in-process after a rejected event creation before it falls back to a re-invocation.
+- A restart reloads the event log and rebuilds the workflow from scratch, so it costs a replay but no queue round trip. A World may attach the missing events to its rejection, in which case the first restart needs no event-log request at all.
+
 ## Inline execution
 
 ### `WORKFLOW_V2_TIMEOUT_MS`

docs/content/worlds/v5/building-a-world.mdx

@@ -106,6 +106,28 @@ Keep the owning Run available for at least as long as its token remains unavaila
 
 **Automatic Hook Cleanup:** When a run ends, remove its live Hooks. Make each token available unless its `tokenRetentionUntil` is still in the future. A `hook_disposed` event always makes the token available immediately.
 
+### Optional: The Event Creation Precondition Guard
+
+A replay writes events derived from the event log it loaded, so a write made from an event log that no longer matches the store can commit events no correct replay would produce. To let a World fence those writes, `events.create()` params may carry a description of the snapshot the caller replayed from:
+
+- `stateUpdatedAt` — the timestamp encoded in the latest loaded event's ID.
+- `stateEventCount` — how many events the caller loaded. Only sent together with `stateUpdatedAt`; ignore a count that arrives without one.
+- `stateCursor` — the caller's event-log cursor. Advisory, and only meaningful on the reject path (see below).
+
+Enforcing the guard means rejecting the creation with a `PreconditionFailedError` when either holds:
+
+1. An event was recorded after `stateUpdatedAt` that the caller could not have loaded.
+2. More events were recorded at or before `stateUpdatedAt` than `stateEventCount` — the caller's snapshot is missing one.
+
+Two rules make this safe:
+
+- **Compare at or below `stateUpdatedAt`, not strictly below.** Events routinely share a millisecond with the caller's latest event, and a strict comparison misses exactly those.
+- **Every uncertainty must allow the write.** If your record of a run's events is incomplete, expired, or cannot answer the question, accept the creation. A rejection must always mean a real discrepancy, because the runtime responds to one by discarding a replay.
+
+A rejection may optionally carry the events the caller was missing, as `{ events, cursor }` on the error's `details`. Only include them when you can prove the set is complete — that those events fully account for the discrepancy and are not truncated. Otherwise omit them, and the runtime performs a full reload instead.
+
+A World that enforces the guard should declare `capabilities.preconditionGuard`, which the runtime also reads to keep event-log delta optimizations enabled. A World that ignores these params must not declare it.
+
 ## Queue Interface
 
 The Queue interface handles asynchronous workflow execution, including queued step work: