Add opt-in Durable Object restart rollouts

[cnluzhang]

Summary

• Add opt-in durableObjectRollout: "restart" while keeping preserve as the default and allowing a later projection to supersede an unobserved restart.
• Commit active routes, rollout projections, and permanent monotonic restart sequences atomically in Control. Redis remains authoritative and Pub/Sub is only a fast lifecycle hint.
• Fence Gateway WebSocket admission and reconnection, close superseded or deleted sessions with 1012 in their owning IoContext, retry transient Redis failures, and bound lifecycle Redis commands.
• Lazily abort stale do-runtime facets without deleting SQLite storage, retarget retained Workflow alarms under restart mode, and clean active projections during whole-worker deletion without reusing sequence values.
• Add shared JS/Rust contract fixtures and synchronized protocol, Redis, runtime, Gateway, Durable Object, Workflow, and deployment documentation.

Deployment

Deploy readers before the Control writer and pause Control mutations while mixed writer versions are present, as documented in the Durable Objects rollout section.

Validation

• npm test (2,294 unit tests)
• npm run lint:unused
• cargo test --locked -p wdl-rust-common -p workflows (241 tests)
• cargo fmt --all --check
• cargo clippy --locked -p wdl-rust-common -p workflows --all-targets -- -D warnings
• npm run test:integration (55 integration files across 4 isolated shards)
• git diff --check

[Copilot]

Pull request overview

This PR introduces an opt-in Durable Object rollout policy (durableObjectRollout: "restart") that coordinates Control, Gateway, do-runtime, and Workflows so restart-driven lifecycle transitions are fenceable, monotonic, and observable without relying on durable Pub/Sub delivery.

Changes:

• Add shared DO rollout projection/event contracts (JS + Rust), new Redis keys/channels, and style-contract guards for key literals.
• Fence Gateway-proxied WebSocket admission/reconnect on an atomic route+rollout snapshot, and close superseded/deleted sessions with 1012 (with bounded Redis command timeouts and transient retry handling).
• Thread rollout mode/sequence through do-runtime owner resolution + host actor dispatch (without entering invoke/connect wire payloads) and retarget Workflows DO alarms when restart is active.

Reviewed changes

Copilot reviewed 64 out of 64 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/unit/worker-contract.test.js	Adds unit coverage for DO rollout projection/event codecs and lifecycle channels.
tests/unit/style-contracts.test.js	Extends style-contract scanning to forbid inline DO rollout key literals.
tests/unit/redis-session.test.js	Adds tests for RedisClient per-command timeout behavior and session() rejection under timeouts.
tests/unit/redis-resp.test.js	Verifies Redis -ERR replies throw RedisReplyError with extracted error code.
tests/unit/gateway-websocket.test.js	Expands unit tests for lifecycle-fenced WebSocket proxy behavior and outcomes.
tests/unit/gateway-index.test.js	Adds gateway-index unit coverage for lifecycle snapshot reads/registration and timing.
tests/unit/do-runtime-protocol.test.js	Ensures rollout fields are not accepted from invoke wire payloads (normalized defaults).
tests/unit/do-runtime-index.test.js	Updates do-runtime index tests for owner resolution options and restart sequence stripping.
tests/unit/do-runtime-actor.test.js	Adds host-actor tests for preserve vs restart facet behavior and stale sequence rejection.
tests/unit/do-owner-registry.test.js	Adds tests for reading/applying rollout projection during owner resolution and dispatch fences.
tests/unit/do-owner-client.test.js	Ensures restart sequence is not forwarded via body/headers to do-runtime.
tests/unit/control-secret-envelope-handlers.test.js	Updates secret handler tests to include new DO rollout keys in watched snapshots.
tests/unit/control-routing.test.js	Adds routing tests for DO rollout meta validation, projection persistence, and restart allocation/publish.
tests/unit/control-deploy-watch.test.js	Adds deploy handler tests for validating/persisting durableObjectRollout policy.
tests/unit/control-delete-handler.test.js	Adds delete tests ensuring residual rollout projection cleanup and worker:delete publication.
tests/integration/gateway-websocket.test.js	Adds integration coverage for draining old WebSockets and avoiding reconnect to inactive versions.
tests/integration/durable-objects-websocket.test.js	Adds integration tests for DO WebSocket closure on restart rollout and whole-worker delete.
tests/integration/durable-objects-storage.test.js	Adds integration test verifying restart rollout swaps class while preserving SQLite storage.
tests/integration/durable-objects-alarms.test.js	Adds integration test for alarm retargeting under restart rollout.
tests/helpers/mocks/fake-redis.js	Re-exports RedisReplyError in the shared Redis stub for tests.
tests/fixtures/do-rollout-projections.json	Introduces cross-language fixture cases for rollout projection validity.
shared/worker-contract.js	Adds DO rollout keys, modes, projection/event codecs, and worker delete event codecs.
shared/redis.js	Re-exports RedisReplyError and RedisCommandTimeoutError via shared-redis entrypoint.
shared/redis-session.js	Aligns session option typedefs to exclude per-command deadlines.
shared/redis-resp.js	Adds RedisConnectionOptions typedef and RedisReplyError.code extraction.
shared/redis-command-client.js	Adds commandTimeoutMs support for socket-per-call RedisClient operations and a timeout error type.
rust/workflows/src/api/do_alarms/scripts.rs	Adds Lua script to read route/storage/retained/rollout state in one snapshot.
rust/workflows/src/api/do_alarms/dispatch.rs	Uses rollout projection to retarget alarms under restart and fail closed on torn state.
rust/common/src/worker_contract.rs	Adds Rust-side rollout key helper + projection parser and validates against shared fixture.
runtime/config-system.capnp	Removes DO_BACKEND from Control worker bindings in the system runtime config.
gateway/websocket.js	Adds lifecycle gating/registration hooks and closes pending/upstream sockets deterministically on lifecycle outcomes.
gateway/runtime.js	Implements process-local WebSocket lifecycle registry, atomic snapshot reads, reconciliation, and transient retry behavior.
gateway/index.js	Wires lifecycle snapshot admission + proxy hooks into routed WebSocket handling and timing.
gateway/dispatch.js	Splits dispatch typing between Control vs runtime forwarding shapes.
docs/workerd-js-standards.zh.md	Documents commandTimeoutMs and strict cross-IoContext signaling boundary (ZH).
docs/workerd-js-standards.md	Documents commandTimeoutMs and strict cross-IoContext signaling boundary (EN).
docs/source-map.zh.md	Updates module ownership map entries for gateway websocket lifecycle + control routing rollout allocation (ZH).
docs/source-map.md	Updates module ownership map entries for gateway websocket lifecycle + control routing rollout allocation (EN).
docs/redis-key-layout.zh.md	Documents new rollout keys/channels and semantics (ZH).
docs/redis-key-layout.md	Documents new rollout keys/channels and semantics (EN).
docs/protocol-contracts.zh.md	Adds rollout projection/lifecycle notification to protocol contract registry (ZH).
docs/protocol-contracts.md	Adds rollout projection/lifecycle notification to protocol contract registry (EN).
docs/modules/workflows.zh.md	Documents alarm retarget behavior under restart rollout and rollout sequencing guidance (ZH).
docs/modules/workflows.md	Documents alarm retarget behavior under restart rollout and rollout sequencing guidance (EN).
docs/modules/runtime.zh.md	Clarifies active-version eviction/drain behavior and how lifecycle snapshots fit (ZH).
docs/modules/runtime.md	Clarifies active-version eviction/drain behavior and how lifecycle snapshots fit (EN).
docs/modules/gateway.zh.md	Documents new lifecycle snapshot + reconciliation behavior and failure semantics (ZH).
docs/modules/gateway.md	Documents new lifecycle snapshot + reconciliation behavior and failure semantics (EN).
docs/modules/durable-objects.zh.md	Documents rollout keys, restart semantics, and lazy facet restart behavior (ZH).
docs/modules/durable-objects.md	Documents rollout keys, restart semantics, and lazy facet restart behavior (EN).
docs/modules/control-auth.zh.md	Documents new API field semantics and rollout deployment ordering constraints (ZH).
docs/modules/control-auth.md	Documents new API field semantics and rollout deployment ordering constraints (EN).
do-runtime/protocol.js	Adds rollout fence fields to invoke model (defaults) and strips them from wire envelopes.
do-runtime/owner-registry.js	Reads rollout projection in owner snapshots, rejects stale versions under restart, and refreshes on dispatch fences.
do-runtime/index.js	Threads allowSupersededVersion option for storage cleanup while keeping rollout fields owner-local.
do-runtime/actor.js	Tracks per-facet restartSequence and lazily aborts stale facets under restart without deleting SQLite.
control/shared.js	Exports WORKER_DELETE_CHANNEL from control-shared.
control/handlers/promote.js	Logs/returns rollout mode and restart sequence from promote results.
control/handlers/deploy.js	Validates/persists durableObjectRollout policy in bundle meta (restart requires a DO binding).
control/handlers/delete.js	Publishes worker:delete and deletes residual rollout projection during whole-worker delete/residual cleanup.
control/handlers/delete-plan.js	Deletes rollout projection during whole-worker delete plan and publishes worker:delete.
CHANGELOG.md	Adds unreleased entries describing restart rollout behavior, lifecycle fencing, and deployment constraints.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 64 out of 64 changed files in this pull request and generated no new comments.

Suppressed comments (1)

gateway/index.js:209

• readWebSocketLifecycleSnapshot() can throw transient Redis transport/reply errors (it intentionally rethrows transient RedisReplyErrors). On initial WebSocket admission those errors currently bubble out of this branch and get mapped to the generic 502 gateway_error, which is inconsistent with the rest of Gateway’s routing failure semantics (and prevents clients from getting the expected 503 gateway_routing_unavailable retry signal).

Consider catching errors from the initial lifecycle snapshot read and rethrowing GatewayRoutingUnavailableError so admission fails closed with 503 instead of 502 when Redis is temporarily unavailable.

[Copilot]

Pull request overview

Copilot reviewed 67 out of 67 changed files in this pull request and generated no new comments.

Suppressed comments (1)

shared/redis-command-client.js:78

• Promise.race() with the timeout can leave the operation promise running after the timeout rejects. If operation later rejects, it can surface as an unhandled rejection because nothing awaits/handles it once the race has already been decided. Add a no-op rejection handler to operation when commandTimeoutMs is enabled so late failures are always observed/handled.