3.9.1

Durable Workflow Server 3.9.1.

Git tag	3.9.1 → 7d7ad9d6c9aee2e4592d74389ba9efd36e741f44
Image index	sha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51 in durableworkflow/server and ghcr.io/durable-workflow/server
Release run	https://github.com/durable-workflow/server/actions/runs/43423285947

Durable Workflow Server 3.9.1 (native)

Server 3.9.1 corrects legacy signal/update request-key readback and ambiguous
identity handling, preserves typed oversized-completion refusals, and advertises
the actual completion wire budget. Upgrades from 3.8.x also gain durable command keys, run-close
settlement of open activities/timers and storage admission. Two inherited changes
affect behavior a client can observe; see
Behavior changes. Its exact images are
durableworkflow/server:3.9.1 and ghcr.io/durable-workflow/server:3.9.1;
the published Compose default selects 3.9.1. Rolling aliases move separately
after published-image qualification.

This patch corrects legacy request-key readback, ambiguous identity handling
and completion transport/discovery from the immutable 3.9.0 image. The 3.9.0 rolling aliases and final GitHub
Release were held for this correction. Use 3.9.1 for the corrected contract;
3.9.0 tags are unchanged. The changes inherited from 3.8.x are included below.

Upgrade

From 3.9.0, no additional schema step is needed. Stop the Server replicas,
back up as usual, select the 3.9.1 image, restart and verify readiness and the
reported version. The physical schema and dw_command_intents contents are
unchanged. The schema transition below applies when coming from 3.8.x or earlier.

Schema upgrade required on every backend. This release adds one
native-only table, dw_command_intents, which is created empty:

Backend	Schema
SQLite	12 → 13
PostgreSQL	13 → 14
MariaDB	13 → 14
MySQL	14 → 15

The step only creates the table; it rewrites no rows and takes seconds. A
server never upgrades its schema at startup. Started on the earlier schema,
3.9.1 refuses with native_schema_upgrade_required.

For the single-node Compose deployment:

1. Stop the Server and leave MySQL running:

   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml stop server

2. Back up MySQL and server_storage, and record the current image
   reference. This backup is the only way back to 3.8.x.

3. Change DW_SERVER_IMAGE to the 3.9.1 image, pull it and run
   schema-upgrade once:

   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml pull server
   docker compose --env-file durable-workflow.prod.env \
     -f docker-compose.published.yml run --rm --no-deps server schema-upgrade

   From 3.8.x on MySQL it reports
   "from_schema_version":14,"schema_version":15,"status":"upgraded". Run
   again, it reports "status":"current".

4. Start the Server with
   docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait,
   then confirm /api/ready, /api/cluster/info (version is 3.9.1) and
   worker registration.

With several Servers on one database, stop every one of them before step 3
and start them all on 3.9.1 afterwards. On MySQL and MariaDB, an interrupted
upgrade resumes when you run schema-upgrade again; SQLite and PostgreSQL
upgrade in one transaction.

Rolling back to 3.8.x requires restoring the backup. 3.8.0 refuses the
upgraded database with native_schema_newer_than_binary.

From earlier releases, the 3.9.1 image's schema-upgrade applies every
earlier step in the same run. From 3.7.x that includes the
3.8.0 step. On MySQL from 3.6.x or earlier (schema 12 or
below), it includes the 3.7.0 JSON rebuild, with its
downtime of about 2 minutes per million history rows.

On Kubernetes, scale the Server to zero, back up, run helm upgrade with
native.schemaUpgrade.enabled=true so the 3.9.1 image runs schema-upgrade
once, then scale back up. Chart 1.7.0 is tracked in
PR #1199 and will default
to Server 3.9.1 after image qualification. Its release-specific instructions
will appear in the chart's upgrade notes.

The worker protocol stays at 1.20 and the control plane at version 2.

Behavior changes

A signal or update request_id is a durable command key

A signal or update request_id now names one command on its workflow
instance (#934).
Cluster info advertises capabilities.command_request_keys: true.

• A retry with the same command intent returns the original receipt,
  including its command_id, run_id and, for an update, update_id, and
  writes nothing. The receipt is found before any run gate, so a retry after
  the run closed or continued as new still gets it. The intent is Core's
  CommandIntent v1 digest over the command type, name, typed arguments and
  explicit run target, so 1, 1.0 and "1" differ and map key order does
  not.
• The same key with a different name, arguments or run target is refused with
  409, reason: "request_id_conflict" and
  outcome: "rejected_request_id_conflict". The response names the differing
  part in conflict and the original command in retained_command_id.
  Nothing is written. Do not retry this 409 unchanged: it means the key
  was reused for different input. Send the original input to recover the
  receipt, or use a new key.
• A receipt retained from before intent recording is found by its exact raw
  request key as well as by the modern hashed identity. A compatible retry
  returns its original command/run/update identity with retained: true and
  intent_verified: false; it writes nothing and does not claim a retry's
  external-payload reference. Changed arguments cannot be proved equal for
  that legacy receipt. A changed operation name or explicit run target is
  refused with request_id_conflict, rather than suggesting that a new target
  was honored. Modern retained update receipts report intent_verified: true.
• A raw/token collision or a match that only exists because of a database's
  case-insensitive collation is refused with 409 request_id_ambiguous,
  retryable: false, retained: true and intent_verified: false. No receipt
  is selected and nothing is written. Inspect the original command before
  deciding whether another key is appropriate.
• New command keys are stored hashed and their intents as digests; the
  intent table stores no argument values. Existing raw-key rows are not
  rewritten. These records live as long as their command; retention removes
  them with the run's details.

This differs from PHP 2.x, which recorded a second signal for a repeated
signal request_id and returned the original update receipt for a retry with
different arguments. Clients that reuse one request_id for different input
get the 409 on native. sdk-php 3.17.0 surfaces it as a typed conflict
(SignalFailed::conflict() / UpdateFailed::conflict()), and
sdk-rust 4.15.0 reports the key it sent; none of the SDKs resend on 409. See
Behavior that differs from PHP 2.x.

A run's own close settles its open activities and timers

When a run's own code closes it with complete_workflow, fail_workflow or
continue_as_new, the close now cancels the activities and timers it still
has open, in the same transaction and before the terminal event, as Workflow
2.28.0 does (#1193,
Core workflow#1117).

• Each open activity, such as a select() loser the workflow never awaited,
  gets ActivityCancelled with cancellation_reason: "run_closed". Each
  pending timer, a wait's timeout timer included, gets TimerCancelled.
  Nothing is appended after the terminal event.
• A late heartbeat or lease renewal for such an activity answers
  reason: "run_closed" with can_continue: false. A late complete or
  fail gets 409 with outcome: "ignored", recorded: false,
  reason: "run_closed" and activity_status: "cancelled". Workers treat it
  as final.
• Replay ignores an ActivityCancelled whose cancellation_reason is
  run_closed; the workflow never observed it. sdk-php 3.17.0 does this, as do
  sdk-python 2.17.0 and sdk-rust 4.15.0.
• Native refuses the abandon cancellation policy, so it never keeps an
  activity running past its run's close. Children are unchanged: they follow
  their parent-close policy.

PHP 2.x left this work open, so the activity still ran and its history was
appended after the terminal event. A workflow that needs a loser's effect
must await it before returning.

Storage admission

Native now honors the operator storage-pressure observation that PHP Server
documents (#951).
It is off unless DW_STORAGE_ADMISSION_FILE, DW_STORAGE_ADMISSION_SOURCE and
DW_STORAGE_ADMISSION_MAX_AGE_SECONDS (1 to 300) are set.

• normal admits everything. draining refuses starts, signals, updates,
  queries, registrations, uploads, polls and other new work with 503,
  Retry-After: 5 and storage_pressure, while existing leases still
  complete, fail, heartbeat and renew. fenced, and any missing, malformed,
  stale, wrong-source or untrusted observation, keeps only reads.
• A long poll that is already waiting answers the poll refusal at its next
  probe. The maintenance loop starts no pass under pressure.
• /api/ready reports pressure as not ready; /api/health keeps its
  database meaning. Refusals never expose the file path or source.
• The file is opened without following symlinks and checked on the same
  descriptor. Native does not yet serve the draining completion-upload
  allowance, so an upload is refused like other new work; a worker keeps its
  lease and uploads after pressure clears.

See Storage Admission.

Other changes

• Worker command names longer than their database columns are refused with
  the typed 422 validation_failed naming commands.N.<field>, before any
  write (#607):
  activity_type, signal_name, condition_key and
  condition_definition_fingerprint at 191 characters; queue,
  connection and workflow_type at 255; a child's workflow_type and
  queue at 191. Before, MySQL and PostgreSQL failed such a completion with a
  database error and the task retried, while SQLite stored the over-long
  name; on SQLite such a completion is now refused.
• DW_MAX_HISTORY_EVENTS has no effect, and the reference now says so
  (#504). The Server
  does not send invocable_http activity requests, and the reference now says
  so too (#527).
• Native GitHub Releases carry the tagged release notes after the identity
  table (#1111).
• Tests now cover numeric-string names, selected-build routing and bounded
  observation for declared worker command contracts
  (#436).
• The two unproved APT paths in conformance and performance images run with
  APT retries off (durable-workflow/.github#295).

SDKs

• sdk-php 3.17.0 is out: typed request_id conflicts, replay that ignores
  run_closed cancellations, and a one-time structural-limit failure for an
  oversized completion.
• sdk-python 2.17.0 and sdk-rust 4.15.0 add the same handling:
  run_closed replay, the structural-limit failure, and, in sdk-rust, the
  key it sent on a request_id conflict.

Completion request limits and typed refusals

This release also corrects #1136:
HTTP/1 callers sending an oversized completion could get a transport error
instead of the server's typed 413 when the request reader closed during upload.
The server now keeps that reader alive for a bounded discard before returning
the original refusal and closing the connection. It discards at most 64 MiB of
remaining data for at most two seconds; uploads exceeding either bound may
still encounter a transport failure.

GET /api/cluster/info advertises the optional positive integer
limits.max_workflow_task_completion_bytes from the same function used by HTTP
admission. It covers both workflow-task completion and activity continuation
completion. Its default is 33,554,432 bytes, measured across the complete
encoded JSON request body, including envelopes, metadata and commands.
limits.max_payload_bytes remains the per-value and ordinary API limit; adding
decoded argument sizes does not measure the completion wire budget.

A completion over the cap returns HTTP 413 with reason: payload_too_large,
limit_kind: workflow_task_completion_bytes and the actual limit, before any
workflow state change. The shared wire vectors cover exact/over-cap requests
on both completion routes, and source controls cover fixed-length and chunked
TCP delivery, bounded discard, a valid 300×10 KiB fan-out and zero-write refusal.

SDK planners must use the explicit positive integer for the two completion
routes, including a positive cap smaller than the ordinary limit. Absence
preserves the existing conservative fallback. Explicit malformed/null/zero/
negative/non-integer values refuse completion planning before upload/submission;
ordinary routes ignore this unused field. The cap grants no new external-payload
upload or reference-placement authority. Activity/result-bound allowances cannot
be reused for continuation command arguments, even when bytes are identical;
continuation command arguments remain inline.

Shared #228 stays open
until the PHP, Python and Rust owners qualify actual installed workers on the
exact published Server/SDK tuple, including valid within-cap batches and the
one-time structural failure/follow-up behavior for oversized batches. Server
source tests and ordinary small fan-out successes do not establish that
installed acceptance. PHP/Rust planner package updates and their published
qualification are tracked there; an older planner can still apply the ordinary
limit to an aggregate and require unavailable external storage.

Separate reliability and qualification records

Source and native-image checks do not by themselves close the PHP reference's
20-run corpus gate (#1194)
or the separate intermittent MariaDB registration pool-timeout investigation (#1204).
This patch claims no underlying pool-timeout fix. Their current evidence and
remaining actions stay on those issues; neither incomplete nor
infrastructure-blocked controls count as passing qualification.