Repository navigation
4.22.0
-
A failed cluster-info read no longer stops a running worker. Startup
discovery (query-task authority, Nexus cooperation) and the inline-payload
policy read before a report retry a connection failure, timeout, 5xx or 429
with the shared jittered backoff, uniform in[0, min(4 s, 0.25 s * 2^(n-1))], logging atracingERROR once the outage has lasted 60 s and
every 5 minutes after, until it succeeds or the worker stops. An explicit
retryable: false, 401, 403 and other refusals are still returned.
Client::cluster_infoand Clients outside a running worker make one attempt.
When the cached payload policy (refreshed every 60 s) expires during such an
outage, reports keep using the last-known policy and discovery is tried
again 10 s later; only a worker that has no policy yet waits. -
An activity report the Server refuses as too large (413
payload_too_large)
because it lowered its inline limit since the policy was cached is no longer
lost. The worker forgets the policy, rediscovers it, encodes the same report
again (externalizing payloads the new limit requires) and resends it once.
A result that is still too large (refused again, above the upload limit, or
with no payload storage) no longer stops the worker: the attempt is failed
as a non-retryableActivityResultTooLarge(failure.typeand
failure.class), as in the PHP SDK, and the worker keeps running. -
BEHAVIOUR CHANGE:
Worker::recover_transient_outagesnow defaults to
true.runandrun_untilkeep running through transient Server
outages instead of returning the error onceWorkerRetryPolicy.max_retries
is spent. Call.recover_transient_outages(false)to restore the previous
bounded behavior. -
A running worker no longer exits on a transient Server outage (the Rust
side of sdk-php#448). Native Server answers any storage failure, including
warm-up, with503 {"reason":"storage_unavailable"}; after the default five
retriesrunandrun_untilused to return that error. Transient failures
are now a connection refused, reset or timed out; an upstream 502/503/504
without a protocol body; a 503 with reasonstorage_unavailableor
backend_unavailable; or any 5xx or 429 withretryable: true.
Worker::recover_transient_outagesis now on by default: poll and heartbeat
retries continue past the retry budget for transients (a poll keeps its
request identity) with the same fully jittered schedule rather than the
fixedmax_backoffdelay, so workers do not retry in step; an outage that
lasts 60 s is logged as atracingerror, then every 5 minutes, and a transient that escapes a poll lane, a task
settlement or registration is logged withtracingand retried after a
jittered wait, uniform in[0, min(4 s, 0.25 s * 2^(n-1))]. With the option
on, only transients retry past the budget; an untyped 500 or a 429 without
retryable: trueis now returned once the budget is spent. An explicit
retryable: false, 401, 403 and other refusals stop the worker with a
tracingerror event..recover_transient_outages(false)restores the
previous bounded behavior. 401 and 403 stop the worker, as in the PHP SDK;
the Python SDK's pollers instead retry them so credentials can rotate. -
An activity completion, failure or
worker_lostrelease that hits a
transient failure is no longer lost. The worker resends the identical report
with the same jittered backoff while the lease lasts, then logs atracing
error and leaves the attempt to the Server. A resend after the Server had
already committed the first copy is answered by native Server 3.17 with
409 {"reason":"stale_attempt","recorded":false}for this task, attempt and
owner with every statuscompleted(orattempt_statusfailed); the
worker counts that as its own recorded outcome. Verified against the
publisheddurableworkflow/server:3.17.0image: the run completed with
exactly oneActivityCompleted. -
A timeout, transport error, 429 or 5xx on the activity lease renewal made
before the callback starts no longer drops the claimed task (#302, the Rust
side of sdk-php#445). Previously the Worker gave up on the claim without
completing, failing or releasing it, so the Server held the lease until it
expired and, with one attempt, failed the workflow. The callback now starts
under the poll's original grant, and lease maintenance retries the renewal
after a fully jittered wait, uniform in[0, min(4 s, 0.25 s * 2^n)]after
the n-th consecutive failure, the same schedule as the other SDKs. Each
attempt spends at most half of the remaining grant. A delivered lease is now
measured from the poll's start rather than its reply. A refusal or a changed
claim identity still ends the claim before the callback starts. Recorded
history and replay are unchanged. -
A claim whose renewal does not recover is now released explicitly instead of
held silently until the lease expires. The Worker keeps a reserve at the end
of the lease (at most 10 s and at most a fifth of the grant). Once renewal
has failed and only that reserve is left, at start or later, it drops the
callback future. Only after the callback has confirmed stopped, and only
while more than a margin of the lease (at most 1 s) is left, it fails the
attempt withfailure.typeworker_lost,failure.class
ActivityWorkerLost,non_retryable: falseand a message naming the local
cause. A callback that blocks past that point is not released; the Server
recordslease_expired, so one attempt never runs twice. The Server's retry
policy then acts at once, andworker_lostcan be listed in
non_retryable_error_types. A Server refusal such aslease_expired, as
well as expiry and Worker shutdown, still ends the claim without a report.
The default retry policy is unchanged. -
ActivityFailure::lease_loss_reason()returns aLeaseLossReason
(LeaseExpired,WorkerDeregisteredorWorkerLost, withas_str()giving
lease_expired,worker_deregisteredorworker_lost; the enum is
#[non_exhaustive]) when an attempt lost its worker, andNoneotherwise.
WorkerLostis a Worker release (exception_typeworker_lost). For a
lease the Server settled (ActivityLeaseExpiredException) it is the
Server's recordedfailure_reason,LeaseExpiredby default. The Server
records that reason beside the exception, so replay now copies it into
ActivityFailure::exceptionasfailure_reason, creating a minimal
{class, failure_reason}descriptor when the event recorded no exception.
No existing type changes. -
Worker egress is much smaller against a Server that supports it (#302).
Requests now sendAccept-Encoding: gzipand decode compressed answers
(reqwest'sgzipfeature; native Server compresses JSON answers of 1 KiB or
more, server#1261). Worker-plane requests also send
X-Durable-Workflow-Server-Capabilities-Digestwith the digest of the
capabilities this Client holds, ornone(server#1262). The Server then
replaces each repeated ~17 KBserver_capabilitiescopy with that digest,
or tags a full copy with adigestmember when the digest changed. The SDK
captures the digest and object when it sends each request and restores that
request's answer, success or error, at the top level and up to two objects
deep, before anything reads it. A tagged copy is stripped ofdigestand
used for later requests, so overlapping requests across a rolling deploy each
restore their own capabilities. A digest string that matches neither the
request's digest nor a tagged copy in the same answer makes the Client
forget its held digest. Only a task poll, idempotent by its poll request ID,
is resent once withnone. Any other request (a completion, failure,
heartbeat, start or signal) is never resent: its answer is kept as received,
because a completion's answer can carry the next leased task, and the full
capabilities are read withGET /api/cluster/info
(worker_protocol.server_capabilities) and put in place of the unknown
digest. If that read fails, the answer is still used without the field and
later requests learn the capabilities again; the request never fails. A
poll whose resend names an unknown digest again isError::Protocolwith
reasonserver_capabilities_digest_unknown.
Decoded response bodies are capped at 128 MiB by default
(ClientBuilder::max_response_bytes); a larger body stops being read and
fails with the newError::ResponseBodyTooLarge, so a small gzip answer
cannot expand without bound. Older
Servers ignore both headers and answer in full. In a lab run of one
start, one 1 KiB activity and one result per workflow against native Server
main (43ad8d8f), response bytes per workflow fell from about 159 KB to 8.8 KB
for the worker and from 19 KB to 4.7 KB for the caller.