Skip to content

4.22.0

Latest

Choose a tag to compare

@github-actions github-actions released this 21 Nov 04:12
· 4 commits to main since this release

4.22.0

  • A failed cluster-info read no longer stops a running worker. Startup
    discovery (query-task authority, Nexus cooperation) and the inline-payload
    policy read before a report retry a connection failure, timeout, 5xx or 429
    with the shared jittered backoff, uniform in [0, min(4 s, 0.25 s * 2^(n-1))], logging a tracing ERROR once the outage has lasted 60 s and
    every 5 minutes after, until it succeeds or the worker stops. An explicit
    retryable: false, 401, 403 and other refusals are still returned.
    Client::cluster_info and Clients outside a running worker make one attempt.
    When the cached payload policy (refreshed every 60 s) expires during such an
    outage, reports keep using the last-known policy and discovery is tried
    again 10 s later; only a worker that has no policy yet waits.

  • An activity report the Server refuses as too large (413 payload_too_large)
    because it lowered its inline limit since the policy was cached is no longer
    lost. The worker forgets the policy, rediscovers it, encodes the same report
    again (externalizing payloads the new limit requires) and resends it once.
    A result that is still too large (refused again, above the upload limit, or
    with no payload storage) no longer stops the worker: the attempt is failed
    as a non-retryable ActivityResultTooLarge (failure.type and
    failure.class), as in the PHP SDK, and the worker keeps running.

  • BEHAVIOUR CHANGE: Worker::recover_transient_outages now defaults to
    true.
    run and run_until keep running through transient Server
    outages instead of returning the error once WorkerRetryPolicy.max_retries
    is spent. Call .recover_transient_outages(false) to restore the previous
    bounded behavior.

  • A running worker no longer exits on a transient Server outage (the Rust
    side of sdk-php#448). Native Server answers any storage failure, including
    warm-up, with 503 {"reason":"storage_unavailable"}; after the default five
    retries run and run_until used to return that error. Transient failures
    are now a connection refused, reset or timed out; an upstream 502/503/504
    without a protocol body; a 503 with reason storage_unavailable or
    backend_unavailable; or any 5xx or 429 with retryable: true.
    Worker::recover_transient_outages is now on by default: poll and heartbeat
    retries continue past the retry budget for transients (a poll keeps its
    request identity) with the same fully jittered schedule rather than the
    fixed max_backoff delay, so workers do not retry in step; an outage that
    lasts 60 s is logged as a tracing error, then every 5 minutes, and a transient that escapes a poll lane, a task
    settlement or registration is logged with tracing and retried after a
    jittered wait, uniform in [0, min(4 s, 0.25 s * 2^(n-1))]. With the option
    on, only transients retry past the budget; an untyped 500 or a 429 without
    retryable: true is now returned once the budget is spent. An explicit
    retryable: false, 401, 403 and other refusals stop the worker with a
    tracing error event. .recover_transient_outages(false) restores the
    previous bounded behavior. 401 and 403 stop the worker, as in the PHP SDK;
    the Python SDK's pollers instead retry them so credentials can rotate.

  • An activity completion, failure or worker_lost release that hits a
    transient failure is no longer lost. The worker resends the identical report
    with the same jittered backoff while the lease lasts, then logs a tracing
    error and leaves the attempt to the Server. A resend after the Server had
    already committed the first copy is answered by native Server 3.17 with
    409 {"reason":"stale_attempt","recorded":false} for this task, attempt and
    owner with every status completed (or attempt_status failed); the
    worker counts that as its own recorded outcome. Verified against the
    published durableworkflow/server:3.17.0 image: the run completed with
    exactly one ActivityCompleted.

  • A timeout, transport error, 429 or 5xx on the activity lease renewal made
    before the callback starts no longer drops the claimed task (#302, the Rust
    side of sdk-php#445). Previously the Worker gave up on the claim without
    completing, failing or releasing it, so the Server held the lease until it
    expired and, with one attempt, failed the workflow. The callback now starts
    under the poll's original grant, and lease maintenance retries the renewal
    after a fully jittered wait, uniform in [0, min(4 s, 0.25 s * 2^n)] after
    the n-th consecutive failure, the same schedule as the other SDKs. Each
    attempt spends at most half of the remaining grant. A delivered lease is now
    measured from the poll's start rather than its reply. A refusal or a changed
    claim identity still ends the claim before the callback starts. Recorded
    history and replay are unchanged.

  • A claim whose renewal does not recover is now released explicitly instead of
    held silently until the lease expires. The Worker keeps a reserve at the end
    of the lease (at most 10 s and at most a fifth of the grant). Once renewal
    has failed and only that reserve is left, at start or later, it drops the
    callback future. Only after the callback has confirmed stopped, and only
    while more than a margin of the lease (at most 1 s) is left, it fails the
    attempt with failure.type worker_lost, failure.class
    ActivityWorkerLost, non_retryable: false and a message naming the local
    cause. A callback that blocks past that point is not released; the Server
    records lease_expired, so one attempt never runs twice. The Server's retry
    policy then acts at once, and worker_lost can be listed in
    non_retryable_error_types. A Server refusal such as lease_expired, as
    well as expiry and Worker shutdown, still ends the claim without a report.
    The default retry policy is unchanged.

  • ActivityFailure::lease_loss_reason() returns a LeaseLossReason
    (LeaseExpired, WorkerDeregistered or WorkerLost, with as_str() giving
    lease_expired, worker_deregistered or worker_lost; the enum is
    #[non_exhaustive]) when an attempt lost its worker, and None otherwise.
    WorkerLost is a Worker release (exception_type worker_lost). For a
    lease the Server settled (ActivityLeaseExpiredException) it is the
    Server's recorded failure_reason, LeaseExpired by default. The Server
    records that reason beside the exception, so replay now copies it into
    ActivityFailure::exception as failure_reason, creating a minimal
    {class, failure_reason} descriptor when the event recorded no exception.
    No existing type changes.

  • Worker egress is much smaller against a Server that supports it (#302).
    Requests now send Accept-Encoding: gzip and decode compressed answers
    (reqwest's gzip feature; native Server compresses JSON answers of 1 KiB or
    more, server#1261). Worker-plane requests also send
    X-Durable-Workflow-Server-Capabilities-Digest with the digest of the
    capabilities this Client holds, or none (server#1262). The Server then
    replaces each repeated ~17 KB server_capabilities copy with that digest,
    or tags a full copy with a digest member when the digest changed. The SDK
    captures the digest and object when it sends each request and restores that
    request's answer, success or error, at the top level and up to two objects
    deep, before anything reads it. A tagged copy is stripped of digest and
    used for later requests, so overlapping requests across a rolling deploy each
    restore their own capabilities. A digest string that matches neither the
    request's digest nor a tagged copy in the same answer makes the Client
    forget its held digest. Only a task poll, idempotent by its poll request ID,
    is resent once with none. Any other request (a completion, failure,
    heartbeat, start or signal) is never resent: its answer is kept as received,
    because a completion's answer can carry the next leased task, and the full
    capabilities are read with GET /api/cluster/info
    (worker_protocol.server_capabilities) and put in place of the unknown
    digest. If that read fails, the answer is still used without the field and
    later requests learn the capabilities again; the request never fails. A
    poll whose resend names an unknown digest again is Error::Protocol with
    reason server_capabilities_digest_unknown.
    Decoded response bodies are capped at 128 MiB by default
    (ClientBuilder::max_response_bytes); a larger body stops being read and
    fails with the new Error::ResponseBodyTooLarge, so a small gzip answer
    cannot expand without bound. Older
    Servers ignore both headers and answer in full. In a lab run of one
    start, one 1 KiB activity and one result per workflow against native Server
    main (43ad8d8f), response bytes per workflow fell from about 159 KB to 8.8 KB
    for the worker and from 19 KB to 4.7 KB for the caller.