Skip to content

Let activities heartbeat during worker shutdown#2903

Open
baekgyu-kim wants to merge 1 commit into
temporalio:masterfrom
baekgyu-kim:2075
Open

Let activities heartbeat during worker shutdown#2903
baekgyu-kim wants to merge 1 commit into
temporalio:masterfrom
baekgyu-kim:2075

Conversation

@baekgyu-kim
Copy link
Copy Markdown
Contributor

What was changed

  • On the worker-shutdown path (heartbeatExecutor already shut down), HeartbeatContextImpl.heartbeat() now emits the heartbeat to the server before throwing ActivityWorkerShutdownException (previously: thrown without sending).
  • Throttled like the normal scheduled path (success → heartbeat interval, transient failure → short retry interval), reusing the last heartbeat's throttle state so a caller looping on heartbeat() cannot flood the server.
  • Send errors are swallowed (WARN-logged); ActivityWorkerShutdownException is always thrown, so a transient failure cannot mask the shutdown signal.
  • ActivityWorkerShutdownException Javadoc updated accordingly.

Why?

  • Before: on shutdown, heartbeat() threw immediately without contacting the server → no heartbeat during the awaitTermination grace period → server times the activity out and retries it → duplicate executions, despite the worker deliberately giving the activity time to finish.
  • After: the activity can keep heartbeating during the grace period (by catching the exception) → each call refreshes the server-side heartbeat deadline → no premature timeout/retry.
  • Design: emit-then-throw is the approach agreed in Add the ability to keep heartbeating while the worker is shutting down #2075; throttling keeps a manual heartbeat loop from flooding the server.

Checklist

  1. Closes Add the ability to keep heartbeating while the worker is shutting down #2075

  2. How was this tested:

    New HeartbeatContextImplTest cases:

    • emit-then-throw on the shutdown path
    • exception still thrown when the send fails (gRPC and non-gRPC)
    • repeated rapid calls throttled to a single send
    • prompt retry after a failed send (short interval, not a full one)
    • resend once the throttle interval elapses
    • throttle carry-over from a preceding normal heartbeat — fails if the throttle-state write is removed
    ./gradlew :temporal-sdk:test --tests "io.temporal.internal.activity.HeartbeatContextImplTest"
    ./gradlew :temporal-sdk:spotlessJavaCheck
    
  3. Any docs updates needed?

    • The behavior is documented in the ActivityWorkerShutdownException Javadoc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the ability to keep heartbeating while the worker is shutting down

1 participant