Skip to content

Fix a race condition that can leave a poll to not get cancelled during shutdown - #11841

Merged
rkannan82 merged 3 commits into
temporalio:mainfrom
rkannan82:fix-worker-shutdown-poll-race
Sep 3, 2026
Merged

Fix a race condition that can leave a poll to not get cancelled during shutdown#11841
rkannan82 merged 3 commits into
temporalio:mainfrom
rkannan82:fix-worker-shutdown-poll-race

Conversation

@rkannan82

@rkannan82 rkannan82 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

What changed?

Register worker polls before checking the shutdown cache.

Why?

Prevent a shutdown request from missing a poll that is still starting. The previous implementation handled polls that arrived after shutdown had populated the cache. It did not handle the case where the poll and shutdown calls are interleaved.

  • The poll checks the cache and sees the worker as active.
  • Shutdown records the worker and cancels registered polls, but this poll is not registered yet.
  • The poll registers afterward and remains outstanding.

How did you test it?

Tried adding a unit test to verify this exact race; but this gets into the guts of the impl and makes it hard to read.

  • existing unit test(s)

@rkannan82
rkannan82 requested a review from dnr August 28, 2026 03:21
@rkannan82
rkannan82 marked this pull request as ready for review August 28, 2026 03:23
@rkannan82
rkannan82 requested a review from a team August 28, 2026 03:23
@rkannan82
rkannan82 requested a review from a team as a code owner August 28, 2026 03:23
@rkannan82
rkannan82 marked this pull request as draft August 28, 2026 03:28
@rkannan82 rkannan82 changed the title Fix worker shutdown poll registration race Fix a race condition that can leave a poll to not get cancelled during shutdown Sep 1, 2026
@rkannan82
rkannan82 marked this pull request as ready for review September 1, 2026 21:14
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Claude finished @rkannan82's task in 4m 8s —— View job


Reviewed 1 file, 3 findings.
branch

Register worker-instance polls before checking the shutdown cache so a concurrent shutdown either cancels the registered poll or leaves a cache marker that rejects it.
@rkannan82
rkannan82 force-pushed the fix-worker-shutdown-poll-race branch from 8153846 to f3bfb78 Compare September 1, 2026 21:16
Comment thread service/matching/matching_engine.go Outdated
Comment thread service/matching/matching_engine.go
Comment thread service/matching/matching_engine.go Outdated
@rkannan82 rkannan82 added the reliability-2026 Reliability related changes label Sep 3, 2026
@rkannan82
rkannan82 requested a review from yiminc September 3, 2026 20:12
@rkannan82
rkannan82 enabled auto-merge (squash) September 3, 2026 20:15
@rkannan82
rkannan82 merged commit e8713b6 into temporalio:main Sep 3, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reliability-2026 Reliability related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants