Skip to content

Release the GIL during the activity heartbeat core call#1643

Open
odedkedemdreeze wants to merge 1 commit into
temporalio:mainfrom
odedkedemdreeze:fix/heartbeat-ffi-release-gil
Open

Release the GIL during the activity heartbeat core call#1643
odedkedemdreeze wants to merge 1 commit into
temporalio:mainfrom
odedkedemdreeze:fix/heartbeat-ffi-release-gil

Conversation

@odedkedemdreeze

Copy link
Copy Markdown

What was changed

WorkerRef::record_activity_heartbeat in the pyo3 bridge now detaches from the GIL (Python::detach, formerly allow_threads) for the call into core. The heartbeat proto is still decoded with the GIL held (it borrows the Python bytes), so only owned data crosses the detach boundary.

Also adds a regression test: a trivial custom slot supplier + a workflow fanning out heartbeating activities (tests/worker/test_worker.py::test_custom_slot_supplier_with_heartbeating_activities).

Why?

Fixes the permanent worker deadlock described in #1642: any worker using a CustomSlotSupplier whose activities heartbeat can wedge forever.

  • The heartbeat FFI held the GIL while core's record_heartbeat blocked on the outstanding_activity_tasks mutex.
  • Meanwhile an activity task start invokes the supplier's mark_slot_used — whose bridge wrapper unconditionally acquires the GIL — while core holds that same mutex (guard alive during insert(...) argument evaluation).

Classic ABBA: thread A holds the mutex and wants the GIL; thread B holds the GIL and wants the mutex. Releasing the GIL during the core call means the cycle cannot form, and as a bonus a heartbeat contending on that mutex no longer freezes the entire Python process (event loop and all unrelated threads) for the duration.

The other half of the cycle (core invoking lang callbacks while holding internal locks, including via Rust temporary-lifetime guard extension in insert(...) argument position) is described in #1642 for the maintainers' consideration; this PR fixes the Python side, which alone breaks the cycle.

Checklist

  1. Closes [Bug] Permanent worker deadlock: any CustomSlotSupplier + heartbeating activities (GIL ↔ sdk-core mutex ABBA) #1642

  2. How was this tested:

  • New regression test test_custom_slot_supplier_with_heartbeating_activities (600 heartbeating activities through a capped custom slot supplier). Passes in ~13s with this change. Note: on regression it hangs rather than fails — the deadlock freezes the event loop, so no in-process timeout can fire (the deterministic non-hanging variant of this test would belong in sdk-core, where the lock ordering can be unit-tested directly).

  • The reproducer from [Bug] Permanent worker deadlock: any CustomSlotSupplier + heartbeating activities (GIL ↔ sdk-core mutex ABBA) #1642 (8000 heartbeating activities, no-op custom slot supplier), unpatched temporalio==1.30.0:

    starting: 8000 activities, cap=12, 150 heartbeats each
    ...wedges before the first activity completes; faulthandler dump at 20s:
    Timeout (0:00:20)!
    Thread 0x00000001f74fdd80 (most recent call first):
      File ".../temporalio/bridge/worker.py", line 258 in record_activity_heartbeat   <-- holds GIL, wants core mutex
      File ".../temporalio/worker/_activity.py", line 290 in _heartbeat_async
      ...
    

    Same reproducer against this branch (editable install, patched bridge):

    [  31.1s] completed=7817
    [  31.6s] completed=7954
    workflow COMPLETED without a deadlock -- raise the load and retry
    
  • cargo fmt --check, cargo clippy -- -D warnings (bridge), ruff check --select I / ruff format --check on the touched test file: clean.

  • All four *slot_supplier* tests in tests/worker/test_worker.py pass locally.

  1. Any docs updates needed?

None — behavior-preserving fix, no API change.

record_activity_heartbeat was a synchronous pyo3 fn that held the GIL
while core's record_heartbeat blocked on the outstanding-activity lock.
Task starts invoke a custom slot supplier's mark_slot_used (which
acquires the GIL) while holding that same lock, so any worker with a
custom slot supplier and heartbeating activities could deadlock
permanently (ABBA on GIL vs. lock).

Decode the heartbeat proto with the GIL held (it borrows the Python
bytes), then detach from the GIL for the core call so the cycle cannot
form.

Fixes temporalio#1642

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@odedkedemdreeze odedkedemdreeze requested a review from a team as a code owner July 14, 2026 13:16
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


“Oded seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


“Oded seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Permanent worker deadlock: any CustomSlotSupplier + heartbeating activities (GIL ↔ sdk-core mutex ABBA)

2 participants