Skip to content

Conversation

@relh
Copy link
Contributor

@relh relh commented Dec 4, 2025

Summary

  • Fixed a shared map cache bug where the multiprocessing resource_tracker would auto-unlink shared-memory map blocks when worker processes exited, leaving stale registry entries and forcing constant map rebuilds.
  • We now unregister each shared-memory block from the tracker when creating or attaching in SharedMapCache, keeping cache entries valid across processes and eliminating the rebuild thrash.

Here’s what we added and why it’s needed:

  • Import resource_tracker (best-effort)
    Allows us to talk to Python’s multiprocessing resource tracker, which automatically unregisters shared-memory blocks when a process exits. On platforms without it, we no-op.
  • _untrack_shared_memory helper
    Central method that unregisters a SharedMemory block from the resource tracker. Without this, every process that attaches to an existing block registers it, and its exit unlinks the block, leaving the cache
    pointing at a missing segment and triggering rebuilds.
  • Untrack after create
    Right after creating a new shared-memory block, we unregister it. This prevents the creating process’s eventual exit from unlinking the block and invalidating the cache.
  • Untrack when reusing existing SHM (FileExistsError path)
    If a block already exists (another process made it), we attach and immediately unregister; otherwise that attaching process would be registered to unlink the block on exit, causing thrash.
  • Untrack when re-attaching on reconstruction
    When reconstructing maps from registry metadata, we may attach to an existing block. We unregister after attaching so that reader processes don’t get ownership and inadvertently unlink the block on exit.

@relh relh enabled auto-merge December 4, 2025 21:34
@relh relh added this pull request to the merge queue Dec 4, 2025
Merged via the queue into main with commit 3706917 Dec 4, 2025
19 checks passed
@relh relh deleted the richard-untrack branch December 4, 2025 22:00
zfogg pushed a commit that referenced this pull request Dec 20, 2025
Summary

- Fixed a shared map cache bug where the multiprocessing
resource_tracker would auto-unlink shared-memory map blocks when worker
processes exited, leaving stale registry entries and forcing constant
map rebuilds.
- We now unregister each shared-memory block from the tracker when
creating or attaching in SharedMapCache, keeping cache entries valid
across processes and eliminating the rebuild thrash.

Here’s what we added and why it’s needed:

  - Import resource_tracker (best-effort)
Allows us to talk to Python’s multiprocessing resource tracker, which
automatically unregisters shared-memory blocks when a process exits. On
platforms without it, we no-op.
  - _untrack_shared_memory helper
Central method that unregisters a SharedMemory block from the resource
tracker. Without this, every process that attaches to an existing block
registers it, and its exit unlinks the block, leaving the cache
    pointing at a missing segment and triggering rebuilds.
  - Untrack after create
Right after creating a new shared-memory block, we unregister it. This
prevents the creating process’s eventual exit from unlinking the block
and invalidating the cache.
  - Untrack when reusing existing SHM (FileExistsError path)
If a block already exists (another process made it), we attach and
immediately unregister; otherwise that attaching process would be
registered to unlink the block on exit, causing thrash.
  - Untrack when re-attaching on reconstruction
When reconstructing maps from registry metadata, we may attach to an
existing block. We unregister after attaching so that reader processes
don’t get ownership and inadvertently unlink the block on exit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants