shared map cache was unlinking, now doesnt #4183
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Here’s what we added and why it’s needed:
Allows us to talk to Python’s multiprocessing resource tracker, which automatically unregisters shared-memory blocks when a process exits. On platforms without it, we no-op.
Central method that unregisters a SharedMemory block from the resource tracker. Without this, every process that attaches to an existing block registers it, and its exit unlinks the block, leaving the cache
pointing at a missing segment and triggering rebuilds.
Right after creating a new shared-memory block, we unregister it. This prevents the creating process’s eventual exit from unlinking the block and invalidating the cache.
If a block already exists (another process made it), we attach and immediately unregister; otherwise that attaching process would be registered to unlink the block on exit, causing thrash.
When reconstructing maps from registry metadata, we may attach to an existing block. We unregister after attaching so that reader processes don’t get ownership and inadvertently unlink the block on exit.