fix: keep the forkserver preload shared with job processes (gc.freeze) - #7118
Closed
Darshak03 wants to merge 1 commit into
Closed
fix: keep the forkserver preload shared with job processes (gc.freeze)#7118Darshak03 wants to merge 1 commit into
Darshak03 wants to merge 1 commit into
Conversation
The forkserver preloads plugin packages and the warmed native models so forked job processes inherit them copy-on-write, but the first full GC in a job process rewrites the GC header of every preloaded object and copies nearly every page holding one, undoing the sharing (~96MB per process). Freeze the preloaded objects into the permanent generation from a side-effect module listed last in the preload list, so the collector in each job process never touches them.
|
|
Author
|
Hi @chenghao-mou, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On Linux the worker preloads the registered plugin packages plus
livekit.agents.inference._warmupinto the forkserver process so job processesinherit the imported modules and the warmed native models copy-on-write.
The sharing does not last. Every job process runs its own cyclic GC, and a full
collection rewrites the
PyGC_Head(two pointers stored immediately before eachtracked object) of every object in the oldest generation — a write, even when
nothing is collectable. Preloaded modules, classes, functions and their dicts
are long-lived, so they all live there: the first full collection in a forked
child dirties nearly every page holding a preloaded object and the kernel copies
it. Measured on a Linux worker at ~96MB moving from shared to private per job
process, and it happens with no job traffic at all — an idle prewarmed process
already crosses a full collection during prewarm.
The native weights from
init_vad()/init_eot()are allocated outside thePython heap and stay shared; it is the Python object graph that is lost.
Fix
gc.freeze()in the forkserver, once everything is preloaded. Frozen objects goto the permanent generation, which the collector never scans, so their headers
are never rewritten and the pages stay shared for the life of the job process.
They are still freed by refcounting if they die, and module-level state lives for
the whole process anyway, so nothing is retained that would not have been.
It has to run inside the forkserver — job processes fork from there, so freezing
in the worker would not affect them. This reuses the mechanism already in the
tree for that: a side-effect module in the preload list, like
livekit.agents.inference._warmup. The new module is listed last so everythingabove it is resident by the time it runs.
The module deliberately imports nothing but
gc. The forkserver's preload loopswallows
ImportError, so putting the call at the end of_warmup.pywouldsilently skip the freeze on any host where
livekit-local-inferencefails toimport, even though the plugin preload — and the problem — is still there.
Also drops ~230k objects from every full collection in the job process, removing
a recurring GC pause from the audio path.
Verification
A child forked from a forkserver that preloads the new module reports
gc.get_freeze_count() == 233838, i.e. the freeze survives the fork and coversthe whole preloaded graph.
To see the memory effect on a Linux worker, sample a job process right after it
is forked and again after a full collection:
Shared_Cleandrops andPrivate_Dirtyrises by the same amount without thischange; with it, both hold.
Notes
spawn(macOS/Windows) — there is no preload there.gc.freeze()in the worker (wrong process), andgc.disable()/ raised thresholds in the job process (leaks real cycles forthe lifetime of a long session).
Fixes #7117