You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(detector): stop leaking memory when a lib init keeps failing (#14)
The detector wrappers cache the library init failure and re-raise the same
exception object on every subsequent init() call to fail fast. Re-raising one
object is a leak: CPython appends a frame to its __traceback__ on each raise,
and those frames retain the caller's (detect()) locals indefinitely.
On a hybrid NVIDIA+AMD host where ROCm SMI init fails permanently, detect()
calls rsmi_init() every heartbeat/metrics tick, so anonymous RSS grows without
bound (gpustack/gpustack#5342).
Raise a fresh clone of the cached exception instead, via a new
clone_exception() helper. It clones with BaseException.__new__ + a copy of
args/__dict__ rather than type(exc)(*exc.args), because some binding error
types (e.g. amdsmi's AmdSmiLibraryException) take an error code and leave args
empty, so they can't be reconstructed from args.
Applied across all wrappers with the pattern: pyrocmsmi, pyamdsmi, pynvml,
pydcmi, pymtml, pyhgml, pyixml, pymxsml.