Fix more logger related memory leaks on Windows #29957
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some observations have been made on Windows:
salt.log.setup.setup_multiprocessing_logging()
will leak. The reason isthat because Windows doesn't copy process state when spawning processes,
each new process will have the temp null and temp queue log handlers
installed (the default state in
salt/log/setup.py
when initializing).Those handlers just store stuff and hence will use much memory.
register_after_fork()
wouldn't work and so
salt.log.setup.setup_multiprocessing_logging()
wouldn't be invoked causing leaks. However, for some of the classes,
register_after_fork()
would work. After investigation, it was determinedthat
register_after_fork()
worked only on processes that used__setstate__
/__getstate__
and hence would invoke__init__()
insidethe child process. Thus,
register_after_fork()
only seems to work onWindows when invoked from within the child process, but not the parent
process, making it not very useful (since if we are in the child process,
we can invoke
salt.log.setup.setup_multiprocessing_logging()
directly).The following changes have been made based on these observations:
MultiprocessingProcess
so as to callsalt.log.setup.setup_multiprocessing_logging()
.MultiprocessingProcess
invokesalt.log.setup.setup_multiprocessing_logging()
themselves.MultiprocessingProcess
invoke__init__
from the child process.In order to accomplish this, all such processes use
__setstate__
and__getstate__
.multiprocessing.Process
is constructed via__init__
in the childprocess, certain features such as
is_alive()
andident()
don't work.Thus we set
self._is_child = True
in each__setstate__
to be able todistinguish whether the
__init__
invocation is from the parent or childprocess.
Signed-off-by: Sergey Kizunov sergey.kizunov@ni.com