New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_packaging reference leak #56376
Comments
Looks like either packaging or test_packaging forgets to clean up after itself: results for 9a16fa0c9548 on branch "default" test_packaging leaked [193, 193, 193] references, sum=579 |
Probably because new extension modules are built and imported on every run. |
Well, test_distutils does the same, doesn't it? |
I could not find any test in distutils/tests that imports extension modules. |
DistributionTestCase.test_hooks_get_run() leaks some references, I'm trying to understand where exactly. |
Let's see: -> test_command_bdist_dumb.py leaks: -> test_dist.py leaks: -> test_mixin2to3.py leaks: -> test_pypi_dist.py leaks: -> test_pypi_simple.py leaks: -> test_uninstall.py leaks: -> test_util.py leaks: |
New changeset 70675864717b by Victor Stinner in branch 'default': New changeset 28c1f8480090 by Victor Stinner in branch 'default': |
In test_command_bdist, the leak is in test_simple_built(). |
In test_mixin2to3.py, the leak is shared between the 3 test methods. |
In test_pypi_dist, the leak is shared between test_download() and test_unpack(). |
Is someone investigating this? |
I’m afraid I don’t understand enough to fix this. I looked at test_simple_built in test_command_bdist_dumb and found no C code involved. |
It means that either packaging or test_packaging keeps references to |
Ah, it’s just refcounting issues? I can probably use the gc module to find them, and add del statements where appropriate to release objects. |
New changeset fd6446a88fe3 by Victor Stinner in branch 'default': |
test_dist and test_bdist_dumb leak because of the _path_created global variable of packaging.util, used indirectly by copy_tree(). # cache for by mkpath() -- in addition to cheapening redundant calls,
# eliminates redundant "creating /foo/bar/baz" messages in dry-run mode
_path_created = set() I don't how/when this set should be cleared. |
Note: bpo-12133 has a patch to fix the ResourceWarning of test_pypi_simple. |
New changeset 27a70dfc38cc by Éric Araujo in branch 'default': |
test_build_ext builds and imports xx. |
We’re down to 100 refleaks. Thanks to the negative refleaks of test_pydoc, we now have a few refleaks to spare! Seriously, what does a negative refleak mean? |
It means that objects were garbage collected. The refleak test runs the test multiple times, and ignores the first N runs to allow the object count to "settle". But sometimes it either doesn't settle, or later runs end up with objects getting garbage collected that were created in earlier runs. |
I changed test test to use a subprocess: New changeset 144cea8db9a5 by Victor Stinner in branch 'default': It should fix refleaks because it is not possible to unload a module written in C. |
Yes, in packaging. I replied to an earlier question about distutils:
|
At least some of the remaining refleaks are caused by lib2to3. lib2to3 uses a logger with the filename as logger name (see |
Attached is a patch that replaces Another approach would be to simply remove the logging attribute of lib2to3 fixers, as it isn't used anyway. |
Thanks a lot for the diagnosis. To fix it, I don’t find the monkey-patching approach very good, I’d prefer properly using the logging API to remove handlers upon cleanup. Would you like to look into that? |
I don't think such stuff exists. |
AFAIU, the problem is that 2to3 creates a whole new logger for each |
Thanks for clarifying, I had misread. IMO, using one logger per file is a lib2to3 bug. |
I can confirm what Andreas said - I just removed the two lines referencing the logger in BaseFix, and ran the regression tests - with no failures. So I, too, think those lines should go. |
Let me clarify my position: I think there is a bug in lib2to3 that should be fixed, after what the refleak would go. (I’ll report it soon if nobody does it before.) If Benjamin rejects the bug, then I’ll agree with the monkey-patch. |
I've created bpo-12536 to cover the lib2to3 issue. |
It looks like we don’t have refleaks anymore! I still have one question. Victor reported some time ago that packaging.util._path_created was a cache that was never cleared; I fixed that in 27a70dfc38cc, but recently I’ve found that regrtest itself clears the similar distutils.util._path_created; I wonder which approach is better: one global cleaning in regrtest or piecemeal cleanup in each leaking test case? I’ve also made a patch to register the caches used by packaging.database with the regrtest unclean environment warning; can someone review it? |
I would call .copy() on the original dicts rather than remembering an |
I thought about that and decided to use an empty dict as a way to add a check that the caches should start empty. Maybe it was misguided and I should instead add a unit test for that, or not bother altogether. |
I edited my patch to use a copy instead of an explicit empty dict, but I found a bug. The restore code unpacks the saved_caches object to (cache, items), but saved_caches is (id(cache), cache, cache.copy()). I’m surprised the unpacking works; I don’t want to commit before I understand that. |
New changeset e76c6aaff135 by Éric Araujo in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: