-
-
Notifications
You must be signed in to change notification settings - Fork 29.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
concurrent.futures.as_completed() memory inefficiency #71331
Comments
as_complite generator keeps reference of all passed futures until StopIteration. It may lead to serious memory inefficiency. Solution is to remove reference from lists and yield future ad-hoc. I have submitted patch and reproduce sample. I can create backport for older versions if needed. |
Is there a reason this hasn't been merged yet? Fixing this bug locally (albeit just by setting Future._result = None when I've extracted the result) reduced the memory footprint of my program from 50 GB to 7 GB, so it seems worth it. |
Any update on this reviewing, patching, and then backporting to 2.7? We've just run into the exact same issue here in Google using a ThreadPoolExecutor.map call growing memory usage without bounds due to the Future holding onto its result even after being returned from the map call. There's a more specific patch possible for just map() itself, but this more general one seems like it'd be better. |
concurrent package was added in 3.2. How backport to 2.7? Nosy myself. |
Git: https://github.com/agronholm/pythonfutures |
I've merged the PR to master and backported it to 3.6. Thank you Grzegorz for contributing this! |
"concurrent.futures.as_completed() memory inefficiency" hum, sadly the commit 97e1b1c introduced a reference leak. Example: $ ./python -m test -R 3:3 -test_concurrent_futures -m test.test_concurrent_futures.ProcessPoolAsCompletedTests.test_zero_timeout
(...)
test_concurrent_futures leaked [27, 27, 27] references, sum=81
test_concurrent_futures leaked [16, 17, 16] memory blocks, sum=49
(...) |
Thank you Victor. There is indeed a logic error in the new code. I'm opening a new PR. |
Tests are reusing finished futures. In the initial merge i propose to clear waiters but after review we decide it should be removed. I am confused now, should we change tests or restore initial |
The regression should be fixed now. |
Thank you both for this nice enhancement. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: