bpo-46309: Added reference to task created by StreamReaderProtocol #30505
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
There was a problem hiding this comment.
Welcome to CPython's repo! Thank you for your contribution. As written by our lovely bot, please sign the CLA agreement.
I've added a few changes to your code. Right now the task will be replaced in case of multiple concurrent connections which is not ideal. I've created a set to hold the tasks and added a callback to remove them once done.
|
|
||
| async def callback(*args): | ||
| await self.loop.create_future() |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
Thanks for looking at the changes! I signed the CLA ~3 hours before this PR, so hopefully it will be OK'd sometime today. Your requested changes makes sense. However from my testing, creating the future outside of the callback will not allow the task to be garbage collected. That is, the test will pass even without keeping a reference to the task in StreamReaderProtocol.
One solution could be to still create the future inside the callback, but cancel the task via the newly created reference in StreamReaderProtocol. Perhaps you have a better suggestion. I can spend some more time on this later today.
There was a problem hiding this comment.
You are completely correct. I've spent quite a few hours triaging the source of the original issue on stackoverflow and realized the bug is caused when the garbage collector breaks a cyclic reference between the writer, the reader, the task and a future created inside the reader. If any of them are referenced outside of the function, the bug will not appear.
I don't suggest accessing the internals from the test. You're welcome to decline my changes in the tests as long as asyncio doesn't throw any warning about the leak. The garbage collector will take care of it when the loop is closed, or it will be cancelled accordingly.
There was a problem hiding this comment.
I've accepted your suggested changes in streams.py. As for the leaking test callback, a different solution could be to use asyncio.all_tasks(self.loop).pop().cancel() right after the assertion.
There was a problem hiding this comment.
I believe it's unnecessary - It will cause more confusion, and will get bugged if the internals will change and a new task will be created in the background.
I think we're better off letting the cleanup take care of it.
| protocol.connection_made(transport) | ||
| await asyncio.sleep(0) | ||
| gc.collect() | ||
|
|
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Co-authored-by: Bar Harel <bzvi7919@gmail.com>
There was a problem hiding this comment.
Looks perfect.
As a side note, the tests contain only one task and not multiple tasks created at once. That might be preferable - the code being tested tested is simple and goes through the same path for both single and multiple tasks. In turn, it allows the test itself to be simpler as well and achieve good coverage.
Thank you for your contribution @simwr872!
|
This PR is stale because it has been open for 30 days with no activity. |
There was a problem hiding this comment.
This appears to be superseded by GH-96323. The latter PR is simpler, it doesn't collect multiple tasks in a set, it just stores a single task, because connection_made() is only ever called once per StreamReader instance. Because it's so simple it doesn't add a test. (The test here seems controversial among past reviewers anyway.)
If anyone here doesn't agree with my approving and merging GH-96323 please let me know!
|
The following commit authors need to sign the Contributor License Agreement: Click the button to sign: |
Seems ok. |
No reference was kept to a task created by StreamReaderProtocol and was therefore eligible for garbage collection under certain circumstances.
https://bugs.python.org/issue46309
https://bugs.python.org/issue46309