Get rid of zombie RequestHandler threads #258
Conversation
This fixes a problem where, because RequestHandler instances registered themselves with atexit, they would always live until program exit. Having replaced the atexit call with the more usual __del__ finaliser, I found that this __del__ would never run: the queue-handling thread instantiated by RequestHandler holds references to self. Having also replaced that ref by a weakref, we did get the finaliser to run. Finally, to make sure we don't block on Queue.get() forever (in which case the thread still doesn't die), this also adds a timeout to that. Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl>
pykafka/handlers.py
Outdated
There was a problem hiding this comment.
We should have some warning message here - perhaps just at loglevel INFO.
There was a problem hiding this comment.
Thanks yes, coming up!
Small addition to the parent commit. I figured we might also benefit from a log line in RequestHandler.stop, since it took a while to realise that might hang (see #218). Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl>
|
This change is kind of scary in terms of how much it touches, but I've done a bunch of testing of both the consumer and producer on this branch, and I can't find any broken behavior. I think it's important to get |
Get rid of zombie RequestHandler threads
This fixes #257 by making sure
RequestHandlerinstances are gc'able.Because it removes the atexit flow completely, it also indirectly deals with #218 - with the changes here,
RequestHandler.stopwill run later in the teardown process, so that the consumers are already gone, so they no longer keep pressure on the request queue.