Hanging on shutdown when using a destructor

[mskucherawy]

The following code hangs at shutdown when run with kazoo 2.0 and python 3.5:

import kazoo.client

zeus_host = '<IP>:<PORT>'

class FooBar(object):
    def __init__(self, host):
        self.init_done = False
        self.kazoo = kazoo.client.KazooClient(
            hosts=host,
            handler=None,
            timeout=5,
            randomize_hosts=False
        )
        self.host = host
        self.kazoo.start()
        self.init_done = True

    def __del__(self):
        if self.init_done:
            print('FooBar.__del__ stop {}'.format(self.host))
            self.kazoo.stop()
            print('FooBar.__del__ close {}'.format(self.host))
            self.kazoo.close()
            print('FooBar.__del__ done {}'.format(self.host))

if __name__ == '__main__':
    x = FooBar(zeus_host)

My guess is that the atexit handler registered by start() is being called before the destructor, but somehow the destructor's call to stop() is still going past the _running check and into the rest of that method.

I couldn't see anything in the published ChangeLog that might indicate this is fixed in versions later than 2.0, so I thought I'd check here.

[bbangert]

Never, ever, do anything of any importance in a Python destructor. It is undefined when/how Python GC will run, what thread it will be called from, etc. To complicate things the atexit handler has similar surprising and fascinating timing/thread behaviors. I wouldn't even want to attempt that this all played well together, because even if it did in one version of Python, it might not later. PyPy for example may call such GC things at totally different times, or Jython, etc.

Make sure a signal handler or somethings shuts it all down so there's no GC or atexist surprises.

[mskucherawy]

Without this or something like it, kazoo can leak descriptors. For example:

    for x in range(10):
        x = FooBar(zeus_host)

...has 10 zeus clients open at shutdown, but a reference to only one of them. Fixing this without a destructor requires consumers of FooBar to know they have to explicitly shut each one down before it goes out of scope, and we have to provide them a method to do so. Is that really unavoidable? It seems to defeat the purpose of having destructors in the first place, and it's possibly a real problem for long-running processes.

[bbangert]

Well, given that there's no guarantee that Python will GC as soon as it goes out of scope, you should be shutting them down when you want them shutdown, not hoping that Python's GC will happen to run relatively close to the object leaving scope.

You would be better off having a "shutdown" method on your FooBar, and register those methods as atexit handlers if you have no other way to track them for clean-up later.

[harlowja]

Yes, never try to do much/anything in __del__ its going to end bad if u try.

Use contextlib.closing (or other context manager) and add a close() method or shutdown() and do all this yourself correctly.