[Python] Generated code is out of contract - application hangs indefinitely

[gaborbernat]

The generated code snippet from - https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-codegen/src/main/resources/python/api_client.mustache#L73 is not allowed:

    def __del__(self):
        if self._pool is not None:
            self._pool.close()
            self._pool.join()

The very simple program of creating a global client, and then just exiting will hang indefinitely on Python3.8. Initially raised as CPython bug under https://bugs.python.org/issue39360, but @pablogsal (core CPython developer) investigated and concluded that this way of cleaning up pools is not supported. Instead one should either allow the user to close or if the user did not do the generated code should do by registering https://docs.python.org/3.8/library/atexit.html callbacks.

The code in question seemed to be added within #6396 by @tuxbotix with approvers being @scottrw93 @taxpon @frol @mbohlool, so tagging them for context.

[pablogsal]

To clarify: the reason is "not supported" is because the interpreter does not promise that __del__ with being even called or in what situations that will happen. In particular, there are multiple locks and threads that can hang if __del__ is called during finalization or from the gc. From https://docs.python.org/3/reference/datamodel.html#object.__del__:

Warning Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderr instead. In particular:
* __del__() can be invoked when arbitrary code is being executed, including from any arbitrary thread. If __del__() needs to take a lock or invoke any other blocking resource, it may deadlock as the resource may already be taken by the code that gets interrupted to execute __del__().

* __del__() can be executed during interpreter shutdown. As a consequence, the global variables it needs to access (including other modules) may already have been deleted or set to None. Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.

This worked on Python <3.7 in a very flaky way: the pool used a busy loop (now it uses a more robust system of sentinels) and the sleep in that busy loop was big enough that make the pool more likely to be consistent in this scenario, but technically is the same problem. Indeed, reducing the timeout of the busy loop makes it hang as well in 3.7 or older.

[fabianvf]

We're seeing indefinite hangs in the __del__ method in the Kubernetes Python client that seem consistent with this issue

[noamgat]

Was this ever merged?

[fabianvf]

@noamgat it was merged to the openapi repo, never here

[MasterSergius]

Is there any update on this issue? It hangs still, here's my code to check:

from swagger_client import ApiClient, Configuration, UserApi

configuration = Configuration()
configuration.host = host
configuration.api_key['Authorization'] = token
configuration.api_key_prefix['Authorization']='Bearer'


client = UserApi(ApiClient(configuration))

for item in client.list_users():
    print(item)

# and it can't exit the script, hangs indefinitely