__del__ behavior on interpreter shutdown

[emcastillo]

When finishing program execution, python does not guarantee the order in which objects are destroyed.

Calls inside a __del__ method calling other objects or modules are not guaranteed to be correctly executed.

https://github.com/cupy/cupy/blob/master/cupy/cuda/stream.pyx#L185
It is known to fail because runtime module is already unloaded when __del__ is called.

https://docs.python.org/3/reference/datamodel.html#object.__del__

This can affect behavior on RAII pattern.

[emcastillo]

Maybe atexit module can be used?

https://docs.python.org/3/library/atexit.html

[leofang]

I think I tried registering the stream’s destructor with atexit and it didn’t work, but I don’t remember how exactly I did it, and I could do it wrong.

I think I also tried letting each stream instance hold a reference to the runtime module and it failed too. It seems the shutdown procedure doesn’t care about reference counting...

[niboshi]

As pointed out at #2658 (comment), the problem does not seem explained by Python's expected behavior?

I don't think I've seen this error before. Looks like runtime is set to None when stream.__del__() is invoked, but this is unexpected because, according to PEP 442, starting at Python 3.4 the modules unloaded during interpreter shutdown are no longer set to None, and I'm on Python 3.6.7.

[emcastillo]

Maybe is because of Cython?
Anyway PEP 442 says that
"The module shutdown procedure may not need to set global variables to None anymore. This could solve a well-known class of irritating issues."
I am not sure of what "may not" exactly means

[leofang]

Probably not Cython itself (as I tested several older and latest versions and the problem persists), but its bad interplay with Python. I also wondered if this could be a bug in Anaconda/conda-forge’s Python packaging, which I always use in production. Finally, it could just be a coincidence, but both bug reports (#2658, #2777) used CUDA 10.0.

[emcastillo]

I use CUDA 10.0 too.

[emcastillo]

We just found that if we install python with pyenv, then it breaks just with python -c 'import cupy; stream = cupy.cuda.Stream()'
Credit goes to @niboshi

[leofang]

Your pyenv is provided by the Linux distribution (apt-get, etc)?

[emcastillo]

Installed manually

Conda flags

-I$CONDA_PREFIX/include/python3.7m 
-I$CONDA_PREFIX/include/python3.7m  
-Wno-unused-result 
-Wsign-compare 
-march=nocona 
-mtune=haswell 
-ftree-vectorize 
-fPIC 
-fstack-protector-strong 
-fno-plt 
-O3 
-ffunction-sections 
-pipe  
-fdebug-prefix-map=/tmp/build/80754af9/python_1553721932202/work=/usr/local/src/conda/python-3.7.3 -fdebug-prefix-map=$CONDA_PREFIX=/usr/local/src/conda-prefix 
-fuse-linker-plugin 
-ffat-lto-objects 
-flto-partition=none 
-flto 
-DNDEBUG 
-fwrapv 
-O3 
-Wall

Pyenv python 3.7.3 flags (python3-config --cflags)

-I$HOME/.pyenv/versions/3.7.3/include/python3.7m 
-I$HOME/.pyenv/versions/3.7.3/include/python3.7m  
-Wno-unused-result 
-Wsign-compare 
-DNDEBUG 
-g 
-fwrapv 
-O3 
-Wall