Is cupy.cuda.runtime set to None too early during interpreter shutdown?

[leofang]

• Conditions (you can just paste the output of python -c 'import cupy; cupy.show_config()')

$ python -c 'import cupy; cupy.show_config()'
CuPy Version          : 7.0.0rc1
CUDA Root             : /usr/local/cuda
CUDA Build Version    : 10000
CUDA Driver Version   : 10000
CUDA Runtime Version  : 10000
cuDNN Build Version   : 7604
cuDNN Version         : 7604
NCCL Build Version    : None
NCCL Runtime Version  : None

Tested with a fresh build on the latest master in a fresh conda environment.

• Code to reproduce

>>> import cupy as cp
>>> stream = cp.cuda.Stream()
>>> # ...then exit the interpreter
>>>
Exception ignored in: <bound method Stream.__del__ of <cupy.cuda.stream.Stream object at 0x7f75830870b8>>
Traceback (most recent call last):
  File "cupy/cuda/stream.pyx", line 186, in cupy.cuda.stream.Stream.__del__
AttributeError: 'NoneType' object has no attribute 'streamDestroy'

I don't think I've seen this error before. Looks like runtime is set to None when stream.__del__() is invoked, but this is unexpected because, according to PEP 442, starting at Python 3.4 the modules unloaded during interpreter shutdown are no longer set to None, and I'm on Python 3.6.7.

Can this be reproduced? Thanks.

[emcastillo]

Couldn't reproduce with python 3.7.3

[leofang]

What's the Cython version that you used? I am on 0.29.14.

[emcastillo]

0.29.13

[leofang]

Still see the error with Python 3.6.7 3.7.3 + Cython 0.29.13...Not sure what happened 😥

[emcastillo]

Have you tried python 3.7.x?

[leofang]

Sorry that was a typo. I tried 3.7.3. 😅

[leofang]

I still have no idea why this happened. I tested it on a DGX-2, which runs pre-installed Ubuntu 18.04 and CUDA 10.0 that come with the machine. My conda env was set up using conda create -n PY37 python=3.7 numpy scipy fastrlock and then conda activate PY37; pip install . Only this machine gives me AttributeError; other GPU machines I have access to are fine.

Closing this issue as it apparently involves some unknown env conflicts, and it actually already happened on this machine since CuPy v6.1.0, so definitely not master's fault.

@emcastillo Thanks for helping me test it! Let me know if you have any thought for how to diagnose. I'll also report back if I resolve it. (For the record, the last time I encountered the same error was in #2475.)

[leofang]

Reopen as a dup of #2777.

[niboshi]

#2809

[leofang]

@emcastillo I confirm that #2809 fixed the error. Thanks!