Fix up shutdown for periodic executors

[ghost]

Hello.

It's from a robot test that uses loop operator. FOR ${lengthOfSCHMCDXXXX} IN 35 15 36

Just temporary debug messages:

_shutdown_executors set([])
_shutdown_executors None
_shutdown_executors None

We get this error when we launch tests.

unexpected error: Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "C:\Python27\lib\site-packages\pymongo\periodic_executor.py", line 133, in _shutdown_executors
    executors = list(_EXECUTORS)
TypeError: 'NoneType' object is not iterable
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "C:\Python27\lib\site-packages\pymongo\periodic_executor.py", line 133, in _shutdown_executors
    executors = list(_EXECUTORS)
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:

[2016.04.26-15:25:09] : accessed by C:\Python27\lib\site-packages\robot\run.py
Traceback (most recent call last):
  File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "C:\Python27\lib\site-packages\pymongo\periodic_executor.py", line 133, in _shutdown_executors
    executors = list(_EXECUTORS)
TypeError: 'NoneType' object is not iterable

test finished 20160426 15:25:09

[behackett]

@ajdavis I've seen this running WinKerberos tests as well. Any idea why we have this problem in master, but seemingly not in v2.9?

[behackett]

Also, note that this only seems to happen on Windows. I've never seen it anywhere else.

[behackett]

@gaal-dev what is the robot package (I don't see this in pypi) and does it spawn processes using multiprocessing or subprocess? See the note here:

https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process.terminate

[behackett]

This might be related (at least for WinKerberos):

https://briandamaged.org/blog/setuptools-test-command-hates-atexit/

The sys.path manipulation mentioned in the blog post is still in the latest version of setuptools.

[behackett]

Possibly related:

pypa/setuptools#38

This is why PyMongo imports multiprocessing in its setup.py, even though it doesn't use it.

[ajdavis]

Until PEP 442 was implemented in Python 3.4, the Python interpreter shutdown sequence set all module globals, like _EXECUTORS, to None before destroying the modules.

https://github.com/python/cpython/blob/2.7/Objects/moduleobject.c#L117

(Interestingly, there's a special case for names starting with underscore, which means _EXECUTORS is set to None before other names.)

In Py_Finalize, the interpreter calls call_sys_exitfunc before PyImport_Cleanup, as you'd expect it to. So I can't imagine how _EXECUTORS is None, unless your code is running in an exotic environment besides the regular Python interpreter.

Regardless, I don't see any harm accepting this patch.

[ghost]

And for Linux too. I'm not sure why it happens and not always repeats.

Robot Framework http://robotframework.org/

[ghost]

Will you accept this patch?

[behackett]

Hi @gaal-dev, we probably will accept your patch for the next release of PyMongo, but I want to spend some time trying to understand why this happens in the first place (global variables shouldn't be set to None before atexit runs). This is a cosmetic problem either way.

[jness]

This pull request has resolved my PyMongo unittest issue, can't wait to see it merged!

Thanks

[behackett]

Thanks for reminding me about this. I've merged it to master.