Error in numpy/numpy/distutils/misc_util.py

[emmenlau]

I'm trying to install a package that compiles against numpy. It fails with error:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 27, in clean_up_temporary_directory
    for d in _tmpdirs:
TypeError: 'NoneType' object is not iterable
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 27, in clean_up_temporary_directory
    for d in _tmpdirs:
TypeError: 'NoneType' object is not iterable
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 27, in clean_up_temporary_directory
    for d in _tmpdirs:
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/dist-packages/numpy/distutils/misc_util.py", line 27, in clean_up_temporary_directory
    for d in _tmpdirs:
TypeError: 'NoneType' object is not iterable

I checked numpy/numpy/distutils/misc_util.py and it seems the problem could be that _tmpdirs is empty? At least the following change compiles for me:

_tmpdirs = []
def clean_up_temporary_directory():
    if _tmpdirs is not None:
        for d in _tmpdirs:
            try:
                shutil.rmtree(d)
            except OSError:
                pass    

[rgommers]

@emmenlau thanks for the report. Can you please tell us exactly how to reproduce this? That would make it a lot easier to check your proposed fix.

[emmenlau]

@rgommers thanks for the quick response. Can you help me isolate a better report? I'm trying to compile CellProfiler (see my link to the issue CellProfiler/CellProfiler#2113). But that is a relatively complex software / complex setup for you to reproduce I guess?

[rgommers]

That is a bit complex, but the link to that issue did help. The issue you're reporting is a secondary problem right, the real issue is the Cython crash when compiling pandas? That explains why I've never seen clean_up_temporary_directory before - it's probably not used when things go well.

Another question: does this fail when you do pip install . instead of python setup.py install? It's failing from easy_install, which is better not used.

Anyway, after looking at your proposed fix I think it is harmless, so even if I can't easily reproduce this and it doesn't make much sense that _tmpdirs is None (there's no code inside misc_util that can change it from a list to None) I'm fine with making this change. Want to send a PR with your fix? If so, please add a comment referring to this issue.

[emmenlau]

Interesting, it seems that from a pip install . it works! I'm not sure how the two are related, is this basically the same thing by different means?

[rgommers]

It's not, the difference is that if you use python setup.py install then pip is not involved and setuptools does everything - including looking at install_requires in setup.py and invoking easy_install for whatever is listed there. Now easy_install is basically a horrible piece of crap, and officially not supported by Numpy. It works in most cases, but not always (as can be seen here).

[rgommers]

As said above though, if you send your proposed change as a PR I'm happy to merge it.

[emmenlau]

Thanks rgommers, I think I learned something today :-) So I will change our install instructions to avoid the python setup.py install and instead use pip install .. I don't know why previous instructions said first to use the one, then the other. It seems to work with just the latter, and I understand now that this is preferred. Thanks!

[rgommers]

You're welcome.

[rgommers]

Historically the command has always been python setup.py install, just the last years pip has become preferred. If there wouldn't be install_requires in setup.py, then it doesn't matter too much.