Some backports namespace packages missing __init__.py

[mattbillenstein]

lI'm not sure why this is happening, but some of the backports packages I'm installing from pypi include backports/init.py -- but when I install them via nix, the init.py isn't included:

[nix-shell:~/src/t3/nix]$ find /nix/store -name 'backports' | xargs ls -l                                                                   
/nix/store/3nh41m1skb2sczh61zhbgvdzmjylvfxp-python2.7-backports.functools-lru-cache-1.2.1/lib/python2.7/site-packages/backports:
total 16
-r--r--r-- 1 mattb mattb 7317 Dec 31  1969 functools_lru_cache.py
-r--r--r-- 1 mattb mattb 7922 Dec 31  1969 functools_lru_cache.pyc

/nix/store/44xsw62p1d2xd38m7qjzd4zfnpf5x13w-python2.7-backports.shutil_get_terminal_size-1.0.0/lib/python2.7/site-packages/backports:
total 12
-r--r--r-- 1 mattb mattb   75 Dec 31  1969 __init__.py
-r--r--r-- 1 mattb mattb  334 Dec 31  1969 __init__.pyc
dr-xr-xr-x 2 mattb mattb 4096 Dec 31  1969 shutil_get_terminal_size

/nix/store/78cqndb6wym88kva5amndnjqxzhxxl7g-python2.7-configparser-3.5.0/lib/python2.7/site-packages/backports:
total 4
dr-xr-xr-x 2 mattb mattb 4096 Dec 31  1969 configparser

/nix/store/idrxbwdnrcsri6wkrz18n2nvg7rn086r-python2.7-backports.shutil-get-terminal-size-1.0.0/lib/python2.7/site-packages/backports:
total 12
-r--r--r-- 1 mattb mattb   75 Dec 31  1969 __init__.py
-r--r--r-- 1 mattb mattb  334 Dec 31  1969 __init__.pyc
dr-xr-xr-x 2 mattb mattb 4096 Dec 31  1969 shutil_get_terminal_size

note functools_lru_cache and configparser missing backports/init.py

One of these packages is getting picked up off of PYTHONPATH when invoking ipython which then can't find the actual backport package that it needs:

[nix-shell:~/src/t3/nix]$ ipython
Traceback (most recent call last):
  File "/nix/store/hv9qgwimw4q7xyklqiavxj2c1pw7ci1k-python2.7-python2.7-ipython-5.1.0/bin/.ipython-wrapped", line 8, in <module>
    from IPython import start_ipython
  File "/nix/store/hv9qgwimw4q7xyklqiavxj2c1pw7ci1k-python2.7-python2.7-ipython-5.1.0/lib/python2.7/site-packages/IPython/__init__.py", line 48, in <module>
    from .core.application import Application
  File "/nix/store/hv9qgwimw4q7xyklqiavxj2c1pw7ci1k-python2.7-python2.7-ipython-5.1.0/lib/python2.7/site-packages/IPython/core/application.py", line 25, in <module>
    from IPython.core import release, crashhandler
  File "/nix/store/hv9qgwimw4q7xyklqiavxj2c1pw7ci1k-python2.7-python2.7-ipython-5.1.0/lib/python2.7/site-packages/IPython/core/crashhandler.py", line 28, in <module>
    from IPython.core import ultratb
  File "/nix/store/hv9qgwimw4q7xyklqiavxj2c1pw7ci1k-python2.7-python2.7-ipython-5.1.0/lib/python2.7/site-packages/IPython/core/ultratb.py", line 128, in <module>
    from IPython.utils.terminal import get_terminal_size
  File "/nix/store/hv9qgwimw4q7xyklqiavxj2c1pw7ci1k-python2.7-python2.7-ipython-5.1.0/lib/python2.7/site-packages/IPython/utils/terminal.py", line 22, in <module>
    from backports.shutil_get_terminal_size import get_terminal_size as _get_terminal_size
ImportError: No module named shutil_get_terminal_size

ipython is wrapped to setup site packages correctly and only includes the single backport, so just emptying out PYTHONPATH will make things work again:

[nix-shell:~/src/t3/nix]$ PYTHONPATH="" ipython
Python 2.7.12 (default, Jun 25 2016, 21:49:32) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:                                                                                                                                                                                                                                                                                  
Do you really want to exit ([y]/n)? 

[nix-shell:~/src/t3/nix]$ 

hSo, I'm not sure if this is a pypi2nix issue directly and it seems there are issues with python environments in general, but dropping the init.py in place in these packages seems to fix this issue.

[garbas]

@mattbillenstein tnx for the report ... i'll look at this before making new release. in the worse case you can fix it within *_generated.nix file

[garbas]

@mattbillenstein finally found some time to debug this.

in short: i think the issue is upstream due to missing namespace_packages=['backports'],

command i used to create an environment is

$ pypi2nix -V "2.7" -e backports.functools-lru-cache -e backports.shutil_get_terminal_size  -e configparser -e backports.shutil-get-terminal-size -e setuptools_scm -v

also this override was needed

  "backports.functools-lru-cache" = python.overrideDerivation super."backports.functools-lru-cache" (old: {
    buildInputs = old.buildInputs ++ [ self."setuptools-scm" ];
  });

above should be able to be build using

$ nix-build requirements.nix -A interpreter

command.

backports packages that work are:

$ ./result/bin/python -c "import backports.functools_lru_cache; import backports.configparser;"

backports packages that dont work is:

$ ./result/bin/python -c "import backports.shutil_get_terminal_size;"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named shutil_get_terminal_size

comparing all three setup.py files (here, here, here) shows that the difference is missing namespace_packages=['backports'], in backports.shutil_get_terminal_size
package.

I would suggest you fork the project (also submit the PR upstream) and point to github archive of your fork like i'm doing in case of Pillow

[mattbillenstein]

Great -- thanks for the excellent debug!

[mattbillenstein]

FYI chrippa/backports.shutil_get_terminal_size#10