Installation issue with --editable option

[Eimis]

Hello,

I'm trying to install pandas on Mac OS X 10.11.6 in a Django project. Pandas library is used with one of the 3rd party libraries that I am using in the project.

Since I don't want to switch to different package manager (Anakonda) just for installing pandas (I'm using pip now), in order for me to get the C files compiled, I decided to install pandas by including --editable git+git://github.com/pydata/pandas.git@v0.18.1#egg=pandas in the requirements.txt file and the library is now installed in my src dir and there's a symlink in my virtualenv's site-packages which points to it.

This gets these headers compiled, but causes many strange importing errors in pandas source code itself when it is imported by a third party library that is also installed with --editable mode. F. ex. file at Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/knmi-py/knmi/parsers.py has this line import pandas as pd. And I had to edit this code block in virtualenv/src/pandas/pandas/__init__.py for things to work:

try:
    # XXX: I had to comment this out for things to work:
    # from pandas import hashtable, tslib, lib
    import hashtable, tslib, lib
except ImportError as e:  # pragma: no cover
    module = str(e).lstrip('cannot import name ')  # hack but overkill to use re
    raise ImportError("C extension: {0} not built. If you want to import "
                      "pandas from the source directory, you may need to run "
                      "'python setup.py build_ext --inplace' to build the C "
                      "extensions first.".format(module))

But then another importing error appears here: virtualenv/src/pandas/pandas/core/config.py:

File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/pandas/pandas/core/config.py", line 57, in <module>
    import pandas.compat as compat
AttributeError: 'module' object has no attribute 'compat'

What should I do to get these things fixed ? I'd love to be able to use this library with pip and Django.

[Eimis]

Edit: I think I solved this by wrapping editable pandas library in virtualenv/src with another pandas folder which contains the library itself and single __init__.py file. So now it looks like this:

[chris-b1]

Is there any reason you need to compile? pip install pandas will install a pre-compiled wheel.

[jreback]

installation instructions: http://pandas.pydata.org/pandas-docs/stable/install.html

using a distribution is best best.

[Eimis]

@chris-b1 I need to compile, because my Django project uses a third party library KNMI-py and one of the dependencies for this 3rd party library is pandas. This KNMI-py library is importing pandas in its source code and I'm getting an error message that C files are not compiled. These C files are not compiled when you do pip install pandas, they're only compiled when you install the package as --editable. This is the code block from pandas source that does that:

try:
    # XXX: these are actualy C files (hashtable.c and etc.):
    from pandas import hashtable, tslib, lib
except ImportError as e:  # pragma: no cover
    module = str(e).lstrip('cannot import name ')  # hack but overkill to use re
    raise ImportError("C extension: {0} not built. If you want to import "
                      "pandas from the source directory, you may need to run "
                      "'python setup.py build_ext --inplace' to build the C "
                      "extensions first.".format(module))

^ I think my case counts as 'importing pandas from source directory', because a 3rd party library is doing that.

@jreback sorry, I'm not exactly sure I understand what you mean by 'using a distribution', but if you mean conda install pandas, this is not quite suitable for my case, because it would require to switch to a new package manager. And we have a lot of automation tools tied to the requirements.txt file used by pip, so I'd really like to continue using this pip approach..

So unless I'm missing something, I can come up with only one workaround which is forking pandas and including it as an --editable package in my requirements.txt file, which will compile these C files for me.

[jorisvandenbossche]

I'm getting an error message that C files are not compiled. These C files are not compiled when you do pip install pandas, they're only compiled when you install the package as --editable

This is not necessarily true. There are binary wheels available on PyPI for many platforms/python versions, including MacOSX.

I think my case counts as 'importing pandas from source directory', because a 3rd party library is doing that.

KNMI-py is importing pandas in its source code (eg https://github.com/EnergyID/KNMI-py/blob/master/knmi/parsers.py#L1), but it is not importing pandas from the source, that is something different. KNMI-py will just import the installed pandas available in the python runtime, whether it is an editable install or not. So it is not needed to have pandas installed as editable.

So unless I'm missing something, I can come up with only one workaround which is forking pandas and including it as an --editable package in my requirements.txt file, which will compile these C files for me.

That should not be necessary. pip does normally install binary wheels. If that is not the case, you should first check why. And even then, when pip installs the source files, it should also try to compile those during installation.

Unless you want to develop on pandas, installing it as --editable is not needed.

[Eimis]

@jorisvandenbossche But if I am gonna use wheels, how am I gonna automate the setup process of the Project for every developers platform, I can not specify which .whl file to download from pip in my requirements.txt file. I really love the smoothness of pip install -r requirements.txt when everything works for everyone just after executing one command.

[jorisvandenbossche]

But if I am gonna use wheels, how am I gonna automate the setup process of the Project for every developers platform

The same applies for building pandas from source (different needs to build c extensions on each platform).
And again, if you list pandas as requirement, pip will just download the correct wheel, and if there is no wheel available for the specific platform (but there are for windows/linux/macosx), pip will also just install the source and try to build the C extensions, just the same as you would do with an editable install.

[Eimis]

And even then, when pip installs the source files, it should also try to compile those during installation.

Unfortunately when pip installs .whl file, C files are not compiled:

(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ pip install --use-wheel pandas
Collecting pandas
  Using cached pandas-0.18.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): six in ./virtualenv/lib/python2.7/site-packages (from python-dateutil->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.18.1
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ ls virtualenv/lib/python2.7/site-packages/pandas/
__init__.py _sparse.so  _version.pyc    computation hashtable.so    info.py     json.so     parser.so   sparse      tools       types
__init__.pyc    _testing.so algos.so    core        index.so    info.pyc    lib.so      rpy     stats       tseries     util
_period.so  _version.py compat      formats     indexes     io      msgpack     sandbox     tests       tslib.so
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$

You can see that there are no C files compiled in this directory :(

So I'm still left with the only available solution which is specifying --editable git+git://github.com/pydata/pandas.git@v0.18.1#egg=pandas in my requirements.txt and then wrapping virtualenv/src/pandas/ in another pandas folder with __init__.py file to avoid those importing errors..

[jorisvandenbossche]

You can see that there are no C files compiled in this directory :(

All the .so files are compiled C files

With what you show above, in you just open a python console and import pandas, what do you get? Does that give an error?

[Eimis]

@jorisvandenbossche thanks and sorry for my mistake. It seems that you're right and now C files can be found after installing .whl file. But everything works only after wrapping virtualenv/lib/python2.7/site-packages/pandas in another pandas folder with __init__.py... I still have no idea why those import errors are present in pandas source code.