pandas: C extension pandas.lib not included

[jksfiftyfive]

Hi,

Im using python 2.7.9 32 bit python on windows 7.

I tried this with the development version of pyinstaller

Im getting this error when trying to run my exe. it seems to build ok, but doesnt execute. The following message appears:

ImportError: C extension: lib not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
DataCombinerApp returned -1

I tried running, 'python setup.py build_ext --inplace' without luck.

I have found several suggestions on the web, including

1. downloading VCC compiler for python 2.7 and re-installing pandas with zip/source
2. uninstalling/installing panda using condas
   3)installing pandas using pip

still get the error

Any ideas?

thanks

[jksfiftyfive]

i should also add that the code works fine. i dont get the error when running within python

im only having issues with the executable

[htgoebel]

You probably need to create a hook for pandas. See http://pythonhosted.org/PyInstaller/#understanding-pyinstaller-hooks. The description there should still be okay for your case.

[jksfiftyfive]

But what file(s) should the hook point to? looks like its not complaining that it cant find the file, it looks like its complaining that a C extension file isnt built.

Any ideas?
thanks

[notgonnacodethat]

hi i'm experiencing the same problem, got the following error during build (both --onedir and --onefile) and running the .exe

ImportError: C extension: lib not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.

i tried to build with "python setup.py build_ext --inplace --force" as well as using pyinstaller 3.1.dev0, still no luck

[matysek]

But what file(s) should the hook point to? looks like its not complaining that it cant find the file, it looks like its complaining that a C extension file isnt built.

Any ideas?
thanks

@jksfiftyfive It looks like pyinstaller does not bundle C extension and pandas thus complains that it was not compiled.

• Question is why do you need to compile anything with pandas yourself when there are prebuilt pandas wheel packages on pypi: https://pypi.python.org/pypi/pandas/
• You still haven't provided enough info for reproducing this issue: https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong, https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs

[dloman]

I'm also having this issue trying to use pyinstaller on centos. I don't understand which files I need to add to hook.

I tried running with the --onedir, --onefile and --noupx flags

[dloman]

fixed this issue on my end. Pyinstaller wasn't including `$PYTHONDIR/site-packages/pandas/lib.so`` just copied this into the dist directory and named it pandas.lib.so and it appears to be working. Pyinstaller is awesome keep up the good work!

[matysek]

@dloman thanks for debugging

[rth]

@matysek Regarding your earlier comment,

Question is why do you need to compile anything with pandas yourself when there are prebuilt pandas wheel packages on pypi

I can confirm that the same issue occurs with prebuilt pandas package for anaconda on Windows 8.1 (python 2.7).

Here is a minimal working example:

test.py

import pandas as pd

assert pd.lib.is_float(1) == 0

Makefile.bat

conda create --yes -n test_env pip future nose cython setuptools=18 pywin32 libpython numpy=1.9 scipy=0.15 pandas=0.14 python=2
call activate test_env
pip install pyinstaller
pyinstaller test.py
call deactivate

The full build output can be found here with in particular this error:

IOError: [Errno 22] invalid mode ('rb') or filename: 'c:\\miniconda\\envs\test_env\\lib\\site-packages\\pandas\\lib.pyd'

which is strange since lib.pyd does exist in the system (although in C:\Miniconda\envs\test_env\Lib\site-packages\pandas I'm not sure why the path would be in lower case in the logs).

Edit: So, at least in anaconda on Windows, the problem is not that pandas.lib is not included by pyinstaller, but that the process errors while doing so (therefore I'm not sure if adding hooks etc will help). On Linux (Gentoo, system Python 2.7) I get the following error with the above test.py:

Traceback (most recent call last):
 File "<string>", line 5, in <module>
 File "pandas/lib.pyx", line 1, in init pandas.lib (pandas/lib.c:79957)
 File "/usr/lib64/python2.7/site-packages/pandas/__init__.py", line 13, in <module>
   "extensions first.".format(module))
ImportError: C extension: lib not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.

Where pandas was installed with the system package manager (portage), so normally everything is installed correctly (although, yes, it is compiled from sources). Same thing happens with Python 3.4.

Great work for the 3.0 release btw!

[notgonnacodethat]

@dloman thanks for your answer, works like a charm
for windows, just copy lib.pyd from pandas folder to dist folder and rename it to pandas.lib.pyd

[tallforasmurf]

could this be managed with an added binaries item? As described here:
https://pythonhosted.org/PyInstaller/#adding-binary-files

[notgonnacodethat]

@tallforasmurf tried modifying the spec file by adding binary / data file, but the "pandas.lib.pyd" file will be automatically placed in "dist/pandas.lib.pyd" folder instead of the root "dist" folder and that doesn't work

anyone know how can I modify in the spec file so that a binary can placed in the same folder as the .exe?

[dloman]

I also tried modifying the binaries section of the spec file and that didn't work. I got --onefile version working by first running pyinstaller --onefile pandasScript.py then going into the build directory and editing *Analysis.toc, *COLLECT.toc, *EXE.toc, and the *PKG.toc files. In all of those files I searched for the word hashtable and found this line ('pandas.hashtable', '/usr/local/lib/python2.7/site-packages/pandas/hashtable.so', 'EXTENSION'),. I added ('pandas.lib', '/usr/local/lib/python2.7/site-packages/pandas/lib.so', 'EXTENSION'), immeditaly following the hashtable line and then reran pyinstaller --onefile pandasScript.py and it works. Not sure why adding pandas.lib into the spec file didn’t work but this is an inelegant work around. Hope this helps

[tallforasmurf]

@dloman if what you say is true, I think that if the following change to your spec file does not work,

a = Analysis([' ... script name ... '],
 ...
 binaries= [   (  '/usr/local/lib/python2.7/site-packages/pandas/lib.so', 'pandas.lib' ) ] ,
...

...then there must be a bug somewhere. Because that is exactly what the binaries arg is supposed to accomplish. (Somebody double-check my syntax plz?)

[matysek]

@jksfiftyfive @dloman or @rth could anyone of you confirm that these commits fix this issue?

[rth]

@matysek I can confirm that the above commits fixed the issue for me on Windows 8.1 with Anaconda, (Python 2.7 and 3.4), as well as on Linux (Python 2.7 and 3.4). Thanks!

[soupault]

Fixed for me also (Win7, Anaconda, Python 3.4).

[dloman]

Fixed for me as well 👍

[ProfPh]

The commits didn't work for me but the latest dev version did fix it!

[joedborg]

Confirmed the dev version (3.1dev0) is the winner.

[MajjigaDeepak]

After including pandas.lib in dist folder & adding in binary in spec file.i executed the app but still same error.Should I do any build after this change?

[matysek]

@MajjigaDeepak if you have any issues please create a new report with exact description technical details.