Installation problem on FreeBSD 10.0 with Python 3.4.3

[meeb]

Hi,

I'm attempting to install pandas in a virtualenv on a FreeBSD system with Python 3.4.3. I can't see that I'm missing any obvious steps or dependancies. I get an error in a C module when importing pandas if I install pandas from pip, from a release tarball and with the repo (same error every time) that would imply something pretty major is missing. numpy, scipy and scikit-learn all install without any issues in the same venv. Example of the error:

(venv)hostname% pip install --upgrade pandas
Collecting pandas
Requirement already up-to-date: pytz>=2011k in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: numpy>=1.7.0 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: python-dateutil>=2 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: six>=1.5 in ./venv/lib/python3.4/site-packages (from python-dateutil>=2->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.16.2

(venv)hostname% python
Python 3.4.3 (default, Jun 18 2015, 17:00:30)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 7, in <module>
    from pandas import hashtable, tslib, lib
ImportError: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 13, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string" not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
>>>

When installing from a tarbal or the a checkout of the repo where setup.py is accessible and following the above suggestion (python setup.py build_ext --inplace) it compiles without issue it does not resolve the issue.

Some system information (print_versions.py taken from a repo checkout):

(venv)hostname% python print_versions.py

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: FreeBSD
OS-release: 10.1-RELEASE-p10
machine: amd64
processor: amd64
byteorder: little
LC_ALL: None
LANG: None

pandas: None
nose: 1.3.7
Cython: 0.22.1
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 1.0.0
tables: None
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)

Any pointers would be appreciated, thanks.

[bashtage]

Are there errors related to Cython in the setup log if you run

python setup.py install

[meeb]

No errors from Cython. Cython was installed from pip. I have also tried removing Cython and installing/compiling it manually from the latest 0.22.1 tarball. Cython installs perfectly without issues as far as I can see.

I removed and re-installed Pandas from the repo after manually installing Cython and this had no effect. I have also been able to reproduce this error on a clean FreeBSD 10.1 vm.

[bashtage]

Sorry, I mean when you build install pandas. Something like:

cd ~/pandas
python setup.py clean
python setup.py build

Does the log from the last step show anything strange?

[meeb]

There's a load of warnings and notes, but it does seem to build all the required files. Here's the full output:

https://gist.githubusercontent.com/meeb/160fd9b409491cb9546a/raw/4dbad70b8f359a3714e82dd999571dfc0f471f28/gistfile1.txt

[jreback]

@meeb these are all ok

[meeb]

Yeah thought as much. Any other info I can provide or is this actually just something wrong in the build process on this platform?

[jreback]

I think you might have a partial pip install. try

pip uninstall pandas
pip install pandas

[meeb]

I've tried installing this in multiple completely clean venvs on different FreeBSD 10.1 installs (on different physical machines) all with the same results.

It's reproducible for Python3.4.2 on every FreeBSD 10.1 install I've tried so far. Interestingly, I've just noticed that installing pandas in a Python2.7.9 venv works fine without any errors at all on the same machines, both building from repo and installing via pip. Log:

https://gist.github.com/meeb/e68a685ad9acf038b7ae

[iblis17]

Hi!
I have python34 on FreeBSD 10.1, and face the same problem.
After digging around the code, I figure out that the root cause is the behavior of inline.
On gcc4x, it will choose gnu89 as the C model.

... But on FreeBSD 10, we are using clang3.4, which choose standard C99 model by default.
Ref is here

So... how to fix it? I will compile it with following command:

CFLAGS='-fgnu89-inline' python setup.py build_ext -f --inplace

Maybe the proper way is patching the Makefile.

[jreback]

way to do this properly is to put in options based on the platform in setup.py. Though AFAIK this has never been an issue before; I would make sure that cython is up to date.

[iblis17]

Hi @jreback !
Hmm... actually the problematic code do not come from cython. Those code from cython inline always carry static.
I will make some explanation.

E.g. consider pandas/src/numpy_helper.h

PANDAS_INLINE npy_int64                                                            
get_nat(void) {                                                                    
  return NPY_MIN_INT64;                                                            
}

Note that there is a macro called PANDAS_INLINE.

This macro is defined in pandas/src/helper.h.
By default, this macro will be expanded as __inline__, because clang will define __GUNC__ as well.

Those code are not generated by cython.

IMO, there are two solution for this inline problem. (To be honest, many project still have this problem on FreeBSD, e.g. our depend: ujson)

1. Simply using the CFLAGS -fgnu89-inline
2. Make sure our(also our depends) macro PANDAS_INLINE consider following situation:

• __GNUC_GNU_INLINE__ (this is the behavior of --std=gnu89) define as __inline__
• __GNUC_STDC_INLINE__ (--std=c99 or later) define as static inline

[jreback]

ok I see

well happy to take a patch that ifdefs things and add platform based CFLAGS is ok too (would add in setup.py)

[jreback]

btw tha pandas.json IS ujson (with a layer on top for pandas use)

[iblis17]

ok, i will send a PR later 😄
It's AM 3:00 here .... zzzzz

[ghost]

I know this issue is closed, but just for anyone interested: The problem also exists with Python 3.4.3 on FreeBSD 10.2 (freshly installed from CD) at least on my machine.

[iblis17]

Hi @usrbinerl !
Hmm... have you tried the patch in #11398?

[ghost]

Yes, I just did, but didn't resolve the issue. But take that with a grain of salt. I lack experience with compiling python from source, so I may have done something wrong.
So far I did not had to do that with python3 and my code works fine on Ubuntu 15.04/15.10 with pip3 installed pandas-package.

[iblis17]

@usrbinerl Here is my workflow, FYI :)

pyvenv-3.4 venv
source venv/bin/activate
pip install cython                          
pip install git+https://github.com/pydata/pandas.git

Note that cython first

Then, pandas works fine for me

(venv) > python
Python 3.4.3 (default, May 26 2015, 17:46:50) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> 

[ghost]

Thanks. I'll try with your tips again this afternoon.

[jvkersch]

@meeb @iblis17 @usrbinerl I encountered a similar problem on Mac OS (which also uses Clang) when building a Pandas with debug symbols. I tackled this by adding the static keyword to inline function declarations and in the process I removed the -fgnu89-inline flag that was added in 0ec93fb. Would you mind giving my branch #12123 a spin on your FreeBSD systems? Thanks!

[iblis17]

@jvkersch Nice work!
It works fine on my bsd box.

[ghost]

Works for me too now on FreeBSD. Thanks for your help.

[bashtage]

You might also want to try adding -std=c99 to the build, which uses modern, standard inlining behavior.

[jreback]

@bashtage is that a more general comment or only for FreeBSD?