New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation problem on FreeBSD 10.0 with Python 3.4.3 #10510

Closed
meeb opened this Issue Jul 4, 2015 · 24 comments

Comments

Projects
None yet
5 participants
@meeb

meeb commented Jul 4, 2015

Hi,

I'm attempting to install pandas in a virtualenv on a FreeBSD system with Python 3.4.3. I can't see that I'm missing any obvious steps or dependancies. I get an error in a C module when importing pandas if I install pandas from pip, from a release tarball and with the repo (same error every time) that would imply something pretty major is missing. numpy, scipy and scikit-learn all install without any issues in the same venv. Example of the error:

(venv)hostname% pip install --upgrade pandas
Collecting pandas
Requirement already up-to-date: pytz>=2011k in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: numpy>=1.7.0 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: python-dateutil>=2 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: six>=1.5 in ./venv/lib/python3.4/site-packages (from python-dateutil>=2->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.16.2

(venv)hostname% python
Python 3.4.3 (default, Jun 18 2015, 17:00:30)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 7, in <module>
    from pandas import hashtable, tslib, lib
ImportError: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 13, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string" not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
>>>

When installing from a tarbal or the a checkout of the repo where setup.py is accessible and following the above suggestion (python setup.py build_ext --inplace) it compiles without issue it does not resolve the issue.

Some system information (print_versions.py taken from a repo checkout):

(venv)hostname% python print_versions.py

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: FreeBSD
OS-release: 10.1-RELEASE-p10
machine: amd64
processor: amd64
byteorder: little
LC_ALL: None
LANG: None

pandas: None
nose: 1.3.7
Cython: 0.22.1
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 1.0.0
tables: None
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)

Any pointers would be appreciated, thanks.

@meeb meeb changed the title from installation problem on FreeBSD 10.0 with Python 3.4.3 to Installation problem on FreeBSD 10.0 with Python 3.4.3 Jul 4, 2015

@bashtage

This comment has been minimized.

Show comment
Hide comment
@bashtage

bashtage Jul 6, 2015

Contributor

Are there errors related to Cython in the setup log if you run

python setup.py install
Contributor

bashtage commented Jul 6, 2015

Are there errors related to Cython in the setup log if you run

python setup.py install
@meeb

This comment has been minimized.

Show comment
Hide comment
@meeb

meeb Jul 7, 2015

No errors from Cython. Cython was installed from pip. I have also tried removing Cython and installing/compiling it manually from the latest 0.22.1 tarball. Cython installs perfectly without issues as far as I can see.

I removed and re-installed Pandas from the repo after manually installing Cython and this had no effect. I have also been able to reproduce this error on a clean FreeBSD 10.1 vm.

meeb commented Jul 7, 2015

No errors from Cython. Cython was installed from pip. I have also tried removing Cython and installing/compiling it manually from the latest 0.22.1 tarball. Cython installs perfectly without issues as far as I can see.

I removed and re-installed Pandas from the repo after manually installing Cython and this had no effect. I have also been able to reproduce this error on a clean FreeBSD 10.1 vm.

@bashtage

This comment has been minimized.

Show comment
Hide comment
@bashtage

bashtage Jul 7, 2015

Contributor

Sorry, I mean when you build install pandas. Something like:

cd ~/pandas
python setup.py clean
python setup.py build

Does the log from the last step show anything strange?

Contributor

bashtage commented Jul 7, 2015

Sorry, I mean when you build install pandas. Something like:

cd ~/pandas
python setup.py clean
python setup.py build

Does the log from the last step show anything strange?

@jreback jreback added the Build label Jul 7, 2015

@meeb

This comment has been minimized.

Show comment
Hide comment
@meeb

meeb Jul 7, 2015

There's a load of warnings and notes, but it does seem to build all the required files. Here's the full output:

https://gist.githubusercontent.com/meeb/160fd9b409491cb9546a/raw/4dbad70b8f359a3714e82dd999571dfc0f471f28/gistfile1.txt

meeb commented Jul 7, 2015

There's a load of warnings and notes, but it does seem to build all the required files. Here's the full output:

https://gist.githubusercontent.com/meeb/160fd9b409491cb9546a/raw/4dbad70b8f359a3714e82dd999571dfc0f471f28/gistfile1.txt

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Jul 7, 2015

Contributor

@meeb these are all ok

Contributor

jreback commented Jul 7, 2015

@meeb these are all ok

@meeb

This comment has been minimized.

Show comment
Hide comment
@meeb

meeb Jul 7, 2015

Yeah thought as much. Any other info I can provide or is this actually just something wrong in the build process on this platform?

meeb commented Jul 7, 2015

Yeah thought as much. Any other info I can provide or is this actually just something wrong in the build process on this platform?

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Jul 7, 2015

Contributor

I think you might have a partial pip install. try

pip uninstall pandas
pip install pandas
Contributor

jreback commented Jul 7, 2015

I think you might have a partial pip install. try

pip uninstall pandas
pip install pandas
@meeb

This comment has been minimized.

Show comment
Hide comment
@meeb

meeb Jul 7, 2015

I've tried installing this in multiple completely clean venvs on different FreeBSD 10.1 installs (on different physical machines) all with the same results.

It's reproducible for Python3.4.2 on every FreeBSD 10.1 install I've tried so far. Interestingly, I've just noticed that installing pandas in a Python2.7.9 venv works fine without any errors at all on the same machines, both building from repo and installing via pip. Log:

https://gist.github.com/meeb/e68a685ad9acf038b7ae

meeb commented Jul 7, 2015

I've tried installing this in multiple completely clean venvs on different FreeBSD 10.1 installs (on different physical machines) all with the same results.

It's reproducible for Python3.4.2 on every FreeBSD 10.1 install I've tried so far. Interestingly, I've just noticed that installing pandas in a Python2.7.9 venv works fine without any errors at all on the same machines, both building from repo and installing via pip. Log:

https://gist.github.com/meeb/e68a685ad9acf038b7ae

@iblis17

This comment has been minimized.

Show comment
Hide comment
@iblis17

iblis17 Oct 20, 2015

Contributor

Hi!
I have python34 on FreeBSD 10.1, and face the same problem.
After digging around the code, I figure out that the root cause is the behavior of inline.
On gcc4x, it will choose gnu89 as the C model.

... But on FreeBSD 10, we are using clang3.4, which choose standard C99 model by default.
Ref is here

So... how to fix it? I will compile it with following command:

CFLAGS='-fgnu89-inline' python setup.py build_ext -f --inplace

Maybe the proper way is patching the Makefile.

Contributor

iblis17 commented Oct 20, 2015

Hi!
I have python34 on FreeBSD 10.1, and face the same problem.
After digging around the code, I figure out that the root cause is the behavior of inline.
On gcc4x, it will choose gnu89 as the C model.

... But on FreeBSD 10, we are using clang3.4, which choose standard C99 model by default.
Ref is here

So... how to fix it? I will compile it with following command:

CFLAGS='-fgnu89-inline' python setup.py build_ext -f --inplace

Maybe the proper way is patching the Makefile.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 20, 2015

Contributor

way to do this properly is to put in options based on the platform in setup.py. Though AFAIK this has never been an issue before; I would make sure that cython is up to date.

Contributor

jreback commented Oct 20, 2015

way to do this properly is to put in options based on the platform in setup.py. Though AFAIK this has never been an issue before; I would make sure that cython is up to date.

@iblis17

This comment has been minimized.

Show comment
Hide comment
@iblis17

iblis17 Oct 20, 2015

Contributor

Hi @jreback !
Hmm... actually the problematic code do not come from cython. Those code from cython inline always carry static.
I will make some explanation.

E.g. consider pandas/src/numpy_helper.h

PANDAS_INLINE npy_int64                                                            
get_nat(void) {                                                                    
  return NPY_MIN_INT64;                                                            
}

Note that there is a macro called PANDAS_INLINE.

This macro is defined in pandas/src/helper.h.
By default, this macro will be expanded as __inline__, because clang will define __GUNC__ as well.

Those code are not generated by cython.

IMO, there are two solution for this inline problem. (To be honest, many project still have this problem on FreeBSD, e.g. our depend: ujson)

  1. Simply using the CFLAGS -fgnu89-inline
  2. Make sure our(also our depends) macro PANDAS_INLINE consider following situation:
  • __GNUC_GNU_INLINE__ (this is the behavior of --std=gnu89) define as __inline__
  • __GNUC_STDC_INLINE__ (--std=c99 or later) define as static inline
Contributor

iblis17 commented Oct 20, 2015

Hi @jreback !
Hmm... actually the problematic code do not come from cython. Those code from cython inline always carry static.
I will make some explanation.

E.g. consider pandas/src/numpy_helper.h

PANDAS_INLINE npy_int64                                                            
get_nat(void) {                                                                    
  return NPY_MIN_INT64;                                                            
}

Note that there is a macro called PANDAS_INLINE.

This macro is defined in pandas/src/helper.h.
By default, this macro will be expanded as __inline__, because clang will define __GUNC__ as well.

Those code are not generated by cython.

IMO, there are two solution for this inline problem. (To be honest, many project still have this problem on FreeBSD, e.g. our depend: ujson)

  1. Simply using the CFLAGS -fgnu89-inline
  2. Make sure our(also our depends) macro PANDAS_INLINE consider following situation:
  • __GNUC_GNU_INLINE__ (this is the behavior of --std=gnu89) define as __inline__
  • __GNUC_STDC_INLINE__ (--std=c99 or later) define as static inline
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 20, 2015

Contributor

ok I see

well happy to take a patch that ifdefs things and add platform based CFLAGS is ok too (would add in setup.py)

Contributor

jreback commented Oct 20, 2015

ok I see

well happy to take a patch that ifdefs things and add platform based CFLAGS is ok too (would add in setup.py)

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 20, 2015

Contributor

btw tha pandas.json IS ujson (with a layer on top for pandas use)

Contributor

jreback commented Oct 20, 2015

btw tha pandas.json IS ujson (with a layer on top for pandas use)

@iblis17

This comment has been minimized.

Show comment
Hide comment
@iblis17

iblis17 Oct 20, 2015

Contributor

ok, i will send a PR later 😄
It's AM 3:00 here .... zzzzz

Contributor

iblis17 commented Oct 20, 2015

ok, i will send a PR later 😄
It's AM 3:00 here .... zzzzz

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 12, 2015

I know this issue is closed, but just for anyone interested: The problem also exists with Python 3.4.3 on FreeBSD 10.2 (freshly installed from CD) at least on my machine.

ghost commented Nov 12, 2015

I know this issue is closed, but just for anyone interested: The problem also exists with Python 3.4.3 on FreeBSD 10.2 (freshly installed from CD) at least on my machine.

@iblis17

This comment has been minimized.

Show comment
Hide comment
@iblis17

iblis17 Nov 12, 2015

Contributor

Hi @usrbinerl !
Hmm... have you tried the patch in #11398?

Contributor

iblis17 commented Nov 12, 2015

Hi @usrbinerl !
Hmm... have you tried the patch in #11398?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 12, 2015

Yes, I just did, but didn't resolve the issue. But take that with a grain of salt. I lack experience with compiling python from source, so I may have done something wrong.
So far I did not had to do that with python3 and my code works fine on Ubuntu 15.04/15.10 with pip3 installed pandas-package.

ghost commented Nov 12, 2015

Yes, I just did, but didn't resolve the issue. But take that with a grain of salt. I lack experience with compiling python from source, so I may have done something wrong.
So far I did not had to do that with python3 and my code works fine on Ubuntu 15.04/15.10 with pip3 installed pandas-package.

@iblis17

This comment has been minimized.

Show comment
Hide comment
@iblis17

iblis17 Nov 12, 2015

Contributor

@usrbinerl Here is my workflow, FYI :)

pyvenv-3.4 venv
source venv/bin/activate
pip install cython                          
pip install git+https://github.com/pydata/pandas.git

Note that cython first

Then, pandas works fine for me

(venv) > python
Python 3.4.3 (default, May 26 2015, 17:46:50) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> 
Contributor

iblis17 commented Nov 12, 2015

@usrbinerl Here is my workflow, FYI :)

pyvenv-3.4 venv
source venv/bin/activate
pip install cython                          
pip install git+https://github.com/pydata/pandas.git

Note that cython first

Then, pandas works fine for me

(venv) > python
Python 3.4.3 (default, May 26 2015, 17:46:50) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> 
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 12, 2015

Thanks. I'll try with your tips again this afternoon.

ghost commented Nov 12, 2015

Thanks. I'll try with your tips again this afternoon.

@jvkersch

This comment has been minimized.

Show comment
Hide comment
@jvkersch

jvkersch Jan 24, 2016

Contributor

@meeb @iblis17 @usrbinerl I encountered a similar problem on Mac OS (which also uses Clang) when building a Pandas with debug symbols. I tackled this by adding the static keyword to inline function declarations and in the process I removed the -fgnu89-inline flag that was added in 0ec93fb. Would you mind giving my branch #12123 a spin on your FreeBSD systems? Thanks!

Contributor

jvkersch commented Jan 24, 2016

@meeb @iblis17 @usrbinerl I encountered a similar problem on Mac OS (which also uses Clang) when building a Pandas with debug symbols. I tackled this by adding the static keyword to inline function declarations and in the process I removed the -fgnu89-inline flag that was added in 0ec93fb. Would you mind giving my branch #12123 a spin on your FreeBSD systems? Thanks!

@iblis17

This comment has been minimized.

Show comment
Hide comment
@iblis17

iblis17 Jan 24, 2016

Contributor

@jvkersch Nice work!
It works fine on my bsd box.

Contributor

iblis17 commented Jan 24, 2016

@jvkersch Nice work!
It works fine on my bsd box.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jan 24, 2016

Works for me too now on FreeBSD. Thanks for your help.

ghost commented Jan 24, 2016

Works for me too now on FreeBSD. Thanks for your help.

jreback added a commit that referenced this issue Jan 24, 2016

DEV: Add static modifier to inline declarations.
It is currently impossible to build a debug version of Pandas on Mac
OS, since some of the C function declarations are declared as
`inline`, and compiling in debug mode turns off optimizations and
hence inlining. While this could be solved by adding a compile
argument to `setup.py` to use the C89 behavior for inline as in
#10510, I believe an ultimately cleaner solution may consist of
declaring inlined functions as `static`, so that the code is properly
C99-compliant.    For the implementation, I added the `static` keyword
to the definitions of `PANDAS_INLINE` and `P_INLINE` (and removed
duplicate occurrences of `static` from some function declarations), so
that future implementations that use those defines are automatically
compliant. This is slightly different from e.g. Cython, which defines
`CYTHON_INLINE` as `inline` and declares everything as `static
CYTHON_INLINE`.

Author: Joris Vankerschaver <jvankerschaver@enthought.com>

Closes #12123 from jvkersch/dev/static-inline and squashes the following commits:

6a225e7 [Joris Vankerschaver] MAINT: Remove compile flag that is no longer needed.
a5138f0 [Joris Vankerschaver] DEV: Add static modifier to inline declarations.
@bashtage

This comment has been minimized.

Show comment
Hide comment
@bashtage

bashtage Jan 24, 2016

Contributor

You might also want to try adding -std=c99 to the build, which uses modern, standard inlining behavior.

Contributor

bashtage commented Jan 24, 2016

You might also want to try adding -std=c99 to the build, which uses modern, standard inlining behavior.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Jan 24, 2016

Contributor

@bashtage is that a more general comment or only for FreeBSD?

Contributor

jreback commented Jan 24, 2016

@bashtage is that a more general comment or only for FreeBSD?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment