Symbol not found: _floatify in pandas/lib.so on Mac 10.8 with numpy master #2188

Closed
cdeil opened this Issue Nov 7, 2012 · 31 comments

Projects

None yet

10 participants

@cdeil
cdeil commented Nov 7, 2012

After updating pandas I can't import it any more because the symbol _floatify is undefined in pandas/lib.so:

$ python -c 'import pandas'
dlopen(/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/__init__.py", line 10, in <module>
    import pandas.lib as lib
ImportError: dlopen(/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so

$ otool -L /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
$ nm /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so | grep floatify
                 U _floatify

The pandas build log doesn't show any problem with lib.so or floatify:
https://gist.github.com/4030153

My numpy install seems OK:
https://gist.github.com/4030177

@wesm
Member
wesm commented Nov 7, 2012

We had a little snafu in the setup.py file that was causing the Cython files to not automatically rebuild. The issue here was that floatify, a C function, had its signature changed but the declaration in the Cythonized C file had changed; hence the symbol lookup failure. At least that's what we think is going on. About to push some fixes (they should show up on here)-- do a clean build (setup.py clean, setup.py install) and let us know...

cc @yarikoptic

oops didn't mean to close issue

@wesm wesm closed this Nov 7, 2012
@wesm wesm reopened this Nov 7, 2012
@cdeil
cdeil commented Nov 7, 2012

With 5c945e1 I still see the same issue:
https://gist.github.com/4034766

@wesm
Member
wesm commented Nov 7, 2012

What Cython version are you on? seems to be some kind of name mangling problem with clang going on. floatify is declared in pandas/src/util.pxd and the C function itself is in pandas/src/numpy_helper.h. I have no ability to debug this myself

@cpcloud
Contributor
cpcloud commented Nov 7, 2012

Can you post the output of which python?

The reason I ask is because using the system Python can conflict with a Python from /opt/local/bin. I've definitely encountered a similar issue in the past.

@cdeil
cdeil commented Nov 8, 2012

My Python and Cython are the latest Macports versions, my numpy is from git master:

$ xcodebuild -version
Xcode 4.5.2
Build version 4G2008a
$ which python
/opt/local/bin/python
$ python --version
Python 2.7.3
$ python -c 'import cython; print cython.__file__'
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cython.pyc$ port installed py27-cython
The following ports are currently installed:
  py27-cython @0.17.1_0 (active)
$ python -c 'import numpy; print numpy.__file__; print numpy.__version__'
/Users/deil/Library/Python/2.7/lib/python/site-packages/numpy/__init__.pyc
1.8.0.dev-c5ccca9

Can you reproduce the issue with Cython 0.17.1?
If it's a name mangling problem, I could check which compilers were used for Python, numpy, pandas.
Or would it help if I post the generated c files that contain floatify?

@dnmiller
Contributor
dnmiller commented Nov 8, 2012

Have you tried reinstalling Cython in /Users/deil/Library/Python/2.7 instead of the MacPorts dir? It seems risky to be grabbing libraries from two site-packages directories.

@cdeil
cdeil commented Nov 8, 2012

@dnmiller Isn't having two site-packages normal?

There's packages that are hard to build with all their dependencies (like e.g. PyTables) which I install with a package manager (Macports on my Mac), and there's packages where I want the development version and in some cases even do development with, so these I install in my home folder.

I don't see how I could work with only one site-packages, because if I sudo install stuff myself in the site-packages of my package manager all hell breaks loose and I don't have the time to install / update all the packages I need myself.

I don't think it matters here where Cython is installed, rather I think it's a problem with the (Compiler, Python, Cython, numpy) combination I have.

@dnmiller
Contributor
dnmiller commented Nov 8, 2012

Well, you should definitely check out the virtual environments package. It solves a lot of the problems you just described. But I agree that's not really the root cause.

While I can't say this is the same problem you're having, I've had problems with MacPorts linking to it's own versions of libc, libc++, liblapack, etc., which are incompatible with other programs compiled outside of MacPorts, which will link to non-MacPorts versions of the same libraries. I've seen a very similar error that was due to a non-MacPorts Python talking to a MacPorts version of NumPy.

My hunch is that the MacPorts Cython behaves differently than a version you would install in your home dir. You might consider trying to build pandas and its dependencies (and Cython) entirely in /Users/deil/Library/Python/2.7.

@cpcloud
Contributor
cpcloud commented Nov 8, 2012

Have you tried manually importing another *.so file (just write a simple function in cython and try to import it)? If that doesn't work, it might hint at the problem being the clang linker flags, for which it seems there is a paucity of documentation, but I suspect that the fact that floatify is declared undefined in the binary might have something to do with your issue.

FYI: When I do nm lib.so | grep floatify I get T floatify as the output, but I'm running Arch Linux on a MacBook 4,2 so the code generation will probably be a bit different.

Also since you have the -g flag enabled when compiling the *.o files can you post the output of nm -l *.so | grep floatify? It should return the line number(s) of the function signature.

@wesm
Member
wesm commented Nov 9, 2012

@fonnesbeck : sir, king of bleeding edge OS X technology, can you confirm that you can build and import pandas git master with your OS 10.8 rig?

@brendam
Contributor
brendam commented Nov 9, 2012

I just tested in a new virtualenv on OS X 10.8.2 with latest xcode installed. I used 'pip install cython' to get cython. Everything else except wsgiref was installed by the pandas install.

clang --version 
Apple clang version 4.0 (tags/Apple/clang-421.0.60) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.2.0
Thread model: posix

xcodebuild -version
Xcode 4.5.2
Build version 4G2008a

pip freeze
Cython==0.17.1
numpy==1.6.2
pandas==0.9.1.dev-0230bbf
python-dateutil==2.1
pytz==2012h
six==1.2.0
wsgiref==0.1.2

I used
python setup.py clean
python setup.py build_ext --inplace
python setup.py install
nosetests pandas

"Symbol not found: _floatify"

using nm lib.so | grep floatify I get U _floatify

@wesm
Member
wesm commented Nov 9, 2012

Got a machine over here where I can reproduce this. Will try to figure out what's up

@cpcloud
Contributor
cpcloud commented Nov 9, 2012

I'm super curious to know what the heck the issue is!
On Nov 8, 2012 9:21 PM, "Brenda Moon" notifications@github.com wrote:

I just tested in a new virtualenv on OS X 10.8.2 with latest xcode
installed. I used 'pip install cython' to get cython. Everything else
except wsgiref was installed by the pandas install.

clang --version
Apple clang version 4.0 (tags/Apple/clang-421.0.60) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.2.0
Thread model: posix

xcodebuild -version
Xcode 4.5.2
Build version 4G2008a

pip freeze
Cython==0.17.1
numpy==1.6.2
pandas==0.9.1.dev-0230bbf
python-dateutil==2.1
pytz==2012h
six==1.2.0
wsgiref==0.1.2

I used
python setup.py clean
python setup.py build_ext --inplace
python setup.py install
nosetests pandas

"Symbol not found: _floatify"

using nm lib.so | grep floatify I get U _floatify


Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/2188#issuecomment-10213525.

@wesm
Member
wesm commented Nov 9, 2012

Well, it basically clang has stricter semantics about inline functions:

http://clang.llvm.org/compatibility.html#inline

Making floatify and its dependent functions not inline gets rid of the linker errors. Still a bit of a mystery because there are numerous other equivalently-declared inline functions that don't have this problem. It evidently has something to do with optimization passes and floatify not actually being inlined by the compiler

git master should build now. pls let us know if not

@wesm wesm closed this Nov 9, 2012
@cdeil
cdeil commented Nov 9, 2012

Works for me. Thanks!

@y-p
Contributor
y-p commented Nov 9, 2012

@changhiskhan , I have a feeling cc099d6 broke #2029, since I started
having trouble with tox a few days ago.
Since inlining seems to be the culprit here, could this be rolled back or could you give
some details on the issues it fixes?
I'd hate to dive back in to python packaging if I can bribe, beg or fight my way out of it.

@wesm
Member
wesm commented Nov 9, 2012

@y-p The problem is that I believe python setup.py cython had stopped working on systems with Cython installed.

@y-p
Contributor
y-p commented Nov 9, 2012

grrr. I'm a continent away, but you yet may hear my shrieks of pain...

@yarikoptic yarikoptic added a commit to neurodebian/pandas that referenced this issue Nov 15, 2012
@yarikoptic yarikoptic Merge commit 'v0.9.0-169-g5c945e1' into debian
* commit 'v0.9.0-169-g5c945e1':
  BUG: fill_value not being applied for mixed type frame #2191
  BUG: bug in setup.py with cython and sdist command class #2188
  BLD: c scope decl wrong
726a8b9
@locojay
locojay commented Nov 19, 2012

can't build on master 35dd6ac

mkvirtualenv pandasdev
pip freeze
Cython==0.17.1
distribute==0.6.28
ipython==0.13.1
matplotlib==1.2.0
numpy==1.6.2
pandas==0.9.2.dev-35dd6ac
python-dateutil==2.1
pytz==2012h
scipy==0.11.0
six==1.2.0
wsgiref==0.1.2

python setup.py build_ext --inplace
python setup.py install
nosetests pandas
(pandasdev)~/D/D/g/pandas git:master ❯❯❯ nosetests pandas
E
======================================================================
ERROR: Failure: ImportError (dlopen(/Users/locojay/Documents/Dev/github/pandas/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so
  Expected in: flat namespace
 in /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/locojay/Documents/Dev/github/pandas/pandas/__init__.py", line 10, in <module>
    import pandas.lib as lib
ImportError: dlopen(/Users/locojay/Documents/Dev/github/pandas/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so
  Expected in: flat namespace
 in /Users/locojay/Documents/Dev/github/pandas/pandas/lib.so

----------------------------------------------------------------------
Ran 1 test in 0.001s
@wesm
Member
wesm commented Nov 19, 2012

I broke master again. Can someone find out how to detect clang at compile time so I can conditionally take out the inline markers?

@wesm wesm reopened this Nov 19, 2012
@dnmiller
Contributor

Adding a "clang" option to the top of numpy_helper.h should work.
Source: http://clang.llvm.org/docs/LanguageExtensions.html#builtinmacros

Though according to the clang docs, this is official C99 behavior anyway, so maybe the behavior of "STDC_VERSION" in the definition of PANDAS_INLINE isn't correct?

@wesm
Member
wesm commented Nov 21, 2012

It should be working now, I'll need someone to verify because I don't have anything to test on right now

@locojay
locojay commented Nov 21, 2012

_floatify is fixed but


python setup.py clean
python setup.py install

In [1]: import pandas as pd
dlopen(/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _xstrtod
  Referenced from: /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-af55e7023913> in <module>()
----> 1 import pandas as pd

/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/__init__.py in <module>()
      8
      9 try:
---> 10     import pandas.lib as lib
     11 except Exception:  # pragma: no cover
     12     import sys

ImportError: dlopen(/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _xstrtod
  Referenced from: /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.9.2.dev_61766ec-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so

for the time being i just remove the PANDAS_INLINE from xstrod in parse_helper.h, end_line in tokenizer

@erg
Contributor
erg commented Nov 22, 2012

I have the same problem as locojay.

modern:~ erg$ uname -a
Darwin modern.local 12.2.1 Darwin Kernel Version 12.2.1: Thu Oct 18 12:13:47 PDT 2012; root:xnu-2050.20.9~1/RELEASE_X86_64 x86_64
@selasley

I added a forward declaration for end_line and removed P_INLINE from the forward declaration of xstrtod in tokenizer.c and now I can build and import pandas in python 2.7.3 in OS X 10.8.2.
Update: 0.9.2.dev-2600432 still has
// forward declaration
double P_INLINE xstrtod(const char *p...);
in tokenizer.c. I had to remove P_INLINE in order to build and import pandas

@milkypostman
Contributor

ditto

@wesm
Member
wesm commented Nov 25, 2012

can someone let me know if git master builds right now? I made some changes per above.

@milkypostman
Contributor

It compiles for me but when I try to import pandas I get:

Traceback (most recent call last):
  File "./testpandas.py", line 5, in <module>
    from pandas import *
  File "/Users/dcurtis/Dropbox/Development/pandas/pandas/__init__.py", line 30, in <module>
    from pandas.io.parsers import (read_csv, read_table, read_clipboard,
  File "/Users/dcurtis/Dropbox/Development/pandas/pandas/io/parsers.py", line 22, in <module>
    import pandas._parser as _parser
ImportError: dlopen(/Users/dcurtis/Dropbox/Development/pandas/pandas/_parser.so, 2): Symbol not found: _xstrtod
  Referenced from: /Users/dcurtis/Dropbox/Development/pandas/pandas/_parser.so
  Expected in: flat namespace
 in /Users/dcurtis/Dropbox/Development/pandas/pandas/_parser.so
@wesm
Member
wesm commented Nov 25, 2012

Could you try again? I made xstrtod not inline

@selasley

On Nov 25, 2012, at 3:18 PM, Wes McKinney notifications@github.com wrote:

Could you try again? I made xstrtod not inline

Version 0.9.2.dev-1d78502 builds and imports fine for me with python 2.7.3 under OS X 10.8.2, and all nosetests pass.

@wesm
Member
wesm commented Nov 25, 2012

Thanks. I'll go ahead and close this for now

@wesm wesm closed this Nov 25, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment