Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

numpy support broken on arch-linux #209

Closed
luca-penasa opened this issue Jun 5, 2018 · 15 comments
Closed

numpy support broken on arch-linux #209

luca-penasa opened this issue Jun 5, 2018 · 15 comments

Comments

@luca-penasa
Copy link

luca-penasa commented Jun 5, 2018

This seems to be a bug affecting several distros. I testes on Manjaro (fresh and updated live) and Ubuntu 18.04 (docker).

Here is the code for reproducing the bug:
https://github.com/luca-penasa/boost-numpy-minimal-bug

it compiles a python module using boost-python and boost-python-numpy
the line of code

np::initialize();

causes an error when trying to import the module in python (here from ipython but it is the same from pure python):

Python 3.6.5 (default, May 11 2018, 04:00:52) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import hello
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: FATAL: module compiled as little endian, but detected different endianness at runtime
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.umath failed to import
---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-1-f81fb083bdeb> in <module>()
----> 1 import hello

SystemError: initialization of hello raised unreported exception

In [2]: 

Could anybody have a look at this? this is making a lot of my code useless!

@stefanseefeld
Copy link
Member

stefanseefeld commented Jun 5, 2018

As the error indicates, it looks like the boost_numpy3 module was compiled for a different architecture (endianness). Please report this to the packager, i.e. the distro you are using.

@luca-penasa
Copy link
Author

luca-penasa commented Jun 5, 2018

thank you for the reply @stefanseefeld
That would also mean that both in the case of ubuntu and arch they are compiling boost with the wrong endianess... It sounds unlikely to me. what do you think? I cannot be sure the problem lies within boost-python-numpy but it seemed to me ther more likely. ... I've been also trying recompiling it several times with no success.

On what distro are you running it?
does the sample code compile (and run) for you?

I am hitting the same problem both on my workstation and on the laptop...

thanks

@luca-penasa
Copy link
Author

luca-penasa commented Jun 5, 2018

btw
i am on manjaro (arch) witht the following packages:
python 3.6.5
numpy 1.14.3
boost 1.67.0

I've been tring several versions of boost and numpy (trunk) with no success... any idea on how I could debug it?

@stefanseefeld
Copy link
Member

stefanseefeld commented Jun 5, 2018

I have been using Boost.Python (including NumPy) on different distros. Notably, we are running unit tests in a trusty container (see https://travis-ci.org/boostorg/python for details) without any issues.

But as I said, the Boost.Python project doesn't itself produce the binaries that are shipped in Linux distributions, so it doesn't control how they get built. I haven't heard of the problem you are reporting before, so I'm not sure where to look for the bug. But the nature of the problem suggests a build (or even runtime !) configuration issue. In either case, it's outside my control.

You mention you have been building Boost.Python yourself, and are observing the problem. How exactly are you building it ? Can you provide the build logs ? And what Boost.Python version are we talking about ?

@luca-penasa
Copy link
Author

luca-penasa commented Jun 5, 2018

Thanks btw for your time. I can understand it does not look a problem related to boost python (which btw works fine).

I've been building several versions of boost (comprising ::python and :.numpy) through the package builder of arch (makepkg) which uses a PKGBUILD file describing the compilation process. (You might be right if this problem comes out during compile the PKGBUILD might just replicate the error).

I got this issue since a recent update of the system so it might be related to new versions of some packages (python? numpy?). unfortunately it is not so easy to go back to previous versions.

Trusty uses also "old" packages that might be the reason for which it works fine on those platforms, as I said I got this error since a couple of days after an update. the same on my laptop where I am running arch too.

I can provide the commands needed to replicate the error in docker with ubuntu if needed.

I will try to recompile boost python by itself and see what happens.

Is there anything I could do to better constraint the origin of this issue? any idea?

@luca-penasa
Copy link
Author

luca-penasa commented Jun 5, 2018

here are the commands to reproduce this issue from within a docker container. If any of you could try to replicate this would be of great help to me, so I can understand if this is just my problem or not

run a fresh container:

docker run -ti ubuntu /bin/bash

and then from within it:

apt-get update
apt-get -y upgrade
apt-get -y install build-essential git cmake libboost-all-dev python3
git clone https://github.com/luca-penasa/boost-numpy-minimal-bug.git bnmb.git
cd bnmb.git/
mkdir build
cd build
cmake ..
make
python3 -c "import hello"

In this case the error is slightly different and is:

ImportError: numpy.core.multiarray failed to import
ImportError: numpy.core.umath failed to import
Traceback (most recent call last):
  File "<string>", line 1, in <module>
SystemError: initialization of hello raised unreported exception
root@3e3639f9d914:/bnmb.git/build# 

@luca-penasa
Copy link
Author

luca-penasa commented Jun 5, 2018

I confirm that compiling boost-python with faber works as expected and all test pass. Thus this problem likely raises from somewhere else.

I'll try to use this version of boost python instead than the official one but this raises the point that there might be some problem with the version distributed within boost

e.g. Arch compile boost/python from the package provided here:
https://sourceforge.net/projects/boost/

I don't know how the confluence of boost/python to boost works and how to get which version of boost/python is used there...

If I can help in any way please don't hesitate
many thanks.
luca

@stefanseefeld
Copy link
Member

stefanseefeld commented Jun 5, 2018

Good to see that you got things working ! I'm not sure what you are referring to as "distributed within boost". Boost (the org) typically only distributes source releases (at least on Linux), so what you may be thinking of as a "release" is actually produced by individual distributions (debian, ubuntu, fedora, etc.). This is why I suggested to submit the issue to them.

@Scimmia22
Copy link

Scimmia22 commented Jun 8, 2018

Note that "FATAL: module compiled as little endian, but detected different endianness at runtime" is happening on an x86 system, so it IS little endian. There's a problem with the detection.

@stefanseefeld
Copy link
Member

stefanseefeld commented Jun 8, 2018

I strongly suggest that you build and test Boost.Python using the standard procedures (faber for stand-alone builds and b2 for an integrated build). It seems from a previous comment that in this context all tests pass, i.e. the issue being reported here isn't seen.
If that's the case, you need to report this downstream, with whatever distro provides the binary packages that exhibit this behaviour.
I unfortunately don't have the bandwidth to support issues related to specific binary packages, if they don't also show up as genuine Boost.Python bugs.

@luca-penasa
Copy link
Author

luca-penasa commented Jun 11, 2018

BTW I've reported to arch. Seems that at the end it is an arch-related build problem.
https://bugs.archlinux.org/task/58881
on ubuntu the problem was a missing package...
many thanks

@luca-penasa luca-penasa changed the title numpy support broken numpy support broken on arch-linux Jun 11, 2018
@urbainvaes
Copy link

urbainvaes commented Jun 24, 2018

Same problem here! Is there a fix for arch, apart from downgrading boost?

@stefanseefeld
Copy link
Member

stefanseefeld commented Jun 24, 2018

@urbainvaes , I suggest you follow the above link to the arch issue tracker and ask there. This is not a Boost.Python issue.

@jwakely
Copy link
Contributor

jwakely commented Jun 29, 2018

@stefanseefeld please could you take a look at the Fedora packaging, because we use b2 and I don't think we do anything to affect the endianness and this has been reported to Fedora.

@stefanseefeld
Copy link
Member

stefanseefeld commented Jul 11, 2018

#218 is now merged into master, and thus will be part of 1.68.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants