@pyimport numpy fails

[ufechner7]

Hello,
I use Ubuntu 12.04, 64 bit. I installed numpy 1.7.1 with pip.

Importing numpy fails:

julia> using PyCall

julia> @pyimport numpy
ERROR: PyError (PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name scimath',)
File "/usr/local/lib/python2.7/dist-packages/numpy/init.py", line 153, in
from . import add_newdocs
File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in
from numpy.lib import add_newdoc
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/init.py", line 17, in
from . import scimath as emath

in pyerr_check at /home/ufechner/.julia/PyCall/src/exception.jl:58
in pyimport at /home/ufechner/.julia/PyCall/src/PyCall.jl:85

Importing numpy from python works fine, importing other packages from julia
also works.

Any idea?

Uwe Fechner

[stevengj]

Do you have some kind of virtualenv setup, or multiple versions of Python?

[ufechner7]

No virtualenv, only one version of python on the harddrive. Any suggestions what I could try?

[stevengj]

Try

using PyCall
unshift!(PyVector(pyimport("sys")["path"]), "")
@pyimport numpy

to see if this is related to #48.

[ufechner7]

I tried this, but no change. I also tried numpy 1.8.0, same behaviour.

[ufechner7]

Hello,
the combination of numpy 1.6.1 and matplotlib 1.1 works now.
pip install -U numpy=="1.6.1"
pip install -U matplotlib=="1.1.1"

I still did not find the reason for this problem, because on other machines the newest version of numpy and matplotlib are working fine. Perhaps I should mention that this laptop uses a haswell CPU.

[ufechner7]

I tried using a clean virtual environment:

sudo pip install virtualenv
sudo apt-get install tk-dev

virtualenv /home/$USER/python-julia --no-site-packages

source /home/ufechner/python-julia/bin/activate
pip install numpy=="1.6.1"
pip install matplotlib=="1.1.1"

This is working, but any newer version of numpy does not work.

So yes, I have a workaround but no, the problem itself is not solved yet.

[arokem]

Just wanted to add that I am seeing the same with numpy 1.8. unshifting sys.path doesn't seem to help. Thanks for your work on PyCall!

[Jutho]

Same error here, on my desktop at work, which has the Canopy python distribution with latest version of all packages (numpy, scipy, matplotlib, ...). On my laptop I have installed python via homebrew and everything is working fine.

[stevengj]

PyCall doesn't work with Canopy (#42), so that's probably a separate issue.

[Jutho]

I noticed this afterwards; my apologies for polluting this issue. However, on my specific machine, the rest of Canopy's python seemed to do ok, it was only this exact same error "ImportError('cannot import name scimath',)" that was making PyPlot fail. I have removed Canopy and switched to Homebrew and everything works, as expected.

[stevengj]

If you google cannot import name scimath, you find lots of people encountering similar problems in random circumstances, but no systematic solution as far as I can find. Obviously something is missing from some path, but I'm not sure what.

[ufechner7]

A workaround for the google app engine is described here:
http://linuxonmac.wordpress.com/2014/03/22/google-app-engine-cannot-import-scimath/

I do not know how to apply this for an Ubuntu/ pip based environment, though.

[cberzan]

@stevengj The issue appears to be about circular imports. numpy.__init__ imports numpy.lib imports numpy.lib.scimath imports numpy.core.numeric (which appears to run numpy.__init__ again).

Here is how circular imports are supposed to work: http://stackoverflow.com/a/744403/744071

To demonstrate the issue: Make a virtualenv and install numpy==1.9.1. Then go to where it was installed, and add the following line near the top of numpy/__init__.py: print("numpy.__init__ running"). Add a similar line to the files numpy/lib/__init__.py and numpy/lib/scimath.py. Now, if you run this from python:

>>> import numpy
numpy.__init__ running
numpy.lib.__init__ running
numpy.lib.scimath running

That looks good. Now run the same from julia (v0.3.2) using PyCall (version 0.7.3):

julia> using PyCall
julia> @pyimport numpy
numpy.__init__ running
numpy.lib.__init__ running
numpy.lib.scimath running
numpy.__init__ running
numpy.lib.__init__ running
ERROR: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name scimath',)
  File "/home/cberzan/.virtualenvs/captcha/local/lib/python2.7/site-packages/numpy/__init__.py", line 172, in <module>
    from . import add_newdocs
  File "/home/cberzan/.virtualenvs/captcha/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/cberzan/.virtualenvs/captcha/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 19, in <module>
    from . import scimath as emath

 in pyerr_check at /home/cberzan/.julia/v0.3/PyCall/src/exception.jl:58
 in pyimport at /home/cberzan/.julia/v0.3/PyCall/src/PyCall.jl:91

As you can see, the line numpy.__init__ running is printed twice, which shows that the circular imports aren't working properly.

I don't know enough about Python's import mechanism to debug this further, but I hope it leads you to a solution.

Update: Things get weirder. Consider this:

a.py:

print "a.py top"
import b
print "a.py bottom"

b.py:

print "b.py top"
import a
print "b.py bottom"

If I run this from the python shell, things work as expected:

>>> import a
a.py top
b.py top
b.py bottom
a.py bottom

But if I run it as a script, it does something else (I don't understand why):

$ python a.py
a.py top
b.py top
a.py top
a.py bottom
b.py bottom
a.py bottom

And Julia / PyCall behaves like the former:

$ PYTHONPATH="." julia
julia> using PyCall
julia> @pyimport a
a.py top
b.py top
b.py bottom
a.py bottom

So in this case Julia / PyCall seems to handle the circular imports properly.

[stevengj]

My first guess was that you have multiple versions of numpy etc. installed and it is getting confused about which one to import... but that doesn't explain the multiple print lines. I dunno...

[stevengj]

The thing is, it is not PyCall that is "handling" the circular imports. PyCall just calls PyImport_ImportModule from the CPython API, and the rest is done by CPython (the same as in python). I can only think that is some path or environment variable that is making Python behave differently in PyCall.

[stevengj]

Can you try __import__ in Python instead (see here)? I wonder if it is something to do with import modifying the global Python namespace.

[cberzan]

@stevengj I found the cause of the problem.

First I realized that sys.modules['numpy'] gets overwritten:

julia> using PyCall
julia> @pyimport numpy
numpy.__init__ running
id(sys.modules) = 89034704
id(sys.modules['numpy']) = 161528304
numpy.__init__ running
id(sys.modules) = 89034704
id(sys.modules['numpy']) = 177135864

Then I realized that numpy gets imported by PyCall even when I don't ask for it:

julia> using PyCall
julia> @pyimport datetime
numpy.__init__ running
...

The culprit is the line pyimport("numpy")["ufunc"] in the function pyinitialize. It appears to import numpy before the python interpreter is fully initialized (?), causing this weird issue. I don't know how to fix it properly, but as a workaround, I can just comment out the ufuncType global from pyinitialize.

BTW, if anyone is getting ImportError("...../lib/julia/libgfortran.so.3: version 'GFORTRAN_1.4' not found (required by /usr/lib/liblapack.so.3gf)",), a solution that worked for me is to invoke julia with LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgfortran.so.3 julia. This loads the system libgfortran instead of the one bundled with julia.

Yay, I can finally use PyPlot from julia!

[stevengj]

Thanks for looking into this. (The need for the numpy ufunc thing will go away in Julia 0.4; see #101.)

However, I don't understand the problem, since Python has been initialized when the pyimport("numpy")["ufunc"] is called. (Otherwise PyImport_ImportModule wouldn't work.)

Could it possibly be that Python is reloading the numpy module because it got garbage-collected? i.e. pyimport("numpy")["ufunc"] loads the module and stores a reference in a temporary object, which Julia will eventually free. When Julia frees the object, the corresponding module reference in Python is decremented, which I suppose tells Python it is okay to unload the module. Then later on when you import numpy again, Python reloads it, and somehow this goes horribly wrong? If this is the problem, a simple workaround would be to modify pyinitialize to store the numpy reference in a global constant so that it is never garbage-collected:

global const numpy_module = try
    pyimport("numpy")
catch
    PyObject() # NumPy not available                                    
end
global const ufuncType = numpy_module.o == C_NULL ? PyObject() : numpy_module["ufunc"]

(Although I don't really believe this explanation — because Julia keeps a reference to numpy.ufunc in ufuncType, that alone should be enough to prevent Python from unloading the module.)

You could also just move the ufuncType initialization further down in pyinitialize in case the numpy initialization is somehow affected by one of the other lines in pyinitialize.

(I'm not sure if anything can be done about the libgfortran thing; it's always a problem to have multiple incompatible Fortran compilers on your system; you have to be very careful about consistently linking the same one.)

[cberzan]

@stevengj I take back everything I said; it was a red herring. I just realized that libgfortran was at fault all along. If I LD_PRELOAD the correct version as I mentioned above, then @pyimport numpy works fine with an unmodified version of PyCall (i.e. without changing anything about ufuncType). I guess the lesson here is that if there's a problem with a .so file, python's import mechanism fails horribly and with completely misleading error messages...

[mlubin]

Also ran into this, LD_PRELOAD trick didn't fix it.

[jmxpearson]

Just to add to the documentation of this:

As of yesterday, running 04.-rc1 on my Ubuntu 14.10 (running Anaconda), I am having this issue. My particular version is that I cannot pyimport("numpy") (or anything that depends on NumPy). The LD_PRELOAD doesn't work for me either. No problem on my Mac (running latest OS X) with this.

PYTHONHOME is set correctly as checked by ccalling into libpython.

Note that if I comment out the scimath import in numpy/lib/__init__.py, a similar error is raised in the next line when polynomial attempts to import from numpy.core. Anything I try to import without these relative package imports seems to be fine. Any import in Python/IPython also works.

[jmxpearson]

I will also note that a simple C version

#include <Python.h>

int
main(int argc, char *argv[])
{
    PyObject *pModule;
    char *pName = "numpy";

    Py_Initialize();

    pModule = PyImport_ImportModule(pName);

    if (pModule != NULL) {
        printf("Success!\n");
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", pName);
        return 1;
    }
    Py_Finalize();

    return 0;
}

Runs successfully.

[holocronweaver]

Having same issue as @jmxpearson. Ubuntu 15.04 + pip2 installed libraries + Julia 0.4-rc1.

[jmxpearson]

Just for reference, I created a minimal Julia example that reproduces the issue.

$ julia pyimport_test.jl json

works, while

$ julia pyimport_test.jl numpy

fails.

What could it be here, except ccall?

############## pyimport_test.jl ######################################
const stub = homedir() * "/anaconda"
const libname = @osx? "libpython2.7.dylib" : "libpython2.7.so"
const libpython = stub * "/lib/" * libname

################# from PyCall.jl #####################################
immutable PyObject_struct
    ob_refcnt::Int
    ob_type::Ptr{Void}
end

typealias PyPtr Ptr{PyObject_struct} # type for PythonObject* in ccall

macro pysym(func)
    :(($func, libpython))
end
################# from PyCall.jl #####################################

# now, do some ccalling to initialize

ccall((@pysym :Py_InitializeEx), Void, (Cint,), 0)

modname = ARGS[1] 
println("About to try loading library $modname")

modptr = ccall((@pysym :PyImport_ImportModule), PyPtr, (Ptr{Uint8},), 
    bytestring(modname))

if modptr == C_NULL
    println("Could not load library.")
else
    println("Success!")
end

ccall((@pysym :Py_Finalize), Void, ())

[stevengj]

@jmxpearson, I really think it is some Python path thingy that is missing; it sounds like ccall is calling libpython successfully, it is PyImport_ImportModule that is not succeeding. See, for example, the persistent problems with Canopy (#42). There is some mysterious path setting that seems to be needed in some Python distros, that I have yet to figure out.

(It sucks that the hardest thing about PyCall has been figuring out how to initialize Python correctly.)

[jmxpearson]

Agree. That really does seem like the most likely solution, except that PyCall is correctly inferring the location of libpython on my system, correctly setting PYTHONHOME, and having run the code more or less line by line, I can't find the failure until you try to import numpy for loading ufunc.

At every point where I have been able to interrogate path settings, they've been the same between my mac (working) and linux (not).

[ihnorton]

You could try setting LD_DEBUG=all before running the Julia script. This will be very noisy, but should tell you if there is some shared library loading error while attempting to import numpy. Call LD_DEBUG=help cat to see all of the options. (You may want to print some separator output at the start of your script so you can find where the Julia initialization ends)

[maximsch2]

@stevengj,
It looks like Julia 0.4 from https://launchpad.net/~staticfloat/+archive/ubuntu/juliareleases is using libopenblas.so on Ubuntu 14.04 (at least that's what it is installing and I'm getting the same problem). Building Julia from source works just fine though

[stevengj]

@staticfloat, is there anything that can be done about the Ubuntu package?

[staticfloat]

@stevengj I've tried to follow the conversation here but I'm having a little trouble. What is the fix that is needed? When running on Ubuntu 14.04, with the following packages installed:

$ dpkg -s julia | grep Version
Version: 0.4.1-trusty1
$ dpkg -s libopenblas-base | grep Version
Version: 0.2.10.1-trusty2

Everything works just fine, as far as I can tell:

$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.1 (2015-11-08 10:33 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release
|__/                   |  x86_64-linux-gnu

julia> using PyCall

julia> @pyimport numpy

julia>

[staticfloat]

A little bit more information:

julia> filter(x -> contains(x,"blas"), Libdl.dllist())
1-element Array{AbstractString,1}:
 "/usr/lib/libopenblas.so.0"

julia> filter(x -> contains(x,"numpy"), Libdl.dllist())
8-element Array{AbstractString,1}:
 "/usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so"
 "/usr/local/lib/python2.7/dist-packages/numpy/core/umath.so"
 "/usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so"
 "/usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so"
 "/usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so"
 "/usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so"
 "/usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so"
 "/usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so"

[stevengj]

@staticfloat, it seems like you are not using the Ubuntu numpy package, which would normally be in /usr/lib rather than /usr/local/lib?

[staticfloat]

Ah, you are correct, I installed numpy from source. I'll try installing the package and see what happens.

[staticfloat]

Even after installing the debian package and removing everything in /usr/local/lib/python2.7/dist-packages/numpy things work the same as before.

[maximsch2]

I'm using Python from Anaconda if that matters.

[maximsch2]

OK, we've talked a bit more with @staticfloat and here is what we found:

1. Julia on Ubuntu is using system libopenblas (not renamed), while loading extra symbols from liblapack.so (LAPACK is not built in OpenBlas)
2. Latest version of Numpy in Anaconda loads libopenblas (provided by Anaconda as well) and expects to find LAPACK symbols there.

So when I try to import Anaconda-provided version of NumPy through PyCall, it fails to find needed LAPACK symbol ("zgelsd_") because libopenblas is already loaded (Ubuntu's version) and it doesn't export symbols that NumPy expects.

Solutions:

1. Using Julia built from source works, because then libopenblas is renamed.
2. Doing LD_PRELOAD=anaconda/lib/libopenblas.so works as well (Anaconda's version of OpenBlas is a superset of system version)
3. Interestingly, this thing also works:

Libdl.dlopen("/usr/lib/liblapack.so.3", Libdl.RTLD_GLOBAL)

using PyCall
@pyimport numpy

This way, NumPy is loading needed LAPACK symbols from Ubuntu's liblapack.so (that is already loaded by Julia btw, but for some reason symbols are not available to NumPy without that extra Libdl.dlopen)

[staticfloat]

@stevengj Do you think there is a way we can auto-detect this condition and load liblapack into the global namespace automatically? Alternatively, should we be loading liblapack into the global namespace automatically anyway? I believe this is the default on OSX but perhaps not on Linux.

[stevengj]

@staticfloat, on most systems the Julia binaries load the ILP64 libopenblas64_.so and all the symbols are suffixed with 64_, so they are not visible at all to other libraries like numpy that want to link the ordinary LP64 blas/lapack.

The problems arise when both numpy and julia both want to link to a library called libopenblas.so, but numpy is expecting different library contents.

[staticfloat]

That is one problem. This problem is slightly different.

OpenBLAS has an option to statically build LAPACK and embed LAPACK within it; thereby creating a single .so file (libopenblas.so) that contains both the BLAS and LAPACK symbols. Anaconda's libopenblas.so does this, the default Ubuntu/Debian packages do not so as to maintain a weak linking between BLAS and LAPACK. (Apparently, in ages long lost to the mists of time, there were developers that wanted to experiment with different pairings of BLAS and LAPACK libraries). None of this is a problem until Julia attempts to load numpy, which has shared libraries that have an explicit dependence on libopenblas but not liblapack. Because a library called libopenblas.so is already linked into the current process, we don't attempt to dlopen() Anaconda's libopenblas.so. This is not a problem in general however, because as long as there is not a mixup of ILP64 symbols we shouldn't have any lacking symbols. (There is no mixup in this case because both the ubuntu libopenblas package and Anaconda's libopenblas are non-ILP64)

The issue arises because numpy expects to get LAPACK symbols from libopenblas.so; not that it is explicitly searching for the symbols in that particular file, but that it does not declare a dependency on liblapack.so. I originally thought this would not be a problem, as Julia already has liblapack.so mapped into its address space, so searching for symbols like zgelsd_ (the actual culprit in @maximsch2's case above) should have been resolvable, but it turns out that liblapack.so is loaded without the RTLD_GLOBAL flag by default, thereby limiting its visibility to external libraries. I'm not sure about the rationale behind this design, and I get the sinking feeling that it is platform-dependent (e.g. I think this would not be the case on OS X) but regardless, the fact that simply explicitly loading liblapack.so into the global namespace (Which is what @maximsch2's dlopen() above does) works shows that a "reasonable" fix should exist.

[tkelman]

Back in the day Atlas etc didn't contain any lapack implementation pieces, so you could keep the netlib reference lapack and just swap out libblas dynamically and still get a speedup. MKL typically put blas and lapack into the same library but is not open source so linux distributions didn't really accomodate it here. Open source implementations that follow the mkl organization of having everything in the same library are a bit newer and distros haven't really caught up yet.

Always dlopening liblapack on linux, or maybe only when lapack_name doesn't match blas_name, would make sense here. SCS.jl has been doing so.

[staticfloat]

I guess my question is why isn't liblapack opened with RTLD_GLOBAL in the first place? What is the use case for that state of affairs?

[tkelman]

Guessing, but probably because the name/ILP64 conflicts used to be an issue, and it hasn't been adjusted since that was rectified?

[tkelman]

I don't know what the numbers look like on openblas' lapack vs netlib lapack linked against openblas for just libblas, but we could raise this with anaconda and see whether it would make sense for them to build blas-only openblas the way ubuntu is doing?

[staticfloat]

I don't think there's any reason to do that; in fact, we build openblas-with-lapack when we do a from-source build.

[tkelman]

Right, but we also do ILP64 and tend to err on the side of performance over compatibility in source builds and tarball binaries. Anaconda doesn't do ILP64 and may care more strongly about compatibility than we do.

[maximsch2]

One solution here would be to get Julia into Anaconda and then just tell people like me that they should install Julia from Anaconda as well if they want it to play nicely with Anaconda's Python. Right now I can see Julia there in some 3rd party channels, but none of them has 0.4.1 or even a release version of 0.4.0.

[tkelman]

Maybe. There are several ways of going about that, some of which would be much more challenging than others. Another option is for you to use the tarball binaries of Julia rather than the Ubuntu PPA, which would be more isolated in terms of library naming.

[maximsch2]

Right. Right now I've switched to building Julia from source, so I don't really have an issue on my end. This is more about what to do when someone else runs into the same setup (Julia package from PPA + Anaconda)

[stevengj]

If we are linking to the system LP64 blas and lapack, I think we should load them with RTLD_GLOBAL. If we are linking to our own ILP64 openblas+lapack, it doesn't matter because the symbols are suffixed with 64_, so we might as well load them with RTLD_GLOBAL as well.

[stevengj]

@staticfloat, would it be enough to add:

    if startswith(basename(Base.liblapack_name), "liblapack")
        Libdl.dlopen(Base.liblapack_name, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)
    end

to PyCall's __init__ function?

[staticfloat]

@maximsch2 could you try putting the following snippet just below this line in PyCall/src/pyinit.jl:

    if basename(Base.liblapack_name) != basename(Base.libblas_name)
        Libdl.dlopen(Base.liblapack_name, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)
    end

Let's see if that works on your machine. @stevengj just to make this a little more general, I would do this anytime liblapack_name and libblas_name are unequal.