Arpack_jll artifacts unable to see OpenBLAS_jll artifacts for linking

[pengwyn]

I'm on Archlinux (using the distro-built package) and I'm currently trying to import the Arpack module but getting an error with a missing library:

ERROR: LoadError: InitError: could not load library "/home/pengwyn/.julia/artifacts/8a63e55e4ce37cc04c039ac5f48c2a8b8e9b3b37/lib/libarpack.so"
libopenblas64_.so.0: cannot open shared object file: No such file or directory

From what I understand, this is an artifact being loaded by Arpack_jll, which itself depends on OpenBLAS_jll which should include the dependency of libopenblas64_.so.0. However, the issue appears to be the system (i.e. the external dlopen, not the internal Julia dlopen) is looking for the library and can't find it.

I can confirm that a unspecific dlopen of libopenblas64_.so works (presumably because this is cached?) but trying to dlopen libopenblas64_.so.0 fails:

julia> using OpenBLAS_jll
julia> using Libdl
julia> dlopen("libopenblas64_.so")
Ptr{Nothing} @0x00005611d02e57b0
julia> dlopen("libopenblas64_.so.0")
ERROR: could not load library "libopenblas64_.so.0"
libopenblas64_.so.0: cannot open shared object file: No such file or directory

If I manually include the artifact lib directory into Libdl.DL_LOAD_PATH then I am able to do an in-Julia dlopen("libopenblas64_.so.0") successfully, but importing the Arpack_jll module still fails, I guess because this is part of the system library lookup.

The only way I've been able to make this work so far, is to make sure the artifact lib directory is in the LD_LIBRARY_PATH environment variable before starting Julia.

I can confirm that importing Arpack works using the official Julia binaries because libopenblas64_.so.0 is present in /usr/lib/julia, although that's not a solution for me as it breaks my code for interfacing with some other programs.

My version info (thanks to Tamas_Papp and Elrod on Discourse)

(@v1.4) pkg> st --manifest Arpack
Status `~/.julia/environments/v1.4/Manifest.toml`
  [7d9fca2a] Arpack v0.4.0
  [68821587] Arpack_jll v3.5.0+2
  [8f399da3] Libdl 
  [37e2e46d] LinearAlgebra 

julia> versioninfo()
Julia Version 1.4.0
Commit b8e9a9ecc6 (2020-03-21 16:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)

julia> using LinearAlgebra; BLAS.vendor()
:openblas

[pengwyn]

And after a little thought, maybe these are relevant too:

(@v1.4) pkg> st --manifest Arpack_jll
Status `~/.julia/environments/v1.4/Manifest.toml`
  [68821587] Arpack_jll v3.5.0+2
  [4536629a] OpenBLAS_jll v0.3.9+0
  [8f399da3] Libdl 
  [44cfe95a] Pkg 

(@v1.4) pkg> st --manifest OpenBLAS_jll
Status `~/.julia/environments/v1.4/Manifest.toml`
  [4536629a] OpenBLAS_jll v0.3.9+0
  [8f399da3] Libdl 
  [44cfe95a] Pkg 

[giordano]

What's the output of

julia> using Libdl

julia> filter!(p -> occursin("blas", p), Libdl.dllist())
1-element Array{String,1}:
 "/usr/bin/../lib/julia/libopenblas64_.so"

for you in a fresh Julia session?

[pengwyn]

Thanks for the quick reply. This is what I get:

julia> using Libdl
julia> filter(p -> occursin("blas", p), Libdl.dllist())
1-element Array{String,1}:
 "/usr/bin/../lib/libblas.so"

and after importing OpenBLAS:

julia> using OpenBLAS_jll
julia> filter(p -> occursin("blas", p), Libdl.dllist())
2-element Array{String,1}:
 "/usr/bin/../lib/libblas.so"
 "/home/pengwyn/.julia/artifacts/b3e6e77906cabeabfc0295792186e01f33613385/lib/libopenblas64_.so"

Obviously, this is using the system libblas.so for the Julia core.

[giordano]

How did you install julia? Where is that libblas coming from?

[giordano]

Ok, I read you're using archlinux, me too, but probably you installed the package https://www.archlinux.org/packages/community/x86_64/julia/? That one is terrible, it has tons of problems. Use instead the official binaries, I use this package: https://aur.archlinux.org/packages/julia-bin/

[pengwyn]

Ah I was afraid that would be the solution... 😢 The problem is the official binaries often have trouble interfacing with programs built with system libraries, e.g. I have to recompile a C python module against the julia libraries for it to be usable, as the official julia binaries only work with the local Conda install.

I will see if I can make OpenBLAS_jll add its library directory to the search path. Maybe that'll be a workaround...

[joachimneu]

For those who arrive at this bug report after google'ing the error, the following might provide a hacky band-aid fix: Creating a symlink to OpenBLAS_jll's libopenblas64_.so.0 in the same folder as Arpack_jll's libarpack.so seems to do the trick.

~/.julia/artifacts $ ls -lAh {Arpack_jll}/lib/libopenblas64_.so.0 
{...} {Arpack_jll}/lib/libopenblas64_.so.0 -> ../.././{OpenBLAS_jll}/lib/libopenblas64_.so.0

[pdebuyl]

Similar issue here.

On debian amd64 the following did it

ln -s /usr/lib/x86_64-linux-gnu/libopenblas.so .julia/packages/Arpack/zCmTA/deps/usr/lib/libopenblas64_.so.0

after installing gfortran-7 (which is available on debian 10 at least).

Thanks @joachimneu for the suggestion.

PS: running debian-provided julia 1.0.3

[giordano]

I'm confused: how it can be the same issue if you use julia 1.0.3? The artifacts system was introduced in julia 1.3

[pdebuyl]

As a end-user, I saw the error about libopenblas64_.so , ended up here, and adapted the solution. I don't even know about artifacts, I found the location of so files by typing

find .julia -name 'lib*.so'

and modified the fix accordingly.

[andreasnoack]

Is this still an issue?

[pengwyn]

Sorry for the incredibly late reply, but this issue hasn't affected me for a while. I tested just now by deleting the entire artifacts folder, reupdating and I was able to import Arpack fine.

I'm getting this same error. I suspect it might have to do with my installation using MKL rather than OpenBLAS. I'm using Manjaro (Arch-based Linux distro).