New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation error when Julia is built from source #5

Closed
yurivish opened this Issue Jun 24, 2018 · 119 comments

Comments

Projects
None yet
@yurivish
Copy link

yurivish commented Jun 24, 2018

I tried installing Arpack on 0.7 beta and saw this error trying to run eigs:

julia> A = Diagonal(1:4);
WARNING: Base.Diagonal is deprecated: it has been moved to the standard library package `LinearAlgebra`.
Add `using LinearAlgebra` to your imports.
 in module Main

julia> λ, ϕ = eigs(A, nev = 2);
ERROR: error compiling saupd: could not load library ""
dlopen(.dylib, 1): image not found
Stacktrace:
 [1] aupd_wrapper(::Any, ::getfield(Arpack, Symbol("#matvecA!#24")){LinearAlgebra.Diagonal{Float64,Array{Float64,1}}}, ::getfield(Arpack, Symbol("##18#25")), ::getfield(Arpack, Symbol("##19#26")), ::Int64, ::Bool, ::Bool, ::String, ::Int64, ::Int64, ::String, ::Float64, ::Int64, ::Int64, ::Array{Float64,1}) at /Users/yurivish/.julia/packages/Arpack/Rkbg/src/libarpack.jl:42
 [2] #_eigs#17(::Int64, ::Int64, ::Symbol, ::Float64, ::Int64, ::Nothing, ::Array{Float64,1}, ::Bool, ::Any, ::Any, ::Any) at /Users/yurivish/.julia/packages/Arpack/Rkbg/src/Arpack.jl:176
 [3] (::getfield(Arpack, Symbol("#kw##eigs")))(::NamedTuple{(:nev,),Tuple{Int64}}, ::typeof(eigs), ::LinearAlgebra.Diagonal{Int64,UnitRange{Int64}}) at ./none:0
 [4] top-level scope at none:0

To see if this was solved on master, I tried installing it. But this happens when I try to develop Arpack#master:

Error: Error building `Arpack`:
│ [ Info: Downloading https://github.com/JuliaLinearAlgebra/ArpackBuilder/releases/download/v3.5.0-0.2.20/ArpackBuilder.x86_64-apple-darwin14.tar.gz to /Users/yurivish/.julia/dev/Arpack/deps/usr/downloads/ArpackBuilder.x86_64-apple-darwin14.tar.gz...
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = macro expansion at OutputCollector.jl:63 [inlined]
│ └ @ Core OutputCollector.jl:63
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = wait(::OutputCollector) at OutputCollector.jl:158
│ └ @ BinaryProvider OutputCollector.jl:158
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = wait(::OutputCollector) at OutputCollector.jl:159
│ └ @ BinaryProvider OutputCollector.jl:159
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = wait(::OutputCollector) at OutputCollector.jl:163
│ └ @ BinaryProvider OutputCollector.jl:163
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/Users/yurivish/.julia/dev/Arpack/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error at ./error.jl:33 [inlined]
│  [2] #write_deps_file#134(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /Users/yurivish/.julia/packages/BinaryProvider/2Hlv/src/Products.jl:389
│  [3] write_deps_file(::String, ::Array{LibraryProduct,1}) at /Users/yurivish/.julia/packages/BinaryProvider/2Hlv/src/Products.jl:376
│  [4] top-level scope at none:0
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1075
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:393
│  [9] top-level scope at none:0
│ in expression starting at /Users/yurivish/.julia/dev/Arpack/deps/build.jl:40
julia> versioninfo()
Julia Version 0.7.0-beta.0
Commit f41b1ecaec (2018-06-24 01:32 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin16.7.0)
  CPU: Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)
Environment:
  JULIA_EDITOR = subl
@andreasnoack

This comment has been minimized.

Copy link
Member

andreasnoack commented Jun 25, 2018

Might be dup of #3. Did you build from source? What do you get from otool -L usr/lib/libopenblas64_.dylib?

@bicycle1885

This comment has been minimized.

Copy link

bicycle1885 commented Jun 25, 2018

I encountered the same problem on macOS, 0.7.0-beta.0.

~/v/julia ((v0.7.0-beta)|…) $ otool -L usr/lib/libopenblas64_.dylib
usr/lib/libopenblas64_.dylib:
        @rpath/libopenblas64_.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/local/opt/gcc/lib/gcc/6/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        /usr/local/opt/gcc/lib/gcc/6/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/local/lib/gcc/6/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Do we need to wait for the official binaries of Julia 0.7.0-beta?

@andreasnoack

This comment has been minimized.

Copy link
Member

andreasnoack commented Jun 25, 2018

Do we need to wait for the official binaries of Julia 0.7.0-beta?

Either that or build your OpenBLAS with GCC 7.

@bicycle1885

This comment has been minimized.

Copy link

bicycle1885 commented Jun 25, 2018

Thank you. I will try.

@yurivish

This comment has been minimized.

Copy link

yurivish commented Jun 25, 2018

@andreasnoack this was with a build from source. Thanks to the pointer to Viral's issue – I didn't see it since it was closed.

$ otool -L usr/lib/libopenblas64_.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/objdump: 'usr/lib/libopenblas64_.dylib': No such file or directory
@andreasnoack

This comment has been minimized.

Copy link
Member

andreasnoack commented Jun 26, 2018

@yurivish You'll need to adjust the path to match the usr directory within your root Julia directory.

@blakejohnson

This comment has been minimized.

Copy link

blakejohnson commented Jun 26, 2018

$ otool -L usr/lib/libopenblas64_.dylib 
usr/lib/libopenblas64_.dylib:
	@rpath/libopenblas64_.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/gcc/lib/gcc/7/libgfortran.4.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
	/usr/local/opt/gcc/lib/gcc/7/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
@bicycle1885

This comment has been minimized.

Copy link

bicycle1885 commented Jun 26, 2018

FYI, Julia 0.7-beta binary for macOS is now available (https://julialang.org/downloads/), which solves the problem.

@andreasnoack

This comment has been minimized.

Copy link
Member

andreasnoack commented Jun 26, 2018

@blakejohnson Things are more wrong than first anticipated. So the system libgfortran is apparently not in the RPATH. I'm able to get it working with DYLD_LIBRARY_PATH=/usr/local/opt/gcc@7/lib/gcc/7 julia-dev which is, of course, not a real solution but it identifies the issue here. Will have to hear @staticfloat what a proper solution would look like.

@RalphAS

This comment has been minimized.

Copy link

RalphAS commented Jun 27, 2018

Another possible workaround is to copy (or maybe hard link) libgfortran.so.4 from another Julia package (e.g. SpecialFunctions) into Arpack/XXXX/deps/usr/lib and then run pkg> build Arpack. This appears to be working for me, with Julia built from source on Linux.

This was referenced Jun 30, 2018

@maleadt

This comment has been minimized.

Copy link

maleadt commented Jul 2, 2018

Just for completeness (coming from #9), this obviously also occurs on Linux when not building with GCC 7. Doesn't seem easy to work around, with eg. Arch Linux' gcc7 package not providing libgfortran.4, and Debian Stable using GCC 6 / libgfortran.3 with no means to (safely) install GCC 7.

@andreasnoack andreasnoack changed the title Installation error on macOS Installation error when Julia is built from source Jul 2, 2018

@simonbyrne

This comment has been minimized.

Copy link
Collaborator

simonbyrne commented Jul 2, 2018

Until we fix this, could we throw a more informative error?

@juliohm

This comment has been minimized.

Copy link

juliohm commented Jul 11, 2018

The new tag of Arpack.jl now fails to compile on Julia binaries as well:

julia> using Arpack
[ Info: Precompiling module Arpack
ERROR: LoadError: No deps.jl file could be found. Please try running Pkg.build("Arpack").
Currently, the build command might fail when Julia has been built from source
and the recommendation is to use the official binaries from julialang.org.
For more info see https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5.

Julia version:

julia> versioninfo()
Julia Version 0.7.0-beta.182
Commit feaee9ebbc (2018-07-06 12:39 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 2
@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jul 11, 2018

Did you actually run Pkg.build("Arpack")?

@juliohm

This comment has been minimized.

Copy link

juliohm commented Jul 11, 2018

I did, there is no error message, and the build.log is empty.

@andreasnoack

This comment has been minimized.

Copy link
Member

andreasnoack commented Jul 11, 2018

It the test for the deps.jl file that is wrong. See #22

@sbromberger

This comment has been minimized.

Copy link

sbromberger commented Jul 13, 2018

This may be stating the obvious, but the problem still exists with 1.4 on a Julia built from source (which is the only way I can do it right now). Is there a workaround for those of us in this unfortunate position?

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jul 13, 2018

You can try downloading the official Julia binaries, extracting the libgfortran.so.4 file, and putting it into your from-source Julia installation's lib directory. That should allow your arpack to find libgfortran.so.4 and get loaded; however all bets are off as to whether that will cause Julia to freak out or not due to there being multiple versions of certain symbols available within the process space. You'll just have to try it and find out until we have a better solution available.

@sbromberger

This comment has been minimized.

Copy link

sbromberger commented Jul 13, 2018

Thank you for the followup.

You can try downloading the official Julia binaries, extracting the libgfortran.so.4 file, and putting it into your from-source Julia installation's lib directory.

Unfortunately I won't be able to do that due to environmental restrictions here.

I know people are working on this, but to set expectations, is there an approximate timeframe? The lack of a working eigs is causing failures in LightGraphs; if a fix is expected to take longer than a few more days, then it might make sense just to pull all uses of eigs out of the package in order to make progress in reviewing current PRs.

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jul 13, 2018

if a fix is expected to take longer than a few more days

It will definitely be more than a few days; it's very complex to create infrastructure to support every reasonable toolchain (rather than a single reasonable toolchain, which is what we do right now); think single-digit months rather than single-digit days.

If it is the official Julia distribution itself that is the problem, you don't have to get libgfortran.so.4 from that source. You could get it from your distribution's GCC channel, you could grab it from any other piece of software on your computer, or even download the GCC 7 source and compile it yourself to get libgfortran and copy that somewhere it can be used.

If none of those solutions will work in your environment, then I'm afraid you're just going to have to wait.

@sbromberger

This comment has been minimized.

Copy link

sbromberger commented Jul 13, 2018

Thanks. That's disappointing but I'm not in a position to help other than state my use case.

it's very complex to create infrastructure to support every reasonable toolchain

I'm curious about this. I wouldn't think that downloading and building the Julia source code would be so far outside the norm that it's not part of the testing strategy. Are people doing something else? Because the upshot is that over 600 packages are using Arpack, and none of them will work on source-built installations.

(This is not really so much a complaint as a question of whether our environment is really that strange.)

@chriselrod

This comment has been minimized.

Copy link

chriselrod commented Jul 13, 2018

Would it be feasible to support building Arpack from source as a generic fallback instead of downloading the binary?

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jul 13, 2018

I wouldn't think that downloading and building the Julia source code would be so far outside the norm that it's not part of the testing strategy.

Building Julia from source works just fine with these binaries, as long as you use a compiler version that is compatible with the binaries (e.g. GCC 7.x) or alternatively, are able to install the compiler support libraries (libgfortran, libgcc_s, etc...). Combine that with supporting the official Julia binaries, and we've covered a very large portion of the ecosystem. I don't have the hard numbers to tell you exactly how much of the ecosystem that covers, but anecdotally, I would guesstimate that the great majority of users use the precompiled binaries, and of those that don't, a good chunk are able to coerce their build system to be GCC 7-compatible.

This transition pain is considered worthwhile because the alternatives (using distro-provided package managers, building from source on user's machines, etc...) run into new issues constantly. It is a very large, ongoing, maintenance burden for us to constantly fix build scripts, debug user's broken toolchains, etc... It is much simpler for everyone if the binaries are built once, and simply distributed to users in the format they were going to compile down to anyway.

This of course raises the compiler compatibility issues, but I will also point out those compiler compatibility issues already existed in the "bad old days"; if you were using the official Julia binaries (as most users are) and you have to compile Arpack, you can run into similar problems. Not the exact same ones (where libraries are missing) but more subtle ones, where symbol duplication means that when Arpack tries to call a function whose functionality has changed between two compiler ABI versions, it just functions incorrectly/crashes.

Would it be feasible to support building Arpack from source as a generic fallback instead of downloading the binary?

Ah, that gives me another idea; Seth, for your particular use case, I'm sure you can manually compile Arpack and just insert it directly into Arpack.jl's deps/usr folder. Do something like this (yes, there's an obscenely long line in these instructions, oh well). Also I suggest doing a pkg> dev Arpack first, so that we can iterate on Arpack itself.

# First, clone arpack-ng and checkout a known good gitsha
git clone https://github.com/opencollab/arpack-ng.git arpack-ng
cd arpack-ng
git checkout b095052372aa95d4281a645ee1e367c28255c947

# We'll build inside the `build` directory
mkdir build; cd build/

# We're going to install to ~/.julia/dev/Arpack/deps/usr.  Change this if you have a different target installation location
prefix=$(echo ~/.julia/dev/Arpack/deps/usr)

# We need to link against OpenBLAS; get that from Julia: (Note; this only works for Julia 0.7)
openblas_dir=$(julia -e 'using Libdl; println(abspath(dirname(Libdl.dlpath(Libdl.dlopen(Base.libblas_name)))))') 

# Use cmake to configure, with a huge number of symbol renames for the fortran code.
cmake .. -DCMAKE_INSTALL_PREFIX=$prefix -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-L$openblas_dir -lopenblas64_" -DLAPACK_LIBRARIES="-L$prefix/lib -lopenblas64_" -DCMAKE_Fortran_FLAGS="-O2 -fPIC -ffixed-line-length-none -cpp -fdefault-integer-8 -Dsaxpy=saxpy_64 -Ddaxpy=daxpy_64  -Dscopy=scopy_64 -Ddcopy=dcopy_64  -Dsgemv=sgemv_64 -Ddgemv=dgemv_64  -Dsgeqr2=sgeqr2_64 -Ddgeqr2=dgeqr2_64  -Dslacpy=slacpy_64 -Ddlacpy=dlacpy_64  -Dslahqr=slahqr_64 -Ddlahqr=dlahqr_64  -Dslanhs=slanhs_64 -Ddlanhs=dlanhs_64  -Dslarnv=slarnv_64 -Ddlarnv=dlarnv_64  -Dslartg=slartg_64 -Ddlartg=dlartg_64  -Dslascl=slascl_64 -Ddlascl=dlascl_64  -Dslaset=slaset_64 -Ddlaset=dlaset_64  -Dsscal=sscal_64 -Ddscal=dscal_64  -Dstrevc=strevc_64 -Ddtrevc=dtrevc_64  -Dstrmm=strmm_64 -Ddtrmm=dtrmm_64  -Dstrsen=strsen_64 -Ddtrsen=dtrsen_64  -Dsgbmv=sgbmv_64 -Ddgbmv=dgbmv_64  -Dsgbtrf=sgbtrf_64 -Ddgbtrf=dgbtrf_64  -Dsgbtrs=sgbtrs_64 -Ddgbtrs=dgbtrs_64  -Dsgttrf=sgttrf_64 -Ddgttrf=dgttrf_64  -Dsgttrs=sgttrs_64 -Ddgttrs=dgttrs_64  -Dspttrf=spttrf_64 -Ddpttrf=dpttrf_64  -Dspttrs=spttrs_64 -Ddpttrs=dpttrs_64  -Dsdot=sdot_64 -Dddot=ddot_64  -Dsger=sger_64 -Ddger=dger_64  -Dslabad=slabad_64 -Ddlabad=dlabad_64  -Dslaev2=slaev2_64 -Ddlaev2=dlaev2_64  -Dslamch=slamch_64 -Ddlamch=dlamch_64  -Dslanst=slanst_64 -Ddlanst=dlanst_64  -Dslanv2=slanv2_64 -Ddlanv2=dlanv2_64  -Dslapy2=slapy2_64 -Ddlapy2=dlapy2_64  -Dslarf=slarf_64 -Ddlarf=dlarf_64  -Dslarfg=slarfg_64 -Ddlarfg=dlarfg_64  -Dslasr=slasr_64 -Ddlasr=dlasr_64  -Dsnrm2=snrm2_64 -Ddnrm2=dnrm2_64  -Dsorm2r=sorm2r_64 -Ddorm2r=dorm2r_64  -Dsrot=srot_64 -Ddrot=drot_64  -Dssteqr=ssteqr_64 -Ddsteqr=dsteqr_64  -Dsswap=sswap_64 -Ddswap=dswap_64  -Dcaxpy=caxpy_64 -Dzaxpy=zaxpy_64  -Dccopy=ccopy_64 -Dzcopy=zcopy_64  -Dcgemv=cgemv_64 -Dzgemv=zgemv_64  -Dcgeqr2=cgeqr2_64 -Dzgeqr2=zgeqr2_64  -Dclacpy=clacpy_64 -Dzlacpy=zlacpy_64  -Dclahqr=clahqr_64 -Dzlahqr=zlahqr_64  -Dclanhs=clanhs_64 -Dzlanhs=zlanhs_64  -Dclarnv=clarnv_64 -Dzlarnv=zlarnv_64  -Dclartg=clartg_64 -Dzlartg=zlartg_64  -Dclascl=clascl_64 -Dzlascl=zlascl_64  -Dclaset=claset_64 -Dzlaset=zlaset_64  -Dcscal=cscal_64 -Dzscal=zscal_64  -Dctrevc=ctrevc_64 -Dztrevc=ztrevc_64  -Dctrmm=ctrmm_64 -Dztrmm=ztrmm_64  -Dctrsen=ctrsen_64 -Dztrsen=ztrsen_64  -Dcgbmv=cgbmv_64 -Dzgbmv=zgbmv_64  -Dcgbtrf=cgbtrf_64 -Dzgbtrf=zgbtrf_64  -Dcgbtrs=cgbtrs_64 -Dzgbtrs=zgbtrs_64  -Dcgttrf=cgttrf_64 -Dzgttrf=zgttrf_64  -Dcgttrs=cgttrs_64 -Dzgttrs=zgttrs_64  -Dcpttrf=cpttrf_64 -Dzpttrf=zpttrf_64  -Dcpttrs=cpttrs_64 -Dzpttrs=zpttrs_64  -Dcdotc=cdotc_64 -Dzdotc=zdotc_64  -Dcgeru=cgeru_64 -Dzgeru=zgeru_64  -Dcunm2r=cunm2r_64 -Dzunm2r=zunm2r_64  -DSCOPY=SCOPY_64 -DDCOPY=DCOPY_64  -DSLABAD=SLABAD_64 -DDLABAD=DLABAD_64  -DSLAMCH=SLAMCH_64 -DDLAMCH=DLAMCH_64  -DSLANHS=SLANHS_64 -DDLANHS=DLANHS_64  -DSLANV2=SLANV2_64 -DDLANV2=DLANV2_64  -DSLARFG=SLARFG_64 -DDLARFG=DLARFG_64  -DSROT=SROT_64 -DDROT=DROT_64  -DSGEMV=SGEMV_64 -DDGEMV=DGEMV_64 -Dscnrm2=scnrm2_64 -Ddznrm2=dznrm2_64 -Dcsscal=csscal_64 -Dzdscal=zdscal_64"

# Do the make install
make install

Once you've verified that the necessary files have been dumped into your Arpack's deps/usr directory, you can write your own deps/deps.jl file:

## This file autogenerated by BinaryProvider.write_deps_file().
## Do not edit.
##
## Include this file within your main top-level source, and call
## `check_deps()` from within your module's `__init__()` method
const Arpack = joinpath(dirname(@__FILE__), "usr/lib64/libarpack.so")
function check_deps()
    global Arpack
    if !isfile(Arpack)
        error("$(Arpack) does not exist, Please re-run Pkg.build(\"Arpack\"), and restart Julia.")
    end

    if Libdl.dlopen_e(Arpack) == C_NULL
        error("$(Arpack) cannot be opened, Please re-run Pkg.build(\"Arpack\"), and restart Julia.")
    end
end

Note that on my machine, the make install process put libarpack.so into deps/usr/lib64, not deps/usr/lib. You may have to change that path to match your machine's output.

@crstnbr

This comment has been minimized.

Copy link

crstnbr commented Nov 9, 2018

Maybe I don't understand but the "fix" doesn't work for me. If I install and build Arpack with Julia + OpenBLAS (binaries) and then use my compiled Julia + MKL to use it I get:

julia> using Arpack
[ Info: Recompiling stale cache file /gpfs/homea/hku27/hku273/.julia-mkl/compiled/v1.0/Arpack/X5VZL.ji for Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97]

julia> eigs(rand(100,100))
ERROR: error compiling aupd_wrapper: error compiling naupd: could not load library "/gpfs/homea/hku27/hku273/.julia-mkl/packages/Arpack/0oGDs/deps/usr/lib/libarpack.so"
libopenblas64_.so.0: cannot open shared object file: No such file or directory
Stacktrace:
 [1] #_eigs#17(::Int64, ::Int64, ::Symbol, ::Float64, ::Int64, ::Nothing, ::Array{Float64,1}, ::Bool, ::typeof(Arpack._eigs), ::Array{Float64,2}, ::LinearAlgebra.UniformScaling{Bool}) at /gpfs/homea/hku27/hku273/.julia-mkl/packages/Arpack/0oGDs/src/Arpack.jl:120
 [2] _eigs at /gpfs/homea/hku27/hku273/.julia-mkl/packages/Arpack/0oGDs/src/Arpack.jl:88 [inlined]
 [3] #eigs#10 at /gpfs/homea/hku27/hku273/.julia-mkl/packages/Arpack/0oGDs/src/Arpack.jl:63 [inlined]
 [4] eigs at /gpfs/homea/hku27/hku273/.julia-mkl/packages/Arpack/0oGDs/src/Arpack.jl:63 [inlined]
 [5] #eigs#9 at /gpfs/homea/hku27/hku273/.julia-mkl/packages/Arpack/0oGDs/src/Arpack.jl:62 [inlined]
 [6] eigs(::Array{Float64,2}) at /gpfs/homea/hku27/hku273/.julia-mkl/packages/Arpack/0oGDs/src/Arpack.jl:62
 [7] top-level scope at none:0

Do have to use a self-compiled Julia + OpenBLAS?

@crstnbr

This comment has been minimized.

Copy link

crstnbr commented Nov 9, 2018

Note that I also can't use Distributions.jl because of this (as it seems to depend on Arpack).

@dlfivefifty

This comment has been minimized.

Copy link

dlfivefifty commented Nov 10, 2018

Deleting and reinstalling Homebrew fixed the issue with Arpack for me.

@riccardosven

This comment has been minimized.

Copy link

riccardosven commented Nov 12, 2018

The problem seems to be that libarpack is linked to libopenblas64_.so as shown in

$ ldd ~/.julia/packages/Arpack/UiiMc/deps/usr/liblibarpack.so
	linux-vdso.so.1 (0x00007fff363c5000)
	libopenblas64_.so.0 => not found
	libgfortran.so.5 => /usr/lib/libgfortran.so.5 (0x00007f67481fe000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f6748079000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f674805f000)
	libquadmath.so.0 => /usr/lib/libquadmath.so.0 (0x00007f674801c000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f6747e58000)
	libz.so.1 => /usr/lib/../lib/libz.so.1 (0x00007f6747c3f000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007f6748709000)

On Archlinux with community LibOpenblas, a symbolic link to libopenblas.so fixes the issue:

# ln -s /usr/lib/libopenblas.so /usr/lib/libopenblas64_.so.0

and Arpack builds fine. (@crstnbr, this also fixes the problem with Distributions.jl).

Why is the "64_" added to the library filename?

@crstnbr

This comment has been minimized.

Copy link

crstnbr commented Nov 12, 2018

@rsrsl Unfortunately, I don't have write access to /usr/lib. I'm on a university cluster.

Also, for me the ouput is

$ ldd packages/Arpack/UiiMc/deps/usr/lib/libarpack.so
        linux-vdso.so.1 =>  (0x00007ffda671f000)
        libopenblas64_.so.0 => not found
        libgfortran.so.5 => not found
        libm.so.6 => /lib64/libm.so.6 (0x00002b1702652000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b17028d6000)
        libquadmath.so.0 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00002b1702aed000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003801600000)

so apparently more than just libopenblas64_.so.0 can't be found.

Regarding the 64_ suffix, maybe this has something to do with the comment here:

Binaries designed to be compatible with Julia are regularly provided in https://github.com/staticfloat/OpenBLASBuilder/releases . Note that their 64bit builds with INTERFACE64=1 have 64 appended to the function symbol names, so your code needs to refer to e.g. gemm_64 rather than gemm_

@aytekinar

This comment has been minimized.

Copy link

aytekinar commented Nov 12, 2018

@crstnbr, you have libgfortran missing on the cluster. Both libgfortran and libquadmath are provided in that package.

@rsrsl Unfortunately, I don't have write access to /usr/lib. I'm on a university cluster.

Maybe you can try

mkdir -p $HOME/usr/lib
ln -fis /usr/lib/libopenblas.so $HOME/usr/lib/libopenblas64_.so.0
LD_LIBRARY_PATH=$HOME/usr/lib julia

to set the symbolic link in your home folder, configure your LD_LIBRARY_PATH environment variable accordingly and fire up julia. Then, ]build Arpack should work.

IMHO the line that forces Julia's preference on the so naming convention of OpenBLAS is the problem. It would be better if ArpackBuilder downloaded OpenBLAS from source and built it, and then built Arpack using cmake's CMAKE_PREFIX_PATH variable to point to $prefix to link against the OpenBLAS built from source. This way, the tarballs would be self-contained.

In principle, I think that the prebuilt binaries should (re-)distribute their dependencies without assuming how the target system is configured. But I am no expert, either :(

@andreasnoack

This comment has been minimized.

Copy link
Member

andreasnoack commented Nov 12, 2018

Why is the "64_" added to the library filename?

Because we build OpenBLAS with 64 bit integers. I think most Linux distros build with 32 bit integers.

@aytekinar

This comment has been minimized.

Copy link

aytekinar commented Nov 12, 2018

I think this choice has nothing to do with the so naming of the library. It affects the symbol names, but does not automagically change the library name. Moreover, SUFFIX64_UNDERSCORE does not seem to be an option for building OpenBLAS, either. One needs to parse CMakeLists.txt manually, as cmake-gui ../ does not show it as an option --- neither normal, nor advanced.

@andreasnoack

This comment has been minimized.

Copy link
Member

andreasnoack commented Nov 12, 2018

See JuliaLang/julia#8734 for some details.

@arnonerba

This comment has been minimized.

Copy link

arnonerba commented Nov 16, 2018

Interesting. I tested @chriselrod's workaround and received the same error as @crstnbr:

error compiling aupd_wrapper: error compiling naupd: could not load library "/home/ec2-user/.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so"
libopenblas64_.so.0: cannot open shared object file: No such file or directory

It looks like libarpack.so is indeed trying to link to libopenblas64_.so.0, which only the OpenBLAS version of Julia provides:

$ ldd /home/ec2-user/.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so
	linux-vdso.so.1 (0x00007ffdc23e8000)
	libopenblas64_.so.0 => not found
	libgfortran.so.4 => /lib64/libgfortran.so.4 (0x00007f98334fc000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f98331b1000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9832f9a000)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f9832d5a000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f98329a4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9833b30000)

On my system (Amazon Linux 2) there is no libopenblas.so in /usr/lib to symlink to. It looks like Julia is providing that shared library itself when I build it from source with OpenBLAS.

Also, both my OpenBLAS and MKL versions are self-compiled. Arpack builds and tests without error on the OpenBLAS version.

@Irratzo

This comment has been minimized.

Copy link

Irratzo commented Dec 25, 2018

The following resolved this 'Arpack Error problem' for my purpose (installing DifferentialEquations.jl which needs Arpack, for use in jupyter) for my setup (precompiled binary package julia 1.0.3; anaconda environment 'julia' with installed julia 1.0.2, conda 4.5.12). It boils down to using ] add IJulia InstantiateFromURL; precompile for the installation.

  • Unrelated to this issue: If you run into an 'LLVM error': solve that as I describe here.

  • Now for the 'Arpack error'. Note that in step 2), I install IJulia not the Arpack-dependent DifferentialEquations, but the command used recompiles that as well as it goes along, and that finally resolves the 'Arpack error'.

# Step 1): in system julia:
julia
julia> import Pkg; Pkg.add("DifferentialEquations")
# build packages error message requires; among
# them, Pkg.build("Arpack"); should work here without error.
#
# Step 2):
# in conda env "julia"'s julia (make sure jupyter is installed in that env):
# Now, If we tried Pkg.add("DifferentialEquations") here directly, we would run
# into the 'Arpack error' again. Instead, do this:
source activate julia
(julia) julia
julia> ]
(v1.0) pkg> add IJulia InstantiateFromURL; precompile
#  Installed InstantiateFromURL ─ v0.2.1
#  Installed Suppressor ───────── v0.1.1
# ...
#   [8f4d0f93] + Conda v1.1.1
#   [7073ff75] + IJulia v1.14.1
# ...
# Precompiling project...
# Precompiling DifferentialEquations
# ...
# [ Info: Recompiling stale cache file /home/johannes/.julia/compiled/v1.0/DifferentialEquations/UQdwS.ji for DifferentialEquations [0c46a032-eb83-5123-abaf-570d42b7fbaa]
# Precompiling InstantiateFromURL
# [ Info: Precompiling InstantiateFromURL [43edad99-fa64-5e4f-9937-1c09a410b73f]

Now I can finally use DifferentialEquaions in jupyter lab.
Note that both julia installations here seem to use the same ~/.julia. Seems that the InstantiateFromURL; precompile command resolves some crucial dependencies more intelligently.

@affans

This comment has been minimized.

Copy link

affans commented Jan 3, 2019

Is there a fix for this yet? I tried some suggestions above but they didn't work. I am not sure how the Julia on our university cluster was installed. I've emailed the admin and he'll let me know shortly.

Here is my error message. I don't seem to have libopenblas.so anywhere on my system. If our Julia installation was compiled from source, I am sure it used OpenBLAS (as far as I know, you need to pay for MKL which we didn't do).

(v1.0) pkg> build Arpack
  Building Arpack → `~/.julia/packages/Arpack/qof0w/deps/build.log`
┌ Error: Error building `Arpack`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/home/affans/.julia/packages/Arpack/qof0w/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/affans/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│  [2] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [3] top-level scope at none:0
│  [4] include(::String) at ./client.jl:392
│  [5] top-level scope at none:0
│ in expression starting at /home/affans/.julia/packages/Arpack/qof0w/deps/build.jl:74
└ @ Pkg.Operations /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097

Some additional information

[affans@hpc ~]$ ldd ./.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so
	linux-vdso.so.1 =>  (0x00002aaaaaacd000)
	libopenblas64_.so.0 => not found
	libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00002aaaaaf26000)
	libm.so.6 => /lib64/libm.so.6 (0x00002aaaab248000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab54a000)
	libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00002aaaab760000)
	libc.so.6 => /lib64/libc.so.6 (0x00002aaaab99c000)
	/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jan 3, 2019

@affans let's figure out some basic information about your Julia. Please post the following:

julia> versioninfo()
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)

julia> Base.libblas_name
"libopenblas64_"

julia> using Libdl; filter(lib -> occursin("blas", lib), Libdl.dllist())
1-element Array{AbstractString,1}:
 "/home/sabae/dist/julia-1.0.3/bin/../lib/julia/libopenblas64_.so"

This will tell us what version of Julia you're using and also what BLAS library you have loaded.

@affans

This comment has been minimized.

Copy link

affans commented Jan 3, 2019

Hi @staticfloat Thanks for the help!

julia> versioninfo()
Julia Version 1.0.3
Commit 04330c0378 (2018-12-16 21:23 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, broadwell)
julia> Base.libblas_name
"libopenblasp64_"

Interesting that i have a p64 appended to this.

julia> using Libdl; filter(lib -> occursin("blas", lib), Libdl.dllist())
1-element Array{AbstractString,1}:
 "/usr/bin/../lib64/julia/libopenblasp64_.so"
@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jan 3, 2019

Yes, that would be the problem. Your system's libopenblas is named differently from what Julia by default names it. You can try creating a symlink from libopenblasp64_.sp to libopenblas64_.so, within /usr/lib64/julia, so that either name works. If you don't have write access to /usr/lib64/julia, create a directory in your home (e.g. ~/lib64), place the symlink there (ln -s /usr/lib64/julia/libopenblasp64_.so ~/lib64/libopenblas64_.so), then tack that location onto the end of LD_LIBRARY_PATH and launch Julia (LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$(echo ~)/lib64 julia)

@affans

This comment has been minimized.

Copy link

affans commented Jan 3, 2019

@staticfloat That's I'll try that in a second. A quick question though. The file libopenblasp64_.sp already exists in /usr/lib64/julia.. i.e. the julia folder. I am assuming that it is the installation of Julia that makes that folder and puts the libraries there. So how come it can't find it's own file (or that it uses a different name?)

Unless the file libopenblasp64.so comes from somewhere else?

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jan 3, 2019

The problem is that libarpack.so is looking for libopenblas64_so., not libopenblasp64_.so, which is what Julia has provided. In a "default" Julia installation, the name of libopenblas is standardized to libopenblas64_.so (e.g. no p, which for historical reasons is added on to the end of libopenblas depending on the architecture it was built on). Something about the environment that your Julia was built within does not have this renaming which is not a problem for Julia, but is a problem for ARPACK, because unfortunately binary dependencies are not as flexible as Julia itself, which can handle dynamically changing which library path it looks for on-the-fly. And so we must create a second mapping to allow your libarpack to get "forwarded" to the libopenblas that already exists on your system when it looks for it.

So I suggest that you create a symlink named libopenblas64_.so that points to libopenblasp64_.so, so that when libarpack.so attempts to load libopenblas64_.so, it finds the symlink and follows it to libopenblasp64_.so.

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jan 3, 2019

Oh, perhaps another more fundamental explanation is helpful; the dynamic linker keys off of the filename of the library that a dynamic object (like libarpack.so) declares as a dependency; meaning that when libarpack.so says it needs libopenblasp64_.so, it completely ignores the existence of libopenblas64_.so because the names are different; from the perspective of the dynamic linker, those two libraries are completely different, even if they would have the same contents.

@affans

This comment has been minimized.

Copy link

affans commented Jan 3, 2019

I don't think it's working.. unless I messed up the LD_LIBRARY_PATH.

(v1.0) pkg> build Arpack
  Building Arpack → `~/.julia/packages/Arpack/qof0w/deps/build.log`
┌ Error: Error building `Arpack`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/home/affans/.julia/packages/Arpack/qof0w/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/affans/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│  [2] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [3] top-level scope at none:0
│  [4] include(::String) at ./client.jl:392
│  [5] top-level scope at none:0
│ in expression starting at /home/affans/.julia/packages/Arpack/qof0w/deps/build.jl:74
└ @ Pkg.Operations /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097

here is my LD_LIRBARY_PATH:

[affans@hpc lib64]$ echo $LD_LIBRARY_PATH 
/cm/shared/apps/openmpi/gcc/64/1.10.1/lib64:/home/affans/lib64

[affans@hpc lib64]$ ls -l
total 4
lrwxrwxrwx 1 affans abmlab  35 Jan  3 17:13 libopenblas64_.so -> /usr/lib64/julia/libopenblasp64_.so

Maybe I need to "flush" everything and start over. Can I delete my .julia folder with significant problems? I only have a handful of packages installed that I can reinstall. I don't know if ~/.julia contains other important bits of Julia.
Thanks for the explanation. I am learning quite a bit!!! Not a linux user :(

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jan 3, 2019

This issue is getting a little bit long, and since the fundamental technical problem with this package that started this issue has been solved, I'm going to close the issue so that we don't bother too many other users with our problems.

Please copy the above message into a new issue, ping me, and I'll help you work through the problems.

@staticfloat staticfloat closed this Jan 3, 2019

@eli-schwartz

This comment has been minimized.

Copy link

eli-schwartz commented Jan 3, 2019

since the fundamental technical problem with this package that started this issue has been solved

But it isn't solved. It's been partially worked around under certain specific conditions. And the binary mappings you do provide do not include anything suitable for source installations where the default blas implementation is not libopenblas64_.so, but libopenblas.so, or where mkl is in use.

This means that you only support a limited subset of the use cases that julia itself supports, a problem which could have been elegantly solved 6 months ago by making a policy decision that users should be allowed to make one last Hail Mary attempt to use the system arpack library (rather than steadfastly refusing to allow users the right to control their own software).

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jan 3, 2019

And the binary mappings you do provide do not include anything suitable for source installations where the default blas implementation is not libopenblas64_.so, but libopenblas.so

It is infeasible for us to provide precompiled .so files for all the variations of BLAS library names that could exist in the wild; there are an infinite number. We provide precompiled binaries that will work as long as users are building with the recommended defaults.

a problem which could have been elegantly solved 6 months ago by making a policy decision that users should be allowed to make one last Hail Mary attempt to use the system arpack library (rather than steadfastly refusing to allow users the right to control their own software).

I understand you feel strongly about this; but I'm not sure what more you're asking for. Users can always compile their own libarpack.so and copy it into (for example) ~/.julia/packages/Arpack/<pkg hash>/deps/usr/lib where it will be picked up by a pkg> build Arpack. That works, and serves as the release valve for users that truly need to build their own binaries, so a 'Hail Mary' attempt is always possible. We have attempted to support packaging systems that are much more eager to push users toward building from source on their own machines, and it doesn't work as well as what we have right now, even with the shortcomings that currently exist.

@simonbyrne

This comment has been minimized.

Copy link
Collaborator

simonbyrne commented Jan 3, 2019

Users can always compile their own libarpack.so and copy it into (for example) ~/.julia/packages/Arpack/<pkg hash>/deps/usr/lib where it will be picked up by a pkg> build Arpack.

Would be good to document this, or throw an informative error message.

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jan 3, 2019

This is true for all Pkg packages; it's one of those implicit pieces of knowledge that isn't really documented anywhere. Once we get JuliaLang/Pkg.jl#841 sorted out (this is high on my list of priorities) this should become a little more "explicit". We plan to have binary packages be dealt with a little more explicitly than they are right now, with Pkg understanding the concept of a "binary artifact" that is downloaded and reasoned within the Pkg universe just like Julia packages are. Just like you can take manual control of your Julia packages with dev, you will be able to take manual control of binary packages by dev'ing them (or some other, similar verb), instructing Pkg to just load whatever is within ~/.julia/dev/Arpack_jll or somesuch without bothering to engage in any kind of version/compatibility resolution.

Between now and then, any improvements to documentation or error messages are welcome, although I think the right place for this kind of thing is probably within Pkg.jl, not Arpack.jl.

@heroxbd

This comment has been minimized.

Copy link

heroxbd commented Jan 4, 2019

@staticfloat I am realy disappointed on how Julia developers take this issue. You are bunding everything, not only arpack, into the user's HOME directory without even attempting to reuse from the host system. That will be a big burden for distribution packagers, say Debian, Gentoo, Arch or Fedora to integrate Julia into a larger operating systems.

It is infeasible for us to provide precompiled .so files for all the variations of BLAS library names that could exist in the wild; there are an infinite number.

Build systems (GNU autotools, CMake, etc.), are designed to deal with the infinite number possibility by allowing users to pass switches to specify which arpack to use.

Repeatedly pointing power users to download big blobs of precompiled binary and bunding everything feels completely unprofessional.

@pengwyn

This comment has been minimized.

Copy link

pengwyn commented Jan 21, 2019

I agree with @heroxbd. I think it is great to provide a fallback in the case that the host operating system doesn't support a particular library. However, including popular libraries as binaries is weird. If you absolutely insisted on boxing an external library with your program, wouldn't the normal thing to do here be to compile libarpack? I suspect this would take a long time to compile... but isn't that exactly why we have distro repositories?

What is really annoying is that "building from source" is not a power user thing, but happens in the packaging of Julia for many (most?) distros of Linux. And what's worse is there is no way for the distro maintainer to control what happens when Pkg.add("Arpack") is run.

In short, why not leave inter-dependencies of libraries to the OS and merely complain (or give a link to a binary to download) when a library doesn't exist?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment