Build issue: Provider BuildProcess failed to satisfy dependency libopenspecfun #162

carstenbauer · 2019-05-15T11:43:37Z

Julia 1.1.0 built from source, linked against MKL, in a fresh environment:

(My gcc version is 8.3.0)

(julia-mkl) pkg> build
  Building SpecialFunctions → `/p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/build.log`
┌ Error: Error building `SpecialFunctions`:
│ ┌ Warning: Cannot make sense of autodetected libstdc++ ABI version ('3.4.0')
│ └ @ BinaryProvider /p/project/chku27/hku273/.julia-mkl/packages/BinaryProvider/TcAwt/src/PlatformNames.jl:624
│ mkdir -p /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/usr/lib
│ mkdir -p /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/usr/lib
│ mkdir -p /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/usr/include
│ cp -a libopenspecfun.*so* /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/usr/lib/
│ cp -a libopenspecfun.a /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/usr/lib/
│ cp -a Faddeeva/Faddeeva.h /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/usr/include
│ [ Info: Attempting to create directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/downloads
│ [ Info: Directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/downloads already exists
│ [ Info: Downloading file https://github.com/JuliaLang/openspecfun/archive/v0.5.3.tar.gz
│ [ Info: Done downloading file https://github.com/JuliaLang/openspecfun/archive/v0.5.3.tar.gz
│ [ Info: Attempting to create directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/src
│ [ Info: Directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/src already exists
│ [ Info: Attempting to create directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps
│ [ Info: Directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps already exists
│ [ Info: Path /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/src/openspecfun-0.5.3 already exists
│ [ Info: Attempting to create directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/builds
│ [ Info: Directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/builds already exists
│ [ Info: Changing directory to /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/src/openspecfun-0.5.3
│ [ Info: Attempting to create directory /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/usr/lib
│ ERROR: LoadError: LoadError: Provider BuildProcess failed to satisfy dependency libopenspecfun
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] satisfy!(::BinDeps.LibraryDependency, ::Array{DataType,1}) at /p/project/chku27/hku273/.julia-mkl/packages/BinDeps/ZEval/src/dependencies.jl:945
│  [3] satisfy!(::BinDeps.LibraryDependency) at /p/project/chku27/hku273/.julia-mkl/packages/BinDeps/ZEval/src/dependencies.jl:922
│  [4] top-level scope at /p/project/chku27/hku273/.julia-mkl/packages/BinDeps/ZEval/src/dependencies.jl:977
│  [5] include at ./boot.jl:326 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1038
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:403
│  [9] top-level scope at /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/build.jl:150
│  [10] include at ./boot.jl:326 [inlined]
│  [11] include_relative(::Module, ::String) at ./loading.jl:1038
│  [12] include(::Module, ::String) at ./sysimg.jl:29
│  [13] include(::String) at ./client.jl:403
│  [14] top-level scope at none:0
│ in expression starting at /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/scratch.jl:113
│ in expression starting at /p/project/chku27/hku273/.julia-mkl/packages/SpecialFunctions/fvheQ/deps/build.jl:149
└ @ Pkg.Operations /p/project/chku27/hku273/software/julia-mkl/julia-src/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

Building the package works on the same machine using the binaries from the website.

Any help is much appreciated!

carstenbauer · 2019-05-15T11:47:14Z

Maybe related to JuliaPackaging/BinaryProvider.jl#153

However, I get

julia> BinaryProvider.detect_libstdcxx_abi()
┌ Warning: Cannot make sense of autodetected libstdc++ ABI version ('3.4.0')
└ @ BinaryProvider /p/project/chku27/hku273/.julia-mkl/packages/BinaryProvider/TcAwt/src/PlatformNames.jl:624
:gcc_any

both with the self-compiled MKL julia and the regular binaries.

carstenbauer · 2019-05-15T12:06:13Z

Julia MKL:

julia> BinaryProvider.detect_compiler_abi()
┌ Warning: Cannot make sense of autodetected libstdc++ ABI version ('3.4.0')
└ @ BinaryProvider /p/project/chku27/hku273/.julia-mkl/packages/BinaryProvider/TcAwt/src/PlatformNames.jl:624
CompilerABI(:gcc_any, :cxx11)

julia> BinaryProvider.detect_libgfortran_abi()
:gcc_any

Julia binaries

julia> BinaryProvider.detect_compiler_abi()
CompilerABI(:gcc7, :cxx11)

julia> BinaryProvider.detect_libgfortran_abi()
:gcc7

Why the difference and why don't I get :gcc8 in either case?

carstenbauer · 2019-05-15T12:16:18Z

Alright I traced the issue down to https://github.com/JuliaPackaging/BinaryProvider.jl/blob/20c5140935276fdf1807866b2ab7898bebb21c96/src/PlatformNames.jl#L578

function detect_libgfortran_abi()
    libgfortran_paths = filter(x -> occursin("libgfortran", x), dllist())
    if isempty(libgfortran_paths)
         # One day, I hope to not be linking against libgfortran in base Julia
        return :gcc_any
    end
    return detect_libgfortran_abi(first(libgfortran_paths))
end

Julia MKL

julia> libgfortran_paths = filter(x -> occursin("libgfortran", x), dllist())
0-element Array{AbstractString,1}

Julia binaries

julia> libgfortran_paths = filter(x -> occursin("libgfortran", x), dllist())
1-element Array{AbstractString,1}:
 "/p/project/chku27/hku273/software/julia/1.1.0/bin/../lib/julia/libgfortran.so.4"

The question is why....

simonbyrne · 2019-05-15T19:01:56Z

@staticfloat any ideas?

staticfloat · 2019-05-15T20:07:28Z

Is your Julia not linked against libgfortran? What fortran compiler did you use to build OpenBLAS? You should be able to tell via using Libdl; Libdl.dllist().

gasagna · 2019-05-21T10:06:10Z

I am also affected by this, but my installation from source is not using MKL. Actually, this pops up when precompiling JuliaDB, which depends on TextParse, which depends on DoubleFloats, which depends on QuadMath, which depends on SpecialFunctions.

gasagna · 2019-05-21T10:56:08Z

Looking at this in more detail, I have noticed that platform_key_abi() returns MacOS(:x86_64, compiler_abi=CompilerABI(:gcc8, :cxx03)) on my platform. This is not listed in any of the download_info in the build.jl script. I have thus ]dev'ed this package and locally changed the build script to accept my platform info to install the binary for the abi key MacOS(:x86_64, compiler_abi=CompilerABI(:gcc8, :cxx03)). This works (test SpecialFunctions works OK) but it is a hack.

staticfloat · 2019-05-21T16:10:39Z

I just helped a user track down a complex problem with SpecialFunctions; it appears that the intermediate build products left behind by the BinDeps compile escape-hatch can interfere with proper functioning of the BinaryProvider download path; so one thing to try is to just wipe out the build directory and re-build. The directory you would delete is findable by running julia -e 'using SpecialFunctions; print(joinpath(dirname(dirname(pathof(SpecialFunctions))), "deps", "usr"))', it should result in something like ~/.julia/packages/SpecialFunctions/fvheQ/deps/usr. Delete that, then re-run pkg> build SpecialFunctions and see if things work better for you.

I have noticed that platform_key_abi() returns MacOS(:x86_64, compiler_abi=CompilerABI(:gcc8, :cxx03)) on my platform. This is not listed in any of the download_info in the build.jl script.

Your platform should match this one; because this package does not have any dependency on cxx11 string ABI, the :cxx03 portion of your platform is unnecessary information and gets discarded by choose_download(). The reason your dev'ed package worked, I'm willing to bet, is because it simply created a new package directory where there were no partial build products from a previous BinDeps-build; if you clear out the deps/usr directory in the main package directory, I bet it will work just fine, even without your patch.

gasagna · 2019-05-21T21:02:17Z

@staticfloat Yes, correct. Wiping the usr folder and recompiling did the trick. Thanks!

clouds56 · 2019-06-15T08:02:56Z

Same here, build from source success (gcc-7.4.0, gfortran-7.4.0, julia-1.1.0 from nixpkgs), but load failure.
I've look into it and found that built dylib could not be dlopen due to openlibm (my openlibm version is 0.6.0, I'm not sure if it matters)

janfrancu · 2020-06-27T20:20:12Z

May be related. I encountered similar problems even on precompiled linux binaries (v1.4.1, running on Ubuntu 20.04), but with MKL backend for BLAS.

julia> using SpecialFunctions
[ Info: Precompiling SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b]                                          
ERROR: LoadError: InitError: could not load library "/home/*****/.julia/artifacts/b7df333bca517adecd37cf639f4dc9462933
6f71/lib/libopenspecfun.so"                                                                                           
libgfortran.so.5: cannot open shared object file: No such file or directory                                           
Stacktrace:                                                                                                           
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /home/*****/Apps/julia-1.4.1/share/julia/stdlib/v1.4/Libdl/src/L
ibdl.jl:109                                                                                                           
 [2] dlopen at /home/*****/Apps/julia-1.4.1/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
 [3] __init__() at /home/*****/.julia/packages/OpenSpecFun_jll/XrUb6/src/wrappers/x86_64-linux-gnu-libgfortran5.jl:35 
 [4] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697                                           
 [5] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:781                                      
 [6] _require(::Base.PkgId) at ./loading.jl:1006                                                                      
 [7] require(::Base.PkgId) at ./loading.jl:927                                                                        
 [8] require(::Module, ::Symbol) at ./loading.jl:922                                                                  
 [9] include(::Module, ::String) at ./Base.jl:377                                                                     
 [10] top-level scope at none:2                                                                                       
 [11] eval at ./boot.jl:331 [inlined]                                                                                 
 [12] eval(::Expr) at ./client.jl:449                                                                                 
 [13] top-level scope at ./none:3                                                                                     
during initialization of module OpenSpecFun_jll                                                                       
in expression starting at /home/*****/.julia/packages/SpecialFunctions/O3MVw/src/SpecialFunctions.jl:3                
ERROR: Failed to precompile SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b] to /home/*****/.julia/compiled/v1.
4/SpecialFunctions/78gOt_TbCmq.ji.                                                                                    
Stacktrace:                                                                                                           
 [1] error(::String) at ./error.jl:33                                                                                 
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
 [3] _require(::Base.PkgId) at ./loading.jl:1029           
 [4] require(::Base.PkgId) at ./loading.jl:927             
 [5] require(::Module, ::Symbol) at ./loading.jl:922

The thing, that allowed me to import it is to load BinaryProvider into scope

julia> using BinaryProvider

julia> using SpecialFunctions
[ Info: Precompiling SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b]

I have not properly tested the functionality of SpecialFunctions, but the factorial function, I needed, worked.

ViralBShah closed this as completed Nov 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build issue: Provider BuildProcess failed to satisfy dependency libopenspecfun #162

Build issue: Provider BuildProcess failed to satisfy dependency libopenspecfun #162

carstenbauer commented May 15, 2019 •

edited

Loading

carstenbauer commented May 15, 2019 •

edited

Loading

carstenbauer commented May 15, 2019 •

edited

Loading

carstenbauer commented May 15, 2019 •

edited

Loading

simonbyrne commented May 15, 2019

staticfloat commented May 15, 2019

gasagna commented May 21, 2019

gasagna commented May 21, 2019

staticfloat commented May 21, 2019

gasagna commented May 21, 2019

clouds56 commented Jun 15, 2019

janfrancu commented Jun 27, 2020

Build issue: Provider BuildProcess failed to satisfy dependency libopenspecfun #162

Build issue: Provider BuildProcess failed to satisfy dependency libopenspecfun #162

Comments

carstenbauer commented May 15, 2019 • edited Loading

carstenbauer commented May 15, 2019 • edited Loading

carstenbauer commented May 15, 2019 • edited Loading

carstenbauer commented May 15, 2019 • edited Loading

simonbyrne commented May 15, 2019

staticfloat commented May 15, 2019

gasagna commented May 21, 2019

gasagna commented May 21, 2019

staticfloat commented May 21, 2019

gasagna commented May 21, 2019

clouds56 commented Jun 15, 2019

janfrancu commented Jun 27, 2020

carstenbauer commented May 15, 2019 •

edited

Loading

carstenbauer commented May 15, 2019 •

edited

Loading

carstenbauer commented May 15, 2019 •

edited

Loading

carstenbauer commented May 15, 2019 •

edited

Loading