Cannot compile HDF5_jll@1.12.0+1

[tbenst]

Hoping that someone can help me understand what changed on the HDF5.jl side between 0.13.6 and 0.14.1--perhaps this issue may originate with BinaryBuilder.jl?

Compiles fine:

[[HDF5]]
deps = ["Blosc", "HDF5_jll", "Libdl", "Mmap", "Random"]
git-tree-sha1 = "0713cbabdf855852dfab3ce6447c87145f3d9ea8"
uuid = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
version = "0.13.6"

[[HDF5_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "bc9c3d43ffd4d8988bfa372b86d4bdbd26860e95"
uuid = "0234f1f7-429e-5d53-9886-15a909be8d59"
version = "1.10.5+7"

Fails to compile:

[[HDF5]]
deps = ["Blosc", "Compat", "HDF5_jll", "Libdl", "Mmap", "Random", "Requires"]
git-tree-sha1 = "96d77533eb46e208e801b939db8a27626c166565"
uuid = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
version = "0.14.1"

[[HDF5_jll]]
deps = ["Artifacts", "JLLWrappers", "LibCURL_jll", "Libdl", "OpenSSL_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "fd83fa0bde42e01952757f01149dd968c06c4dba"
uuid = "0234f1f7-429e-5d53-9886-15a909be8d59"
version = "1.12.0+1"

The error:

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: LoadError: InitError: could not load library "/home/tyler/.julia/artifacts/519367e9365948074c1fcc9f4365597f147a5ab7/lib/libmbedtls.so"
/nix/store/7xkbxjxaryrz8ka1padvgh61ljfn85cr-mbedtls-2.16.3/lib/libmbedx509.so.0: undefined symbol: mbedtls_mutex_unlock
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] dlopen(::String, ::UInt32) at /build/source/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [3] macro expansion at /home/tyler/.julia/packages/JLLWrappers/KuIwt/src/products/library_generators.jl:61 [inlined]
 [4] __init__() at /home/tyler/.julia/packages/MbedTLS_jll/qMb7d/src/wrappers/x86_64-linux-gnu.jl:16
 [5] _include_from_serialized(::String, ::Array{Any,1}) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [6] _require_search_from_serialized(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [7] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [8] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [9] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [10] include at ./Base.jl:368 [inlined]
 [11] include(::String) at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/LibSSH2_jll.jl:1
 [12] top-level scope at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/LibSSH2_jll.jl:54
 [13] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)

 [14] top-level scope at none:2
 [15] eval at ./boot.jl:331 [inlined]
 [16] eval(::Expr) at ./client.jl:467
 [17] top-level scope at ./none:3
during initialization of module MbedTLS_jll
in expression starting at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/wrappers/x86_64-linux-gnu.jl:4
in expression starting at /home/tyler/.julia/packages/LibSSH2_jll/T20dp/src/LibSSH2_jll.jl:47
ERROR: LoadError: LoadError: Failed to precompile LibSSH2_jll [29816b5a-b9ab-546f-933c-edad1886dfa8] to /home/tyler/.julia/compiled/v1.5/LibSSH2_jll/K6mup_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [5] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [6] include at ./Base.jl:368 [inlined]
 [7] include(::String) at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/LibCURL_jll.jl:1
 [8] top-level scope at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/LibCURL_jll.jl:54
 [9] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:331 [inlined]
 [12] eval(::Expr) at ./client.jl:467
 [13] top-level scope at ./none:3
in expression starting at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/wrappers/x86_64-linux-gnu.jl:4
in expression starting at /home/tyler/.julia/packages/LibCURL_jll/9bFfE/src/LibCURL_jll.jl:47
ERROR: LoadError: LoadError: Failed to precompile LibCURL_jll [deac9b47-8bc7-5906-a0fe-35ac56dc84c0] to /home/tyler/.julia/compiled/v1.5/LibCURL_jll/9JWaY_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [5] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [6] top-level scope at /home/tyler/.julia/packages/JLLWrappers/KuIwt/src/toplevel_generators.jl:170
 [7] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [8] top-level scope at none:2
 [9] eval at ./boot.jl:331 [inlined]
 [10] eval(::Expr) at ./client.jl:467
 [11] top-level scope at ./none:3
in expression starting at /home/tyler/.julia/packages/HDF5_jll/BGk9m/src/wrappers/x86_64-linux-gnu.jl:6
in expression starting at /home/tyler/.julia/packages/HDF5_jll/BGk9m/src/HDF5_jll.jl:8
ERROR: LoadError: LoadError: Failed to precompile HDF5_jll [0234f1f7-429e-5d53-9886-15a909be8d59] to /home/tyler/.julia/compiled/v1.5/HDF5_jll/w9xrx_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [5] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [6] include at ./Base.jl:368 [inlined]
 [7] include(::String) at /home/tyler/.julia/packages/HDF5/LW5kO/src/HDF5.jl:1
 [8] top-level scope at /home/tyler/.julia/packages/HDF5/LW5kO/src/HDF5.jl:43
 [9] include(::Function, ::Module, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:331 [inlined]
 [12] eval(::Expr) at ./client.jl:467
 [13] top-level scope at ./none:3
in expression starting at /home/tyler/.julia/packages/HDF5/LW5kO/deps/deps.jl:3
in expression starting at /home/tyler/.julia/packages/HDF5/LW5kO/src/HDF5.jl:42
ERROR: Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /home/tyler/.julia/compiled/v1.5/HDF5/L7Dga_kELNb.ji.
Stacktrace:
 [1] error(::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [2] compilecache(::Base.PkgId, ::String) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [3] _require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:?
 [4] require(::Base.PkgId) at /nix/store/2028672rmpnlghw66hv6gh9a9ivssghy-julia-1.5.3/lib/julia/sys.so:? (repeats 2 times)

Further testing reveals that the issue arises from HDF5_jll at 1.12.0+1, and not HDF5.jl.

I believe this may have to do with the addition of LibCURL_jll and OpenSSL_jll...? It's not immediately clear how/why these libraries are needed? Many thanks for any thoughts!

Edit: possibly related: https://discourse.julialang.org/t/mbedtls-1-0-2-failed-to-load-on-julia-1-5-2-solved-by-downgrading-to-mbedtls-0-7-0/47430/7

[musm]

Could you post your versioninfo() ?

[tbenst]

sure,

julia> versioninfo()
Julia Version 1.5.3
Commit 788b2c77c1* (2020-11-09 13:37 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, cascadelake)
Environment:
  JULIA_PKGDIR = 

[giordano]

What distribution is that?

[tbenst]

It’s NixOS, a fully declarative and reproducible distro of Linux based on the Nix package manager. In order to be reproducible, all dependencies must be declared (but that’s our job ;). If you're curious and want to get a flavor of it, here's how we package and compile julia.

My system mbedtls is at 2.16.3. julia is using:

[[MbedTLS]]
deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"]
git-tree-sha1 = "1c38e51c3d08ef2278062ebceade0e46cefc96fe"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
version = "1.0.3"

[[MbedTLS_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "0eef589dd1c26a3ac9d753fe1a8bcad63f956fa6"
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
version = "2.16.8+1"

The mbedtls_mutex_lock symbol seems to be listed in system libmbedcrypto:

$ nm -D /nix/store/7xkbxjxaryrz8ka1padvgh61ljfn85cr-mbedtls-2.16.3/lib/libmbedcrypto.so | grep mbedtls_mutex_lock
0000000000061078 D mbedtls_mutex_lock

[musm]

Isn't that an MbedTLS_jll issue not HDF5_jl?

[jmert]

Yeah, I was just looking at this as well. I think it's an issue upstream somewhere since as far as I can tell, the MbedTLS_jll version of libmbedx509.so doesn't have an mbedtls_mutex_unlock that it needs resolved:

$ nm -D /home/justin/.julia/artifacts/519367e9365948074c1fcc9f4365597f147a5ab7/lib/libmbedx509.so | grep unlock

versus what I see in my system's copy (Arch linux):

$ nm -D /usr/lib/libmbedx509.so | grep unlock
                 U mbedtls_mutex_unlock

My guess is that something in the dependency chain is accidentally picking up a system library rather than the artifact libraries. That's not an issue in HDF5/HDF5_jll, so this issue should get moved somewhere else in the chain. (I'm unfortunately not super well versed in jll packages yet...)

[giordano]

You can see with

using Libdl
dllist()

what libraries are loaded

[musm]

This might be due to JuliaLang/julia#38409 (comment) and fixed by JuliaLang/julia#38409 ?

[giordano]

This might be due to JuliaLang/julia#38409 (comment) and fixed by JuliaLang/julia#38409 ?

I don't think it's related in any way

[musm]

😄

So MbedTLS_jll is built incorrectly and missing mbedtls_mutex_unlock ?

[giordano]

My guess is that something in the dependency chain is accidentally picking up a system library rather than the artifact libraries.

[musm]

Makes sense now. Ok I'm closing this. This looks like it might be a bug in the build system dependency chain locater itself then and should be moved to https://github.com/JuliaPackaging/BinaryBuilder.jl/issues

[giordano]

No, it's some other packages loading system libraries instead those served through artifacts