NCDatasets fails to install on Mac OS with brew julia install

[clessig]

Describe the bug

NCDatasets fails to install on Mac OS with brew julia install.

To Reproduce

julia> versioninfo()

Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
OS: macOS (x86_64-apple-darwin20.6.0)
CPU: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

julia> using Pkg

julia> Pkg.installed()["NCDatasets"]
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg /private/tmp/julia-20211223-81055-17amu57/julia-1.7.1/usr/share/julia/stdlib/v1.7/Pkg/src/Pkg.jl:595
v"0.11.7"

julia> Pkg.test("NCDatasets")
Testing NCDatasets
Status /private/var/folders/7w/c50fl15s1f1fwbhsvgp45c840000gn/T/jl_2p0wFr/Project.toml
[179af706] CFTime v0.1.1
[864edb3b] DataStructures v0.18.11
[85f8d34a] NCDatasets v0.11.7
[7243133f] NetCDF_jll v400.702.400+0
[ade2ca70] Dates @stdlib/Dates
[de0858da] Printf @stdlib/Printf
[9a3f8284] Random @stdlib/Random
[8dfed614] Test @stdlib/Test
Status /private/var/folders/7w/c50fl15s1f1fwbhsvgp45c840000gn/T/jl_2p0wFr/Manifest.toml
[179af706] CFTime v0.1.1
[34da2185] Compat v3.41.0
[864edb3b] DataStructures v0.18.11
[692b3bcd] JLLWrappers v1.3.0
[85f8d34a] NCDatasets v0.11.7
[bac558e1] OrderedCollections v1.4.1
[21216c6a] Preferences v1.2.3
[0234f1f7] HDF5_jll v1.12.0+1
[7243133f] NetCDF_jll v400.702.400+0
[458c3c95] OpenSSL_jll v1.1.10+0
[0dad84c5] ArgTools @stdlib/ArgTools
[56f22d72] Artifacts @stdlib/Artifacts
[2a0f44e3] Base64 @stdlib/Base64
[ade2ca70] Dates @stdlib/Dates
[8bb1440f] DelimitedFiles @stdlib/DelimitedFiles
[8ba89e20] Distributed @stdlib/Distributed
[f43a241f] Downloads @stdlib/Downloads
[b77e0a4c] InteractiveUtils @stdlib/InteractiveUtils
[b27032c2] LibCURL @stdlib/LibCURL
[76f85450] LibGit2 @stdlib/LibGit2
[8f399da3] Libdl @stdlib/Libdl
[37e2e46d] LinearAlgebra @stdlib/LinearAlgebra
[56ddb016] Logging @stdlib/Logging
[d6f4376e] Markdown @stdlib/Markdown
[a63ad114] Mmap @stdlib/Mmap
[ca575930] NetworkOptions @stdlib/NetworkOptions
[44cfe95a] Pkg @stdlib/Pkg
[de0858da] Printf @stdlib/Printf
[3fa0cd96] REPL @stdlib/REPL
[9a3f8284] Random @stdlib/Random
[ea8e919c] SHA @stdlib/SHA
[9e88b42a] Serialization @stdlib/Serialization
[1a1011a3] SharedArrays @stdlib/SharedArrays
[6462fe0b] Sockets @stdlib/Sockets
[2f01184e] SparseArrays @stdlib/SparseArrays
[10745b16] Statistics @stdlib/Statistics
[fa267f1f] TOML @stdlib/TOML
[a4e569a6] Tar @stdlib/Tar
[8dfed614] Test @stdlib/Test
[cf7118a7] UUIDs @stdlib/UUIDs
[4ec0a83e] Unicode @stdlib/Unicode
[e66e0078] CompilerSupportLibraries_jll @stdlib/CompilerSupportLibraries_jll
[deac9b47] LibCURL_jll @stdlib/LibCURL_jll
[29816b5a] LibSSH2_jll @stdlib/LibSSH2_jll
[c8ffd9c3] MbedTLS_jll @stdlib/MbedTLS_jll
[14a3606d] MozillaCACerts_jll @stdlib/MozillaCACerts_jll
[4536629a] OpenBLAS_jll @stdlib/OpenBLAS_jll
[83775a58] Zlib_jll @stdlib/Zlib_jll
[8e850b90] libblastrampoline_jll @stdlib/libblastrampoline_jll
[8e850ede] nghttp2_jll @stdlib/nghttp2_jll
[3f19e933] p7zip_jll @stdlib/p7zip_jll
Precompiling project...
✗ NCDatasets
0 dependencies successfully precompiled in 2 seconds (15 already precompiled)
1 dependency errored. To see a full report either run import Pkg; Pkg.precompile() or load the package
Testing Running tests...
ERROR: LoadError: InitError: could not load library "/Users/lessig/.julia/artifacts/0b92c3ca1de2e9dc46a381553278edd25af54980/lib/libnetcdf.18.dylib"
dlopen(/Users/lessig/.julia/artifacts/0b92c3ca1de2e9dc46a381553278edd25af54980/lib/libnetcdf.18.dylib, 1): Library not loaded: @rpath/libmbedtls.13.dylib
Referenced from: /Users/lessig/.julia/artifacts/0b92c3ca1de2e9dc46a381553278edd25af54980/lib/libnetcdf.18.dylib
Reason: image not found
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl ./libdl.jl:117
[2] dlopen(s::String, flags::UInt32)
@ Base.Libc.Libdl ./libdl.jl:117
[3] macro expansion
@ ~/.julia/packages/JLLWrappers/bkwIo/src/products/library_generators.jl:54 [inlined]
[4] init()
@ NetCDF_jll ~/.julia/packages/NetCDF_jll/xjKDT/src/wrappers/x86_64-apple-darwin.jl:14
[5] _include_from_serialized(path::String, depmods::Vector{Any})
@ Base ./loading.jl:768
[6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
@ Base ./loading.jl:854
[7] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1097
[8] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:1013
[9] require(into::Module, mod::Symbol)
@ Base ./loading.jl:997
[10] include
@ ./Base.jl:418 [inlined]
[11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
@ Base ./loading.jl:1318
[12] top-level scope
@ none:1
[13] eval
@ ./boot.jl:373 [inlined]
[14] eval(x::Expr)
@ Base.MainInclude ./client.jl:453
[15] top-level scope
@ none:1
during initialization of module NetCDF_jll
in expression starting at /Users/lessig/.julia/packages/NCDatasets/opjna/src/NCDatasets.jl:1
ERROR: LoadError: Failed to precompile NCDatasets [85f8d34a-cbdd-5861-8df4-14fed0d494ab] to /Users/lessig/.julia/compiled/v1.7/NCDatasets/jl_9pV4jZ.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
@ Base ./loading.jl:1466
[3] compilecache(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1410
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1120
[5] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:1013
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:997
[7] include(fname::String)
@ Base.MainInclude ./client.jl:451
[8] top-level scope
@ none:6
in expression starting at /Users/lessig/.julia/packages/NCDatasets/opjna/test/runtests.jl:1
ERROR: Package NCDatasets errored during testing
Stacktrace:
[1] pkgerror(msg::String)
@ Pkg.Types /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68
[2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
@ Pkg.Operations /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1672
[3] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
@ Pkg.API /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:421
[4] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
[5] test(pkgs::Vector{Pkg.Types.PackageSpec})
@ Pkg.API /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:144
[6] #test#87
@ /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:142 [inlined]
[7] test
@ /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:142 [inlined]
[8] #test#86
@ /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:141 [inlined]
[9] test(pkg::String)
@ Pkg.API /usr/local/Cellar/julia/1.7.1/share/julia/stdlib/v1.7/Pkg/src/API.jl:141
[10] top-level scope
@ REPL[23]:1

Expected behavior

Tests should succeed.

Environment

See above.

Full output

See above.

[Alexander-Barth]

In order to narrow down where the problem is, can you also install HDF5 in Julia home-brew an rund the tests with Pkg.test("HDF5"). Can you also check that these two packages work on our system when using Julia's official binaries from https://julialang.org/downloads/ ? Maybe it is an issue with the C dependencies of Julia and NetCDF.

[clessig]

Hi, thanks for the reply. I installed HDF but the issue persists. Looking at the first lines of the error I also installed libmbedtls but same error (see below).

Things work with Julia's official binaries.

Best,
Christian

---

julia> using Oceananigans
[ Info: Precompiling Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09]
ERROR: LoadError: InitError: could not load library "/Users/lessig/.julia/artifacts/0b92c3ca1de2e9dc46a381553278edd25af54980/lib/libnetcdf.18.dylib"
dlopen(/Users/lessig/.julia/artifacts/0b92c3ca1de2e9dc46a381553278edd25af54980/lib/libnetcdf.18.dylib, 1): Library not loaded: @rpath/libmbedtls.13.dylib
Referenced from: /Users/lessig/.julia/artifacts/0b92c3ca1de2e9dc46a381553278edd25af54980/lib/libnetcdf.18.dylib
Reason: image not found
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl ./libdl.jl:117
[2] dlopen(s::String, flags::UInt32)
@ Base.Libc.Libdl ./libdl.jl:117
[3] macro expansion
@ ~/.julia/packages/JLLWrappers/bkwIo/src/products/library_generators.jl:54 [inlined]
[4] init()
@ NetCDF_jll ~/.julia/packages/NetCDF_jll/xjKDT/src/wrappers/x86_64-apple-darwin.jl:14
[5] _include_from_serialized(path::String, depmods::Vector{Any})
@ Base ./loading.jl:768
[6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
@ Base ./loading.jl:854
[7] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1097
[8] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:1013
[9] require(into::Module, mod::Symbol)
@ Base ./loading.jl:997
[10] include
@ ./Base.jl:418 [inlined]
[11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
@ Base ./loading.jl:1318
[12] top-level scope
@ none:1
[13] eval
@ ./boot.jl:373 [inlined]
[14] eval(x::Expr)
@ Base.MainInclude ./client.jl:453
[15] top-level scope
@ none:1
during initialization of module NetCDF_jll
in expression starting at /Users/lessig/.julia/packages/NCDatasets/opjna/src/NCDatasets.jl:1
ERROR: LoadError: Failed to precompile NCDatasets [85f8d34a-cbdd-5861-8df4-14fed0d494ab] to /Users/lessig/.julia/compiled/v1.7/NCDatasets/jl_EitU8S.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
@ Base ./loading.jl:1466
[3] compilecache(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1410
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1120
[5] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:1013
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:997
[7] include
@ ./Base.jl:418 [inlined]
[8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
@ Base ./loading.jl:1318
[9] top-level scope
@ none:1
[10] eval
@ ./boot.jl:373 [inlined]
[11] eval(x::Expr)
@ Base.MainInclude ./client.jl:453
[12] top-level scope
@ none:1
in expression starting at /Users/lessig/.julia/packages/Oceananigans/ECYGS/src/Oceananigans.jl:1
ERROR: Failed to precompile Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09] to /Users/lessig/.julia/compiled/v1.7/Oceananigans/jl_QYjZa2.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
@ Base ./loading.jl:1466
[3] compilecache(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1410
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1120
[5] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:1013
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:997

[Alexander-Barth]

Can you also please paste here the results of Pkg.test("HDF5")?

[clessig]

Here you go:

julia> Pkg.test("HDF5")
Testing HDF5
Status /private/var/folders/7w/c50fl15s1f1fwbhsvgp45c840000gn/T/jl_i3h5vc/Project.toml
[a74b3585] Blosc v0.7.2
[34da2185] Compat v3.41.0
[5789e2e9] FileIO v1.11.2
[f67ccb44] HDF5 v0.15.7
[ae029012] Requires v1.2.0
[0234f1f7] HDF5_jll v1.12.0+1
[8bf52ea8] CRC32c @stdlib/CRC32c
[8ba89e20] Distributed @stdlib/Distributed
[8f399da3] Libdl @stdlib/Libdl
[37e2e46d] LinearAlgebra @stdlib/LinearAlgebra
[a63ad114] Mmap @stdlib/Mmap
[44cfe95a] Pkg @stdlib/Pkg
[9a3f8284] Random @stdlib/Random
[8dfed614] Test @stdlib/Test
Status /private/var/folders/7w/c50fl15s1f1fwbhsvgp45c840000gn/T/jl_i3h5vc/Manifest.toml
[a74b3585] Blosc v0.7.2
[34da2185] Compat v3.41.0
[5789e2e9] FileIO v1.11.2
[f67ccb44] HDF5 v0.15.7
[692b3bcd] JLLWrappers v1.3.0
[21216c6a] Preferences v1.2.3
[ae029012] Requires v1.2.0
[0b7ba130] Blosc_jll v1.21.1+0
[0234f1f7] HDF5_jll v1.12.0+1
[5ced341a] Lz4_jll v1.9.3+0
[458c3c95] OpenSSL_jll v1.1.10+0
[3161d3a3] Zstd_jll v1.5.0+0
[0dad84c5] ArgTools @stdlib/ArgTools
[56f22d72] Artifacts @stdlib/Artifacts
[2a0f44e3] Base64 @stdlib/Base64
[8bf52ea8] CRC32c @stdlib/CRC32c
[ade2ca70] Dates @stdlib/Dates
[8bb1440f] DelimitedFiles @stdlib/DelimitedFiles
[8ba89e20] Distributed @stdlib/Distributed
[f43a241f] Downloads @stdlib/Downloads
[b77e0a4c] InteractiveUtils @stdlib/InteractiveUtils
[b27032c2] LibCURL @stdlib/LibCURL
[76f85450] LibGit2 @stdlib/LibGit2
[8f399da3] Libdl @stdlib/Libdl
[37e2e46d] LinearAlgebra @stdlib/LinearAlgebra
[56ddb016] Logging @stdlib/Logging
[d6f4376e] Markdown @stdlib/Markdown
[a63ad114] Mmap @stdlib/Mmap
[ca575930] NetworkOptions @stdlib/NetworkOptions
[44cfe95a] Pkg @stdlib/Pkg
[de0858da] Printf @stdlib/Printf
[3fa0cd96] REPL @stdlib/REPL
[9a3f8284] Random @stdlib/Random
[ea8e919c] SHA @stdlib/SHA
[9e88b42a] Serialization @stdlib/Serialization
[1a1011a3] SharedArrays @stdlib/SharedArrays
[6462fe0b] Sockets @stdlib/Sockets
[2f01184e] SparseArrays @stdlib/SparseArrays
[10745b16] Statistics @stdlib/Statistics
[fa267f1f] TOML @stdlib/TOML
[a4e569a6] Tar @stdlib/Tar
[8dfed614] Test @stdlib/Test
[cf7118a7] UUIDs @stdlib/UUIDs
[4ec0a83e] Unicode @stdlib/Unicode
[e66e0078] CompilerSupportLibraries_jll @stdlib/CompilerSupportLibraries_jll
[deac9b47] LibCURL_jll @stdlib/LibCURL_jll
[29816b5a] LibSSH2_jll @stdlib/LibSSH2_jll
[c8ffd9c3] MbedTLS_jll @stdlib/MbedTLS_jll
[14a3606d] MozillaCACerts_jll @stdlib/MozillaCACerts_jll
[4536629a] OpenBLAS_jll @stdlib/OpenBLAS_jll
[83775a58] Zlib_jll @stdlib/Zlib_jll
[8e850b90] libblastrampoline_jll @stdlib/libblastrampoline_jll
[8e850ede] nghttp2_jll @stdlib/nghttp2_jll
[3f19e933] p7zip_jll @stdlib/p7zip_jll
Testing Running tests...
[ Info: libhdf5 v1.12.0
Test Summary: | Pass Broken Total
HDF5.jl | 752 1 753
Testing HDF5 tests passed

[Alexander-Barth]

Thank you for the additional info.
Maybe julia-homebrew uses an incompatible version of libmbedtls.
Can you check which version of libmbedtls your julia brew uses?

On Linux it is libmbedtls.so.13.

cd ~/opt/julia-1.7.0/lib/julia  
$ ls -l libmbedtls.so*
lrwxrwxrwx 1 abarth abarth     16 Nov 30 19:59 libmbedtls.so -> libmbedtls.so.13
lrwxrwxrwx 1 abarth abarth     20 Nov 30 19:59 libmbedtls.so.13 -> libmbedtls.so.2.24.0
-rwxr-xr-x 1 abarth abarth 315472 Nov 30 19:59 libmbedtls.so.2.24.0

Maybe this issue is related: #126

[Alexander-Barth]

Unfortunately supporting julia on homebrew is probably not possible (at least not for me). I added this to the docs:

Julia and NetCDF_jll have several common dependencies (curl, MbedTLS, zlib).
Non-official julia builds will work only if they use exactly the same library version as those used to compile NetCDF. This is unlikely to be the case in general and outside of our control. Therefore non-official julia builds are not supported.
Official julia builds are available at https://julialang.org/downloads/.