mbedtls issue in Fedora linux

[cgeoga]

With NCDatasets v0.11.3, I get the same issue as #118. I am using Fedora 33 and the julia from nalimilan's COPR repo. Here are my environment variables that you mention:

julia> ENV["PATH"]
"/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/cg/.local/bin:/home/cg/bin"

julia> ENV["LD_LIBRARY_PATH"]
ERROR: KeyError: key "LD_LIBRARY_PATH" not found
Stacktrace: [...]

julia> ENV["LD_PRELOAD"]
ERROR: KeyError: key "LD_PRELOAD" not found
Stacktrace: [...]

And here are my results from adding NetCDF_jll and trying using:

julia> using NetCDF_jll
ERROR: InitError: could not load library "/home/cg/.julia/artifacts/0a7f440ba143b238cc3dd0def2956d34d76755f3/lib/libnetcdf.so"
libmbedtls.so.13: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(s::String, flags::UInt32)
   @ Base.Libc.Libdl ./libdl.jl:114
 [2] macro expansion
   @ ~/.julia/packages/JLLWrappers/WnWcZ/src/products/library_generators.jl:63 [inlined]
 [3] __init__()
   @ NetCDF_jll ~/.julia/packages/NetCDF_jll/xjKDT/src/wrappers/x86_64-linux-gnu.jl:14
during initialization of module NetCDF_jll

As an additional data point, if I manually clone out the package, check out v0.11.0, then do ] activate . and ]instantiate, I am able to do using NCDatasets. I'm not sure what that means and to be honest I don't have a good understanding of the artifact system. But maybe that's another helpful data point.

Thanks so much for the help with this!

[Alexander-Barth]

Thanks for the detailed issue description. libmbedtls.so.13 is a file distributed with julia-1.6.0. Maybe this file is not included in nalimilan's COPR repo. Can you, as a test, try the official julia binaries from https://julialang.org/downloads/ after moving .julia to e.g. .julia_backup ?

The output of rpm -ql your-julia-packagename | grep mbedtls would also be interesting.

artifact is a way to distribute binaries (quite often precompiled C libraries) as Julia packages. The advantage over the classical linux package manager is that is cross-platform, you can have multiple version installed (to satisfy the requirement of different projects) and you do not need to be root to install libraries.

[cgeoga]

Good call all around. The official binary works without a hitch, and the rpm command you suggest shows /usr/lib64/julia/libmbedtls.so, so I presume that's the problem. I'll go bug nalimilan. Thanks again!

[nalimilan]

Indeed Fedora 33 ships libmbedtls.so.12 only and since it's relatively recent I used that in the Copr package. By default the Julia build scripts hardcodes version 13, but it's not clear whether it's intended to be exactly that version. That seems overly restrictive to me as it means one can almost never use the system library even if it's reasonably recent, making it impossible to provide distribution packages. I had commented at JuliaLang/julia#38347 (comment). @staticfloat What do you think?

[Yujie-W]

I have a similar issue, but it relates libmbedcrypto.so. The manjaro/arch delivers libmbedcrypto.so.6

julia> using NCDatasets
[ Info: Precompiling NCDatasets [85f8d34a-cbdd-5861-8df4-14fed0d494ab]
ERROR: LoadError: InitError: could not load library "/home/user/.julia/artifacts/0a7f440ba143b238cc3dd0def2956d34d76755f3/lib/libnetcdf.so"
libmbedcrypto.so.5: cannot open shared object file: No such file or directory

Here is the system file

$ ls -l /usr/lib64/julia/libmbedtls.so
/usr/lib64/julia/libmbedtls.so -> /usr/lib/libmbedtls.so.13
$ ls -l /usr/lib64/julia/libmbedcrypto.so
/usr/lib64/julia/libmbedcrypto.so -> /usr/lib/libmbedcrypto.so.6

[Alexander-Barth]

I am wondering what would happen if you install libnetcdf.so using your linux package manager and start julia as follows:

LD_PRELOAD=/path/to/package/manager/libnetcdf.so julia

Or alternatively, temporarily substitute the artifact libnetcdf.so with the corresponding file from the package manager:

mv /home/user/.julia/artifacts/HASH/lib/libnetcdf.so /home/user/.julia/artifacts/HASH/lib/libnetcdf.so.backup
ln -s /path/to/package/manager/libnetcdf.so  /home/user/.julia/artifacts/HASH/lib/libnetcdf.so 

Do you get an error message while loading NetCDF_jll or NCDatasets ? Can you send me output of ]test NCDatasets ?

[staticfloat]

The issue here isn't what is hardcoded into Julia; it's that the libnetcdf.so library itself is requesting a particular SOVERSION of libmbedtls.so. From the outside, it is very difficult to know why that is; it could be overly restrictive, or libnetcdf.so could be using features that only exist within libmbedtls.so.13 and do not exist within libmbedtls.so.12. We do our best to synchronize things within Yggdrasil such that you get the correct versions of things, but when Julia comes from a distro package, sometimes these assumptions get broken. I do not believe there is a general solution to this; the best thing I can come up with is to provide libnetcdf.so through your distro packager as well.

You can override which libnetcdf gets loaded through Preferences: you'd do something like this:

# We need a currently-unreleased version of JLLWrappers (once v1.3.0 is released, this will no longer be necessary)
Pkg.add(;name="JLLWrappers", rev="master")

# Load Preferences and NetCDF_jll
using Preferences, NetCDF_jll

# Configure NetCDF_jll to load a different path
set_preferences!(NetCDF_jll, "libnetcdf_path" => "/usr/lib/x86_64-linux-gnu/libnetcdf.so.13")

This will create a LocalPreferences.toml file in your top-level project that has an entry like the following:

[NetCDF_jll]
libnetcdf_path = "/usr/lib/x86_64-linux-gnu/libnetcdf.so.13"

If you "export" the preferences (by adding export_prefs=true to set_preferences!()), this will instead be put within your top-level Project.toml rather than its own file. Doing this will tell the NetCDF_jll package to load that library instead of the one from the current JLL. You will need to do this for each troublesome library that relies on versions of libraries that differ from the default Julia distribution.

[nalimilan]

Ah right it's always a problem when mixing system and Yggdrasil libraries. Would it be possible to get another copy of mbedtls from Yggrasil? Or would that create conflicts?

(FWIW, the Fedora libgit2 package links to libssl.so, so actually mbedtls may not even have to be loaded.)

[staticfloat]

Can you confirm that doing some Pkg operations doesn't pull in libmbedtls?

I believe HTTP explicitly relies on mbedtls though, so you can still run into ecosystem problems, perhaps.

[nalimilan]

There's probably a better way to check that, but I went the brute force way by moving /lib64/libmbedtls.so.12 out of the way. ]up went fine. But yes, using HTTP fails as it can't find the library.

[staticfloat]

Would it be possible to get another copy of mbedtls from Yggrasil? Or would that create conflicts?

Hmmm. Do your builds of Julia contain an MbedTLS_jll at all? If base Julia doesn't actually depend on libmbedtls.so at all, you could try removing MbedTLS_jll from the stdlib directory, which may convince Pkg to actually download and install the MbedTLS_jll that we publish, but this is truly untested waters.

[nalimilan]

Yes, using MbedTLS_jll works fine. If I remove its directory, ]add MbedTLS works and using HTTP also works. So it indeed looks like this approach could work. But then users would need to install it manually.

[leifdenby]

Or alternatively, temporarily substitute the artifact libnetcdf.so with the corresponding file from the package manager:

mv /home/user/.julia/artifacts/HASH/lib/libnetcdf.so /home/user/.julia/artifacts/HASH/lib/libnetcdf.so.backup
ln -s /path/to/package/manager/libnetcdf.so  /home/user/.julia/artifacts/HASH/lib/libnetcdf.so 

Do you get an error message while loading NetCDF_jll or NCDatasets ? Can you send me output of ]test NCDatasets ?

I did this and it worked for me. On the HPC system I use netcdf is provided by the "modules" utility (module load ...). Thanks for making this suggestion @Alexander-Barth. It sounds like this is a tricky issue to resolve more generally.

[Alexander-Barth]

@leifdenby I am wondering if you tried also to use @staticfloat solution. The symbolic link need to be remade should you upgrade NetCDF_jll. The set_preferences! approach looks like a more durable solution to me.

For what it is worth, I also use a HPC system with modules. I did not have any problem with NCDatasets, but I have this (minor) issue described here. Essentially, when the LD_LIBRARY_PATH is set globally (as it is usually the case for environment modules), the risk of loading an incompatible library is quite high. I found this post quite useful, in particular the suggestion to limit the scope of LD_LIBRARY_PATH and modules by using wrapper scripts.

[Alexander-Barth]

I am closing the issue because it is problem of library synchronization between julia and Fedora independent of NCDatasets.jl.