On Mac - ERROR: LoadError: could not load symbol "gzopen64": dlsym(0xfff147d636b0, gzopen64): symbol not found

[logankilpatrick]

Per here: JuliaML/MLDatasets.jl#86 it looks like this is failing on macOS Monterey. If I run test GZip in Julia 1.6 I get the same error as I got in the issue linked above:

ERROR: LoadError: could not load symbol "gzopen64":
dlsym(0xfff147d636b0, gzopen64): symbol not found
Stacktrace:
 [1] gzopen(fname::String, gzmode::String, gz_buf_size::Int64)
   @ GZip ~/.julia/packages/GZip/JNmGn/src/GZip.jl:249
 [2] gzopen(fname::String, gzmode::String)
   @ GZip ~/.julia/packages/GZip/JNmGn/src/GZip.jl:263
 [3] top-level scope
   @ ~/.julia/packages/GZip/JNmGn/test/runtests.jl:39
 [4] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [5] top-level scope
   @ none:6
in expression starting at /Users/logankilpatrick/.julia/packages/GZip/JNmGn/test/runtests.jl:30
ERROR: Package GZip errored during testing

[linearray]

I don't think this is fixed.

The Apple-provided libz for whatever reason does not provide gzopen64, only gzopen:

> grep gzopen /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/lib/libz.1.tbd 
                       _gzgetc, _gzgetc_, _gzgets, _gzoffset, _gzopen, _gzprintf, 

In https://github.com/JuliaIO/GZip.jl/blob/57cff17fe791e6befdb544922fecc0601144c513/src/gz.jl#L158C34-L158C34 there is a check if the gzopen64 symbol can be found, otherwise gzopen is used.

But it's no use, because here https://github.com/JuliaIO/GZip.jl/blob/57cff17fe791e6befdb544922fecc0601144c513/src/lib/aarch64-apple-darwin20.jl#L481C24-L481C24 gzopen is set to use gzopen64 once again.

The result is that GZip fails on macOS:

ERROR: could not load symbol "gzopen64":
dlsym(0x3a6260d58, gzopen64): symbol not found
Stacktrace:
  [1] gzopen64
    @ ~/.julia/packages/GZip/I6uhZ/src/lib/aarch64-apple-darwin20.jl:92 [inlined]
  [2] gzopen(fname::String, gzmode::String, gz_buf_size::Int64)
    @ GZip ~/.julia/packages/GZip/I6uhZ/src/gz.jl:210
  [3] gzopen
    @ ~/.julia/packages/GZip/I6uhZ/src/gz.jl:226 [inlined]
  [4] open
    @ ~/.julia/packages/GZip/I6uhZ/src/gz.jl:233 [inlined]

Workaround is installing zlib from homebrew and starting julia with DYLD_LIBRARY_PATH=/opt/homebrew/opt/zlib/lib.

[ViralBShah]

Julia should really be using its own gzip and not the apple provided one. Are you using the latest GZip.jl? I don't have this problem and I have the same architecture. Do you have some other env variables that are interfering perhaps?

Now that we are building arch-specific binaries, we should not need the first check - and get rid of the _gzopen etc. But if we do that, it will always use gzopen64 on mac arm64.

Just for fun, I have a branch to try out zlib-ng, which may be a better solution.

[ViralBShah]

@linearray Can you try the latest master? Also, we explicitly use Zlib_jll.libz_path in every ccall statement. What is pulling in the system zlib here? Can you check you have no other LD_LIBRARY_PATH or DYLD_LIBRARY_PATH?

This works fine on my arm64 mac.

[linearray]

I run GZip.jl 0.6.0, installed julia 1.9.2 from homebrew and don't use any relevant envvars.

julia> import Zlib_jll

julia> Zlib_jll.libz_path
"/usr/lib/libz.1.dylib"

Apparently the problem sits deeper. What it the value of Zlib_jll.libz_path on your system?

[ViralBShah]

This is what it should be for everyone using official binaries/build process:

julia> Zlib_jll.libz_path
"/Users/viral/.julia/juliaup/julia-1.9.2+0.aarch64.apple.darwin14/lib/julia/libz.1.2.13.dylib"

Are you using Julia binaries that are provided by homebrew (not cask) which links to system zlib?

[linearray]

Correct, I just did brew install julia.

So is this in fact a homebrew bug?

[ViralBShah]

Yes, homebrew likes to build Julia differently than the official distribution, reusing libraries built by homebrew and system libraries. I would recommend using juliaup or install the official binaries directly.

We could possibly add a check for this case and perhaps issue a warning. The other possibility is to use zlib-ng (#104), which I am very close to and will avoid this altogether.