Incompatible version of libcurl dylib on Mac OSX

[james-d-mitchell]

I get the following on Mac OSX:

 ┌───────┐   GAP 4.10dev-1060-gec8e41a of today
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-apple-darwin17.7.0-default64
 Configuration:  gmp 6.1.2
 Loading the library and packages ...
#I  method installed for Matrix matches more than one declaration
 Packages:   Digraphs 0.13.0, GAPDoc 1.6.1, genss 1.6.5, GRAPE 4.7, IO 4.5.1,
             orb 4.8.0, PrimGrp 3.3.0, Semigroups 3.0.19, SmallGrp 1.2,
             Smallsemi 0.6.11, TransGrp 2.0.2
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> LoadPackage("packagemanager");
#W dlopen() error: dlopen(/Users/jdm/gap/pkg/curlInterface/bin/x86_64-apple-da\
rwin17.7.0-default64/curl.so, 9): Library not loaded: @rpath/libcurl.4.dylib
  Referenced from: /Users/jdm/gap/pkg/curlInterface/bin/x86_64-apple-darwin17.\
7.0-default64/curl.so
  Reason: Incompatible library version: curl.so requires version 9.0.0 or late\
r, but libcurl.4.dylib provides version 7.0.0
Error, module '/Users/jdm/gap/pkg/curlInterface/bin/x86_64-apple-darwin17.7.0-default\
64/curl.so' not found in
  if not LOAD_DYN( arg[1], false ) then
    Error( "no support for dynamic loading" );
fi; at /Users/jdm/gap/lib/files.gd:583 called from

which looks like curlInterface requires some specific version of libcurl, but that this isn't checked at compile time. So, the package compiles just fine, but this error occurs when I load the package in GAP.

[james-d-mitchell]

Hmm, I looked into this for about 30 minutes, it seems that there is some kind of mismatch between the version of curl/libcurl detected at compile time, and the dynamic module that is being loaded at run time. I'm going to change the title of the issue, since it seems you do check the version of libcurl at compile time, but I still have this issue, I haven't been able to resolve it, so can't currently use curlInterface or PackageManager.

[james-d-mitchell]

So, it seems like during configure the "correct" version of libcurl is found on my system:

checking for curl-config... /Users/jdm/anaconda/bin/curl-config
checking for the version of libcurl... 7.61.0
checking whether libcurl is usable... yes
checking for curl_free... yes
checking for curl-config... /Users/jdm/anaconda/bin/curl-config
curl found - remote install options available

but when I load GAP, the system libcurl (which is version 7.0.0) is the one that dlopen tries to link.

I don't know if this is an issue with my system or with configure in curlInterface, but I managed to get things working by doing:

export DYLD_FALLBACK_LIBRARY_PATH="/Users/jdm/anaconda/lib"

[james-d-mitchell]

Not that the above "fix" only works if I delete the system libcurl, which had some other rather dire consequences.

[mtorpey]

So you’re saying I shouldn’t merge my PR that randomly deletes system libraries until the package loads?

[james-d-mitchell]

Maybe

[mtorpey]

Markus suggested that @fingolfin might be able to shine some light on this. I'm well out of my depth. Perhaps also @ChrisJefferson?

[fingolfin]

So this usually meand that no absolute (r)path for the library was linked in, so upon loading, it'll link against the first libcurl shared library it can find. This can be resolved via suitable linker flags; if you use libtool (am on phone right now, can't check), then using -R /path/to/sharedlib/lib in the link command shouzdo the trick ... but that's hypothetical right now, didn't verify it.

[fingolfin]

BTW what is the actual problem here? Sure, the wrong dylib is loaded; but why is that a problem?

[james-d-mitchell]

CurlInterface requires a newer version of libcurl than the system version of libcurl on my computer, this is the problem. At config time the "right" version of libcurl is detected (the one I installed with conda), then the "wrong" version (the system one) is linked against at link time. So, arguable this is a problem in the autohell setup of CurlInterface.

[fingolfin]

I just noticed this ancient issue, had all but forgotten about it.

I don't believe this is an issue with the "autohell" of this package, at least I don't see how. Rather, if one version of libcurl is linked at compile time and another is linked at runtime, this hints at either an issue with the compile time dylib, or one caused by global environment variables people used to override dylib search paths. Esp. on macOS, where shared library paths are by default hard linked into executables.

E.g. for me, linking this package results in a linker command including the arguments ... -L/sw/lib /sw/lib/libcurl.dylib.

Then I can run otool -L to inspect the resulting binary:

otool -L bin/x86_64-apple-darwin18.7.0-default64-kv8/curl.so
bin/x86_64-apple-darwin18.7.0-default64-kv8/curl.so:
	/sw/lib/libcurl.4.dylib (compatibility version 11.0.0, current version 11.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)

Note the hardcoded path.

I am guessing @james-d-mitchell has long since resolved the issue and/or moved to some other system config. But just in case the issue can still be reproduced, I'd be super keen on seeing the output of otool -L there, and also the exact linker invocation when building the kernel extension of this package.

If it can't be reproduced by anybody right now, we should just close this :-)