Setting DYLD_LIBRARY_PATH causes configure to fail

[ryandesign]

I'm the maintainer of curl in MacPorts. I tried updating curl in MacPorts from 7.80.0 to 7.81.0 which failed for me on macOS 10.15.7. The error is:

checking run-time libs availability... failed
configure: error: one or more libs available at link-time are not available run-time. Libs used at link-time: -lidn2 -lpsl -lssl -lcrypto -lssl -lcrypto -lzstd  -lz

config.log shows us this is because:

configure:34022: ./conftest
dyld: Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /opt/local/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib
./configure: line 2481: 11810 Abort trap: 6           ./conftest$ac_exeext

This means that something that's being used here links with /usr/lib/libarchive.2.dylib, and /usr/lib/libarchive.2.dylib links with /usr/lib/libiconv.2.dylib:

$ otool -L /usr/lib/libarchive.2.dylib
/usr/lib/libarchive.2.dylib:
	/usr/lib/libarchive.2.dylib (compatibility version 9.0.0, current version 9.2.0)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/liblzma.5.dylib (compatibility version 6.0.0, current version 6.3.0)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

But the build system has set DYLD_LIBRARY_PATH=/opt/local/lib thereby telling the operating system that any libraries that exist in /opt/local/lib should be used instead of the ones that would normally be used (e.g. in /usr/lib). But /usr/lib/libiconv.2.dylib and /opt/local/lib/libiconv.2.dylib are not interchangeable. /usr/lib/libiconv.2.dylib is a "vendor build" and /opt/local/lib/libiconv.2.dylib is a "third party" build. They deliberately have different symbol names so that they do not collide.

I don't know what your build system is trying to do with DYLD_LIBRARY_PATH but this failure demonstrates why you must never set DYLD_LIBRARY_PATH to a directory that contains libraries that are outside your control. The only valid thing to do with DYLD_LIBRARY_PATH is to set it to a directory where you have just built your libraries that you have not installed yet but that you wish to use to run a program that you just built.

curl 7.80.0 didn't have this problem. The problem was introduced in ba0657c. Reverting that fixes the problem for me.

[ryandesign]

That doesn't fix it. You're still setting DYLD_LIBRARY_PATH to /opt/local/lib (CURL_LIBRARY_PATH) which you must not do.

What problem was ba0657c trying to solve? It mentions a "non-standard installation of openssl" but does not elaborate what that means. Could it perhaps be a broken installation of openssl that curl needn't concern itself with supporting? Any installed library that requires you to set DYLD_LIBRARY_PATH to use it is broken.

[bagder]

When you tell configure to look for a library at a custom location and it finds it to be there - it uses the linker to verify that the library is there and works to link with.

Later in the configure script, it will run an executable for some tests. When configure runs an executable the previously detected libraries need to be found at run-time as opposed to build-time. That's what it sets the variable for.

Why does it find the library in the non-standard directory /opt/local/lib and yet can't set the variable to search for it there? (Edit: The answer is of course that it then wrongly also finds other libs there that you have there but don't want to use.)

[bagder]

Any installed library that requires you to set DYLD_LIBRARY_PATH to use it is broken.

If I install a build of a library in ~/funky, how is using DYLD_LIBRARY_PATH wrong for that?

[bagder]

Your problem is probably that you're using one library in that directory but then you also have other libs in that directory that are using the same names as ones you have in another path, and you only want that one lib used from that directory.

I might be naive but that sounds like a recipe just waiting for problems like this. I don't have a short term fix for a setup like that, other than perhaps avoid having configure detect that directory specifically.

[ryandesign]

Your problem is

My problem is that the curl build system is setting DYLD_LIBRARY_PATH=/opt/local/lib. It must not do so. It causes the problem I reported.

you also have other libs in that directory

Yes, MacPorts installs libs primarily in /opt/local/lib, just as most other package managers I expect primarily install their libs in their own prefix or like a user might install several libraries in /usr/local. MacPorts has done so since 2002. It is not a problem.

a recipe just waiting for problems like this

It is the decision of the developers of libiconv that a "third party" build (such as the one in MacPorts) shall have symbols beginning with _libiconv so as not to conflict with a "vendor" iconv implementation (which may or may not be GNU libiconv) that has symbols beginning with _iconv. The difference is transparent to programs that use the iconv.h header correctly. The problem only appears here because you are subverting normal library loading by setting DYLD_LIBRARY_PATH=/opt/local/lib. Don't do that.

Later in the configure script, it will run an executable for some tests. When configure runs an executable the previously detected libraries need to be found at run-time as opposed to build-time. That's what it sets the variable for.

I do not understand why setting DYLD_LIBRARY_PATH=/opt/local/lib would be thought to be necessary to achieve that. macOS dynamic libraries do not work like Linux shared libraries. When a macOS executable is linked with a dynamic library, the library's install_name is copied into the executable. For most libraries, such as those installed by the OS or MacPorts, the install_name is most usually an absolute path, so if you link an executable with such a library, it knows where to find the library by its absolute path, no DYLD_LIBRARY_PATH needed at build time or runtime. Setting DYLD_LIBRARY_PATH at runtime tells the OS to ignore the library paths that were baked into the executable, which is not something you usually want to do, unless as I said you're trying to run an executable you just built that is linked with a library you just built whose install_name references its ultimate install location but it has not yet been installed there. I am assuming that is not the case here.

Again, I'd like to understand what specific issue the commit that introduced the problem was meant to correct.

If I install a build of a library in ~/funky, how is using DYLD_LIBRARY_PATH wrong for that?

The install_name of the library should be set to its absolute install path. Then setting DYLD_LIBRARY_PATH is not necessary. Or if you want the library and the executable linking with it to be easily relocatable there are solutions involving @executable_path, @loader_path, or @rpath which are outside the scope of the normal building of a library like curl.

[bagder]

I do not understand why

#8028 is why

[ryandesign]

I saw #8028 but it does not explain what is meant by "non-standard installation of openssl".

[bagder]

I presume that means you build openssl with something like:

./config --prefix=/home/daniel/build-openssl

[ryandesign]

I can't see how setting DYLD_LIBRARY_PATH would be necessary to support such an installation. Is there a reproduction recipe for the problem that the commit was trying to solve?

[dfandrich]

It sounds (after casually reading this thread and without any particular knowledge of Mac OS builds) that the argument here is that Mac OS automatically uses the equivalent of GNU's -rpath on binaries so setting an environment variable shouldn't be necessary to find a library. But, if that's the case, what happened in #8028 to cause that mechanism to fail?

I failed to configure 7.81.0 on macOS, with libjpeg in my case.
I'm not using MacPorts or other package managers but I have libjpeg-turbo installed under /usr/local. Had no problem with 7.80.0.

I remembered that DYLD_LIBRARY_PATH has been a source of problem from the old days. You will find examples caused by this on Google.
I may be wrong, but I personally understand that this variable is for debugging or testing purposes and should usually be avoided.

[bagder]

Does it help if we change it to use DYLD_FALLBACK_LIBRARY_PATH instead?