activate script should manage LD_LIBRARY_PATH

[teoliphant]

The activate script should check to see if LD_LIBRARY_PATH is set. If it is, then activate should pre-pend the correct /lib directory to LD_LIBRARY_PATH as well as updating PATH

[ilanschnell]

After having seen many of the problems people have been having with regard to LD_LIBRARY_PATH, I'm wondering if it was a good idea to use RUNPATH (instead of RPATH). While RPATH has been deprecated for a long time, it is still provides a more robust way for the linker to resolve libraries, as it takes precedence over the setting of LD_LIBRARY_PATH.

[asmeurer]

Is there any benefit to using both? According to http://blog.tremily.us/posts/rpath/, chrpath can't increase the length of RPATH on Linux.

[ilanschnell]

You cannot set both (RPATH and RUNPATH).

[ilanschnell]

commit 0209503 (in particular the equivalent in Anaconda (which means packages will all use RPATH (at some point))) will fix this problem

[dan-blanchard]

Is there an equivalent functionality for OS X? I just created a package for Perl's ZMQ:FFI library, and that tries to dynamically load libzmq. Unfortunately it cannot find it, even though it's inside the same conda environment. If I set DYLD_FALLBACK_LIBRARY_PATH appropriately, it works, but it would be nice if that happened automatically on environment activation.

[ilanschnell]

Unfortunately the situation is more complicated on OS X. The equivalent if LD_LIBRARY_PATH is DYLD_LIBRARY_PATH, but there is also DYLD_FALLBACK_LIBRARY_PATH and DYLD_VERSIONED_LIBRARY_PATH. You cal always check the linkage using otool -L. However, if a conda package is build correctly, it should not be necessary to set any of these. The conda-build command fix the dynamic shared library install names using install_name_tool to use the relative location with respect to @loader_path, which is the path of the executable in the bin directory.

[dan-blanchard]

Well, in this case the library is being loaded via libffi at runtime, since it's a C library being loaded by a Perl module, so it doesn't "link" in the traditional sense. libffi doesn't seem to look anywhere other than the DYLD_LIBRARY_PATH-related directories when it tries to load the library though.

This is actually the same issue that any Python module that uses ctypes should have to deal with (since that uses libffi underneath), so how do you create packages that find libraries in PREFIX/lib without requiring someone to set an environment variable themselves?

[dan-blanchard]

After I made the previous comment I looked at the recipe for pyzmq on in the conda-recipes repository, and it doesn't look like you're doing anything special for that. Maybe I just need to patch the Perl module to make it smarter. I was just hoping to avoid that.

[dan-blanchard]

Well, it looks like I can't just patch the Perl zeromq library (ZMQ::FFI) because the underlying FFI library it's using doesn't appear to support specifying the full path to the library. It will only look for libraries based on a filename, which requires that LD_LIBRARY_PATH or DYLD_FALLBACK_LIBRARY_PATH is set to the environments lib directory.

Doesn't this same sort of problem crop up with anything that uses ctypes in Python?

[asmeurer]

You can also check if @minrk does anything special with his zeromq recipe.

[minrk]

Setting [DY]LD_LIBRARY_PATH seems like the right thing to do to me. I think PyZMQ itself doesn't need it because rpath is set in a way that doesn't rely on it. But that probably shouldn't be necessary.

[dan-blanchard]

Okay, so that sounds like maybe this issue should be re-opened. What do you think @ilanschnell?

[ilanschnell]

Setting (DY)LD_LIBRARY_PATH almost never seems to be the right thing to do. The only case where one should set LD_LIBRARY_PATH is during experimental development, when it cannot be avoided otherwise. For deployment of code it should never be necessary.

[minrk]

Except the one case where it is the right thing to do is when creating environments, e.g. conda or virtualenv.

[asmeurer]

But the whole point of conda packages is that they should be relocatable, so that setting these variables should be completely unnecessary. I would rather look at how to fix/work around the libffi issues than to jump the gun with setting LD_LIBRARY_PATH in activate.

The discussion at conda/conda-build#50 is also relevant here. As soon as we start adding magic to activate, then it becomes something that you have to do to use an environment, which causes them to lose some of their current power.

[minrk]

It shouldn't affect conda packages themselves, but it would benefit things compiled into the env by other means. This seems like the whole point of envs, to me. Do you not want libraries in a conda env to be linkable by default while the env is active?

[minrk]

I do agree that conda packages themselves should not rely on LD_LIBRARY_PATH being set, but that doesn't mean that activate shouldn't set it.

[dan-blanchard]

@asmeurer That's a good point that I hadn't thought of before. After further examination, it does indeed look like Perl's FFI::Raw library actually is supposed to be able to handle absolute paths, but I couldn't get that to work when I tried it before. I'll take another stab at it.

[ilanschnell]

@minrk the problem with activate setting LD_LIBRARY_PATH (even when conda packages themselves don't need it) is that it might break other things on the users system.

[nnashok]

Another valid scenario is when you have a software collection (e.g rh-nodejs4 on RHEL) which require setting the LD_LIBRARY_PATH for its programs to work. Now when I run condo to build a package that depends on this software collection, the LD_LIBRARY_PATH is being unset causing the build to fail. Specifically:
My package requires rh-nodejs4. rh-nodejs4 is installed as a software collection (scl, https://www.softwarecollections.org/). Even after I activate the collection, when I try to build my package, condo is unsetting the LD_LIBRARY_PATH causing the build to fail.

Please correct me if I'm looking in the wrong direction.

[github-actions]

Hi there, thank you for your contribution to Conda!

This issue has been automatically locked since it has not had recent activity after it was closed.

Please open a new issue if needed.