Use patchelf to set RPATH

[sjackman]

Nicolas Hillegeer wrote...

Hey!

I just tried to post an issue to the issue tracker but it seems it won't budge, so I'll just paste my findings over to you:

NOTE: this issue might be double-posted, I think there was some issue with github so it didn't appear for me...

I really wanted ag, the silver searcher and I really wanted it through homebrew because I like pretty colours and upgrades, and dependency building, and rainbows.

I basically installed with

git clone git://github.com/Homebrew/linuxbrew.git
sudo mv linuxbrew /opt/homebrew
sudo chown `whoami`:`whoami` /opt/homebrew -R
edit [.zshrc/.bashrc/...] and add /opt/homebrew/bin in front of $PATH

So, that left me with three tiny hacks to get it to work:

1. Setup more environment variables in .zshrc, which basically comes down to

export PATH="/opt/homebrew/bin:$PATH"                  # already done earlier
export HOMEBREW_PREFIX="/opt/homebrew"
export HOMEBREW_REPOSITORY="/opt/homebrew"
export HOMEBREW_CELAR="/opt/homebrew/Cellar"
export HOMEBREW_CACHE="/opt/homebrew/hbcache"

2. to fix brew doctor, it seems to assume mktemp is in/usr/bin/mktemp, on debian it's in /bin/mktemp

Success!!! ag installs, drags in pcre kicking and screaming, and I am the proud father of a new compiled silver searcher.

But wait... it says that it can't find libpcre.so.1 when it runs. But I do have it, it got installed to /opt/homebrew/lib as part of the dependency resolving phase. What do we do?

3. The first reflex would be to add /opt/homebrew/lib to LD_LIBRARY_PATH. Which would probably cause weird troubles for users down the road. There are some articles out there detailing the pitfalls of LD_LIBRARY_PATH. Especially on a linux system such as debian, I just wouldn't do it.

But, is there a way to change the LD_LIBRARY_PATH for just one executable? Well yea, we could write a wrapper script! There are a lot of packages that do that. And it's bad, but not quite as evil as changing LD_LIBRARY_PATH. So, I could hack homebrew to have it generate wrappers instead of symlinks.

But that's so dirty...

And I'm so lazy...

And I don't know ruby...

So I bust out google-fu and find that in fact, there is a solution that works for many unices. Basically, every executable has the possibility to have a RPATH and a RUNPATH. It's basically exactly what we want, the runtime linker will look first in RPATH, then in the normal path and last but not least in RUNPATH. At least that's my understanding.

But I find that you can set this variable at link-time, with a switch to ld or gcc. :(

That would mean hacking homebrew again.

I google once more before giving up and I find the venerable patchelf. Which... patches elfs to have a different RUNPATH. Exactly what we need.

cd /opt/homebrew/bin
patchelf --set-rpath '$ORIGIN/../lib' ag                   # $ORIGIN refers to the current directory, don't evaluate it, use the single quotes

Does it work? Why yes, it works. Now I can run homebrew on my debian wheezy. Nice.

[aktau]

Ok forgot to mention that I also had to adjust ENV.rb to look for gcc and g++ instead of gcc-4.2 and g++-4.2. Seems to work nicely.

[sjackman]

@aktau The compiler name issue (gcc/g++ vs gcc-4.2/g++-4.2) is now fixed.

[sjackman]

@aktau Hi Nicolas,

Homebrew is usually installed in /usr/local, and on OS X /usr/local/lib is in the default library search path. When I install Homebrew in my home directory, I set LD_LIBRARY_PATH to $HOME/local/lib.

On Linux, /usr/local/lib is not in the default library search path. One simple solution is to add the Homebrew library path to the ld.so config file, /etc/ld.so.conf.

Cheers,
Shaun

[aktau]

Actually, we could also just set it as a flag to gcc if and when that compiler is used, to obviate the need for patchelf:

LDFLAGS += -Wl,-rpath,/path/to/homebrew/lib

Actually I'm not entirely sure if the -Wl, is necessary when it's put in LDFLAGS...

[sjackman]

This issue affects the master branch of Homebrew just as well as Linuxbrew. Would you open an issue at http://github.com/mxcl/homebrew? It would be good to see it solved upstream.

[wildmichael]

The problem discussed here is IMHO actually a bit more complex, and not yet resolved (despite the referenced issue #9 being marked as closed). Quite many libraries are keg-only, and hence dependent binaries must include ${HOMEBREW_PREFIX}/opt/<dep_pkg>/lib in their RPATH. Take for example the fontconfig package, which depends on the freetype package, which in turn depends on libpng. All of them are keg-only. Linking the programs in fontconfig (without any modifications) fails, because libpng16.so.16, required by libfreetype.so.6, cannot be found.

The hackish solution is to update every single formula to add -Wl,-rpath,... flags to LDFLAGS for every single dependency, which is frankly quite unmaintainable. However, I don't see how this could be handled automagically right now.

[mistydemeo]

The hackish solution is to update every single formula to add -Wl,-rpath,... flags to LDFLAGS for every single dependency, which is frankly quite unmaintainable.

Does linuxbrew use superenv? If so, it wouldn't be difficult to add to flags automatically based on the existing code there.

[wildmichael]

Well, of course {home,linux}brew might just do it based on the listed dependencies. Would that be safe?

[sjackman]

@mistydemeo Linuxbrew does not currently use superenv. It would require a few patches to Library/ENV to get it going, I think.

@themiwi How does this work on Mac OS?

[wildmichael]

@sjackman It's been a while I used Mac OS, but if I remember correctly, each dylib contains an embedded path (the so-called install_name, something like the SONAME on Linux, but usually it is absolute), and this is then used when linking. Hence, (almost) every binary references its libraries via an absolute path, and thus the issue does not exist. I think you can dump this info using otool -L <somefile> and can be modified using the install_name_tool.

[aktau]

Just in case anyone was still wondering why changing the LD_LIBRARY_PATH is not necessarily the best solution on linux (this is one of the less subtle issues):

[wildmichael]

@aktau Absolutely, I stumbled across this issue myself just recently (both with ViM and also with GTK+ programs no longer working properly because those got linked).

For linuxbrew it probably is almost always a bad idea to link /lib/ and to modify LD_LIBRARY_PATH. But this would require to extend the rpath accordingly.

[sjackman]

@aktau @themiwi Could you please test the branch bottle. It uses patchelf to set the RPATH. If the package glibc is installed, it also set the dynamic loader. Be sure to brew install patchelf. With this work, setting LD_LIBRARY_PATH should no longer be necessary.

I'm using this patch to implement portable bottles on Linux. I've tested building a bottle on Ubuntu 14.04 and running it on CentOS 5.10, and it works! I'm pretty happy with that result.

Cheers,
Shaun

[sjackman]

I'd appreciate your help if you could also test brew install glibc.

[aktau]

I'd appreciate your help if you could also test brew install glibc.

Don't have any systems I can safely mess up at the moment.

I'm using this patch to implement portable bottles on Linux. I've tested building a bottle on Ubuntu 14.04 and running it on CentOS 5.10, and it works! I'm pretty happy with that result.

Pretty great, but it's still risky unless all libraries have been source from homebrew's own I guess. I like the rpath fix much more for not having to mess with my LD_LIBRARY_PATH. I have no problem compiling my own stuff (little need for bottles).

Could you details a few steps for how I would go about testing this? I'm not a homebrew/ruby expert by any stretch of the imagination.

[sjackman]

Hi, Nicolas. Checkout the bottle branch of Linuxbrew cd $HOME/.linuxbrew; git checkout bottle, install patchelf brew install patchelf, and from then on any executables installed with Homebrew will have their RPATH set to $HOMEBREW_PREFIX/lib. Here's the relevant commit:
34b4c36#diff-9f183506315342e7eb6e1ab973ea1996R66

[aktau]

Apparently it didn't work:

➜  .linuxbrew git:(34b4c36) cd
➜  ~  brew install ag
==> Installing dependencies for the_silver_searcher: autoconf, automake, pkg-config, pcre, xz
==> Installing the_silver_searcher dependency: autoconf
==> Downloading http://ftpmirror.gnu.org/autoconf/autoconf-2.69.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/player/.linuxbrew/Cellar/autoconf/2.69
==> make install
==> Caveats
This formula is keg-only, so it was not symlinked into /home/player/.linuxbrew.

Xcode provides this software prior to version 4.3.
==> Summary
/home/player/.linuxbrew/Cellar/autoconf/2.69: 66 files, 3.2M, built in 11 seconds
==> Installing the_silver_searcher dependency: automake
==> Downloading http://ftpmirror.gnu.org/automake/automake-1.14.1.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/player/.linuxbrew/Cellar/automake/1.14.1
==> make install
==> Caveats
This formula is keg-only, so it was not symlinked into /home/player/.linuxbrew.

Xcode provides this software prior to version 4.3.
==> Summary
/home/player/.linuxbrew/Cellar/automake/1.14.1: 131 files, 3.3M, built in 11 seconds
==> Installing the_silver_searcher dependency: pkg-config
==> Downloading http://pkgconfig.freedesktop.org/releases/pkg-config-0.28.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/player/.linuxbrew/Cellar/pkg-config/0.28 --disable-host-tool --with-internal-glib --with-pc-path=/home/player/.linuxbrew/lib/pkgconfig:/home/player/.linuxbrew/sha
==> make
==> make check
==> make install
/home/player/.linuxbrew/Cellar/pkg-config/0.28: 10 files, 644K, built in 96 seconds
==> Installing the_silver_searcher dependency: pcre
==> Downloading https://downloads.sourceforge.net/project/pcre/pcre/8.35/pcre-8.35.tar.bz2
######################################################################## 100.0%
==> ./configure --prefix=/home/player/.linuxbrew/Cellar/pcre/8.35 --enable-utf8 --enable-pcre8 --enable-pcre16 --enable-pcre32 --enable-unicode-properties --enable-pcregrep-libz --enable-pcregr
==> make
==> make test
==> make install
/home/player/.linuxbrew/Cellar/pcre/8.35: 146 files, 5.5M, built in 2.2 minutes
==> Installing the_silver_searcher dependency: xz
==> Downloading http://fossies.org/linux/misc/xz-5.0.5.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/player/.linuxbrew/Cellar/xz/5.0.5
==> make install
/home/player/.linuxbrew/Cellar/xz/5.0.5: 63 files, 1.7M, built in 30 seconds
==> Installing the_silver_searcher
==> Downloading https://github.com/ggreer/the_silver_searcher/archive/0.23.0.tar.gz
######################################################################## 100.0%
==> Downloading https://github.com/thomasf/the_silver_searcher/commit/867dff8631bc80d760268f653265e4d3caf44f16.diff
######################################################################## 100.0%
==> Patching
patching file ag.bashcomp.sh
==> aclocal -I /home/player/.linuxbrew/share/aclocal
==> autoconf
==> autoheader
==> automake --add-missing
==> ./configure --prefix=/home/player/.linuxbrew/Cellar/the_silver_searcher/0.23.0
==> make
==> make install
==> Caveats
Bash completion has been installed to:
  /home/player/.linuxbrew/etc/bash_completion.d
==> Summary
/home/player/.linuxbrew/Cellar/the_silver_searcher/0.23.0: 8 files, 148K, built in 12 seconds
You have new mail.
➜  ~  which ag
/home/player/.linuxbrew/bin/ag
➜  ~  ag -v
ag: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory

This was on a fresh linuxbrew install. Not sure what I did wrong.

[aktau]

Reported too soon. I had forgotten to install patchelf. New results: works perfectly!

➜  ~  brew install patchelf
==> Downloading https://github.com/NixOS/patchelf/archive/0.8.tar.gz
######################################################################## 100.0%
==> ./bootstrap.sh
==> ./configure --disable-silent-rules --prefix=/home/player/.linuxbrew/Cellar/patchelf/0.8
==> make install
/home/player/.linuxbrew/Cellar/patchelf/0.8: 6 files, 172K, built in 11 seconds
➜  ~  brew remove ag
Uninstalling /home/player/.linuxbrew/Cellar/the_silver_searcher/0.23.0...
➜  ~  brew install ag
==> Downloading https://github.com/ggreer/the_silver_searcher/archive/0.23.0.tar.gz
Already downloaded: /home/player/.cache/Homebrew/the_silver_searcher-0.23.0.tar.gz
==> Downloading https://github.com/thomasf/the_silver_searcher/commit/867dff8631bc80d760268f653265e4d3caf44f16.diff
Already downloaded: /home/player/.cache/Homebrew/the_silver_searcher--patch-09502c60a11658d9a08a6825e78defad96318bd9.diff
==> Patching
patching file ag.bashcomp.sh
==> aclocal -I /home/player/.linuxbrew/share/aclocal
==> autoconf
==> autoheader
==> automake --add-missing
==> ./configure --prefix=/home/player/.linuxbrew/Cellar/the_silver_searcher/0.23.0
==> make
==> make install
==> Caveats
Bash completion has been installed to:
  /home/player/.linuxbrew/etc/bash_completion.d
==> Summary
/home/player/.linuxbrew/Cellar/the_silver_searcher/0.23.0: 8 files, 152K, built in 8 seconds
➜  ~  ag -v
ERR: What do you want to search for?
➜  ~  which ag
/home/player/.linuxbrew/bin/ag

[aktau]

@sjackman need any more input? It seems to work fine here.

About the only suggestion I'd make is to make it more obvious that patchelf needs to be installed for correct operation without altering LD_LIBRARY_PATH.

In fact, if it were up to me, I'd make patchelf mandatory. Linux is not OSX, and I firmly believe patchelf is a superior solution for Linux systems.

[sjackman]

I've create a new wiki page with instructions for a Standalone Installation of Linuxbrew. Would you be able to test it out?

[sjackman]

@aktau With the recent commits to bottle branch, patchelf is no longer necessary. Setting the RPATH and dynamic linker is handled using LDFLAGS -rpath and --dynamic-linker. The patchelf code remains as a backup plan for builds that ignore or reset LDFLAGS.

[aktau]

Very nice :). It's good that there's a fallback to patchelf. Especially when/if there are toolchains that don't have the concept of LDFLAGS but do generate elf binaries that need to dynamically link with something.