New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpath problem with librocksdb on macOS #182

Closed
rsmmr opened this Issue Sep 30, 2018 · 7 comments

Comments

Projects
None yet
3 participants
@rsmmr
Copy link
Member

rsmmr commented Sep 30, 2018

I've got RocksDB installed via MacPorts in /opt/local. I'm building Bro without any special configure options, and with prefix /opt/bro-demo. Configure finds RocksDB fine and the build links against it. Then I do make install. Now when I run the resulting binary /opt/bro-demo/bin/bro binary, I get:

dyld: Library not loaded: librocksdb.5.14.dylib
  Referenced from: /opt/bro-demo/bin/bro
  Reason: image not found
Abort trap: 6

I can fix that by setting DYLD_LIBRARY_PATH to /opt/local/lib. However, that environment variable doesn't propagate to subprocess on macOS, meaning that when btest spawns bro from that location, it runs into the same problem.

I believe rocksdb is the only library that has that problem, and it's probably because of how it's linker path differs from others:

# otool -L  /opt/bro-demo/bin/bro 
/opt/bro-demo/bin/bro:
	/usr/lib/libpcap.A.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libbroker.0.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libcaf_core.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	@rpath/libcaf_io.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	@rpath/libcaf_openssl.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	librocksdb.5.14.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)

@rsmmr rsmmr added the Area: Broker label Sep 30, 2018

@rsmmr

This comment has been minimized.

Copy link
Member

rsmmr commented Sep 30, 2018

And this indeed fixes it for the installed binary:

# install_name_tool -change librocksdb.5.14.dylib /opt/local/lib/librocksdb.5.14.dylib /opt/bro-demo/lib/libbroker.0.dylib
@jsiwek

This comment has been minimized.

Copy link
Member

jsiwek commented Oct 1, 2018

Likely you could also correct the install_name directly in the librocksdb.dylib:

install_name_tool -id /opt/local/lib/librocksdb.5.14.dylib /opt/local/lib/librocksdb.5.14.dylib

At least that's sufficient for me when when reproducing the problem via building rocksdb from source and installing to a non-standard path.

Makes me think this is more of an upstream problem. I expect the various build systems (autotools, cmake, etc.) usually have standard ways of setting the install_name for .dylibs, but rocksdb just has a basic Makefile that does things manually. This would be a guess at an upstream patch that sets the install_name to the correct absolute path:

diff --git a/Makefile b/Makefile
index db6099b67..f1fe65b94 100644
--- a/Makefile
+++ b/Makefile
@@ -1546,6 +1546,9 @@ install-shared: install-headers $(SHARED4)
                ln -fs $(SHARED4) $(INSTALL_PATH)/lib/$(SHARED3) && \
                ln -fs $(SHARED4) $(INSTALL_PATH)/lib/$(SHARED2) && \
                ln -fs $(SHARED4) $(INSTALL_PATH)/lib/$(SHARED1)
+ifeq ($(PLATFORM), OS_MACOSX)
+       install_name_tool -id $(INSTALL_PATH)/lib/$(SHARED4) $(INSTALL_PATH)/lib/$(SHARED4)
+endif

 # install static by default + install shared if it exists
 install: install-static

Or else it could be something for the MacPorts package maintainer to improve. As a reference, the Homebrew package for rocksdb uses an absolute path for the .dylib's install_name and works fine. Also curious that your otool output shows that linking against the OpenSSL .dylib's provided by MacPorts use an absolute path, so that's just an example of an inconsistency within MacPorts packaging, not anything that Bro is doing to handle rocksdb that is different from other dependencies.

I'd say this can be closed, unless there's ideas on how to handle it ourselves.

@rsmmr

This comment has been minimized.

Copy link
Member

rsmmr commented Oct 1, 2018

Yeah, I can see that. The only thing that struck me is that librocksdb.5.14.dylib stands out in the otool output in that it has neither an absolute path nor an @rpath. Is that something controlled on our end, or is that information pulled in from the installed library?

@0xxon

This comment has been minimized.

Copy link
Member

0xxon commented Oct 1, 2018

Also - to point out the obvious - if this is an upstream problem, we should probably file a bug with them.

@jsiwek

This comment has been minimized.

Copy link
Member

jsiwek commented Oct 1, 2018

It's the name rocksdb picked for itself at link-time via the -install_name flag:

https://github.com/facebook/rocksdb/blob/be5cc4c7b85dc48aad509deb1a285b4c556d2361/build_tools/build_detect_platform#L131
https://github.com/facebook/rocksdb/blob/be5cc4c7b85dc48aad509deb1a285b4c556d2361/Makefile#L669

If you build/install current rocksdb from source you get:

$ otool -D ~/sandbox/rocksdb/lib/librocksdb.dylib
/Users/jon/sandbox/rocksdb/lib/librocksdb.dylib:
librocksdb.5.14.dylib

And linking against that reproduces the Bro linking issue:

$ otool -L /opt/bro-demo/bin/bro
/opt/bro-demo/bin/bro:
	/usr/lib/libpcap.A.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libbroker.0.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libcaf_core.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	@rpath/libcaf_io.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	@rpath/libcaf_openssl.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	librocksdb.5.14.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)

$ /opt/bro-demo/bin/bro -e 'print "good"'
dyld: Library not loaded: librocksdb.5.14.dylib
  Referenced from: /opt/bro-demo/bin/bro
  Reason: image not found
Abort trap: 6

After patching rocksdb to name it via absolute path:

$ otool -D ~/sandbox/rocksdb/lib/librocksdb.dylib
/Users/jon/sandbox/rocksdb/lib/librocksdb.dylib:
/Users/jon/sandbox/rocksdb/lib/librocksdb.5.14.3.dylib

And Bro is happy:

$ otool -L /opt/bro-demo/bin/bro
/opt/bro-demo/bin/bro:
	/usr/lib/libpcap.A.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libbroker.0.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libcaf_core.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	@rpath/libcaf_io.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	@rpath/libcaf_openssl.0.16.0.dylib (compatibility version 0.16.0, current version 0.16.0)
	/Users/jon/sandbox/rocksdb/lib/librocksdb.5.14.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)

$ /opt/bro-demo/bin/bro -e 'print "good"'
good

Here's what Homebrew has:

$ otool -D /usr/local/lib/librocksdb.dylib
/usr/local/lib/librocksdb.dylib:
/usr/local/opt/rocksdb/lib/librocksdb.5.14.dylib

And Bro works with that, no changes needed.

So that's all why I think an action in either rocksdb or MacPorts is appropriate . Maybe try asking MacPorts first and see if they want to take it further upstream to rocksdb ?

@jsiwek

This comment has been minimized.

Copy link
Member

jsiwek commented Oct 1, 2018

The Homebrew example also worked the same way when using a custom /opt/local installation prefix, so it doesn't appear any shenanigans happened regarding some of the special/default handling of /usr/local on macOS.

@jsiwek

This comment has been minimized.

Copy link
Member

jsiwek commented Jan 11, 2019

Closing as I think the issue is upstream (or else not sure what exactly would be appropriate to change in our own config that would help).

@jsiwek jsiwek closed this Jan 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment