cmd/link: would like a way to emit DT_RPATH tags (instead of DT_RUNPATH)

[mwhudson]

What version of Go are you using (go version)?

Anything (since the linker's -r flag was added in 2010 anyway)

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Ubuntu 20.04, but really: any ELF system.

What did you do?

I maintain the Go snap, which is a "classic snap": classic snaps run in the same mount namespace as everything else but load dynamic libraries from a "core" snap, which is mounted at a path like /snap/core18/current. To do this, they need to have a DT_RPATH like /snap/core18/current/lib/x86_64-linux-gnu. The linker's -r option causes a DT_RUNPATH tag to be used, which doesn't work for the reasons explained in https://www.qt.io/blog/2011/10/28/rpath-and-runpath. (It almost works, but when glibc tries to load a NSS component, it loads it from the host not the core snap and that can break).

It would be a trivial change to the linker to add an option "-rpath-instead-of-runpath" or whatever, but would that be acceptable?

[ianlancetaylor]

As far as I can tell, this is only going to be an issue when linking against C shared libraries, which means when using the C linker. Therefore, the go build option you want is -ldflags=-extldflags=--disable-new-dtags.

[ianlancetaylor]

Or am I confused about this? Does this somehow arise with a pure Go binary?

[mwhudson]

Sure, it breaks the "go" binary:

mwhudson@anduril:~/tmp/nssargh$ /snap/go/7350/bin/go get -v github.com/canonical/pebble 
/snap/go/7350/bin/go: relocation error: /lib/x86_64-linux-gnu/libnss_files.so.2: symbol __libc_readline_unlocked version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
mwhudson@anduril:~/tmp/nssargh$ readelf -d /snap/go/7350/bin/go

Dynamic section at offset 0x989140 contains 20 entries:
  Tag        Type                         Name/Value
 0x0000000000000004 (HASH)               0xb327e0
 0x0000000000000006 (SYMTAB)             0xb32d20
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0xb32aa0
 0x000000000000000a (STRSZ)              624 (bytes)
 0x0000000000000007 (RELA)               0xb323c0
 0x0000000000000008 (RELASZ)             24 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000001d (RUNPATH)            Library runpath: [/snap/core18/current/lib/x86_64-linux-gnu]
 0x0000000000000003 (PLTGOT)             0xd89020
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000006ffffffe (VERNEED)            0xb32760
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0xb32700
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000002 (PLTRELSZ)           792 (bytes)
 0x0000000000000017 (JMPREL)             0xb323d8
 0x0000000000000000 (NULL)               0x0
mwhudson@anduril:~/tmp/nssargh$ ./go.handhacked get -v github.com/canonical/pebble 
mwhudson@anduril:~/tmp/nssargh$ readelf -d ./go.handhacked 

Dynamic section at offset 0x989140 contains 20 entries:
  Tag        Type                         Name/Value
 0x0000000000000004 (HASH)               0xb327e0
 0x0000000000000006 (SYMTAB)             0xb32d20
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0xb32aa0
 0x000000000000000a (STRSZ)              624 (bytes)
 0x0000000000000007 (RELA)               0xb323c0
 0x0000000000000008 (RELASZ)             24 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000000f (RPATH)              Library rpath: [/snap/core18/current/lib/x86_64-linux-gnu]
 0x0000000000000003 (PLTGOT)             0xd89020
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000006ffffffe (VERNEED)            0xb32760
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0xb32700
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000002 (PLTRELSZ)           792 (bytes)
 0x0000000000000017 (JMPREL)             0xb323d8
 0x0000000000000000 (NULL)               0x0

("go.handhacked" is a copy of the same binary where I used a hex editor to change the DT_RUNPATH to a DT_RPATH)

[ianlancetaylor]

OK, so we would need this new option only for building the go binary itself. And we would need some way to build the go binary with the new option.

The difference between DT_RUNPATH and DT_RPATH is that DT_RPATH applies before LD_LIBRARY_PATH and DT_RUNPATH applies after. Also, the dynamic linker will ignore DT_RPATH in a shared library, and will instead use DT_RPATH from the main executable to search for indirectly linked shared libraries, which I guess is the feature that you are looking for. The ELF ABI more or less considers DT_RPATH to be deprecated in favor of DT_RUNPATH, so it would be a little bit weird for us to start generating it.

What if you replace the go tool with a tiny shell script

LD_LIBRARY_PATH="path-you-want" exec real-go $*

?

[mwhudson]

OK, so we would need this new option only for building the go binary itself. And we would need some way to build the go binary with the new option.

What I am doing today is setting GO_LDFLAGS to "-I {interp} -r {rpath}" when invoking ./make.bash and this works fine, apart from the RPATH/RUNPATH thing.

'pprof' and 'trace' are the other dynamic executables in the snap today.

The difference between DT_RUNPATH and DT_RPATH is that DT_RPATH applies before LD_LIBRARY_PATH and DT_RUNPATH applies after. Also, the dynamic linker will ignore DT_RPATH in a shared library, and will instead use DT_RPATH from the main executable to search for indirectly linked shared libraries, which I guess is the feature that you are looking for.

Right, this is about dlopen and friends.

The ELF ABI more or less considers DT_RPATH to be deprecated in favor of DT_RUNPATH, so it would be a little bit weird for us to start generating it.

Yeah but this attitude seems to fly in the face of reality a bit, unfortunately. They don't do the same thing!

What if you replace the go tool with a tiny shell script

LD_LIBRARY_PATH="path-you-want" exec real-go $*

?

That could cause problems when go tries to execute the host compiler or linker. I think I could replace go (and trace and pprof) with shell scripts that run "/snap/core18/current/lib64/ld-linux-x86-64.so.2 --library-path /snap/core18/current/lib/x86_64-linux-gnu $realbinary".

[ianlancetaylor]

Since we have an option -rpath that sets DT_RUNPATH I guess that we should add an option -runpath that sets DT_RPATH.

[mwhudson]

That's the host linker I think :-) cmd/link just takes -r, nicely ambiguous.

FWIW I have a workaround for now but it's pretty ghastly.

[ianlancetaylor]

Ah, right, sorry.