xrootd: fix plugin location using RPATH

[veprbl]

Description of changes

Closes: #200748
Resolves: #169677

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[ShamrockLee]

This PR will (also) fix #169677 if applied.

[ShamrockLee]

Regrettably, xrdcp built with nixpkgs-review still doesn't work on x86_64-linux (NixOS).

[ShamrockLee]

Update:

LD_LIBRARY_PATH="/nix/store/1awy5ywcl0bs38hjxaqqxsa2qjmh0z23-xrootd-5.5.1/lib${LD_LIBRARY_PATH:+:}${LD_LIBRARY_PATH-}" xrdcp root://eospublic.cern.ch//eos/opendata/cms/Run2011A/ElectronHad/AOD/12Oct2013-v1/20001/001F9231-F141-E311-8F76-003048F00942.root /dev/shm

does work.

That's probably why ROOT has setxrd.sh and setxrd.csh in its $out/bin.

[ShamrockLee]

How about creating something like wrapXrootdHook instead?

Update: or just ship xrootd with a setuphook containing the bash array xrootdWrapperArgs.

[veprbl]

I think RPATH should work for this case.

[veprbl]

@ShamrockLee Can you post some logs from LD_DEBUG=all?

[ShamrockLee]

@veprbl https://gist.github.com/ShamrockLee/1d8975be628b943ff04fd8a0439f2dfc#file-xrdcp_pr_200830_543c75d-err

LD_DEBUG=all xrdcp root://eospublic.cern.ch//eos/opendata/alice/2010/LHC10h/000138275/ESD/0000/AliESDs.root /dev/shm/ 2> /dev/shm/xrdcp_pr_200830_.err

[veprbl]

@ShamrockLee Thank you. Just to make sure, could it be truncated? I can't find a mention of XrdSec

[ShamrockLee]

@veprbl Here's the log truncated with grep -n XrdSec
https://gist.github.com/ShamrockLee/1d8975be628b943ff04fd8a0439f2dfc#file-xrdcp_pr_200830_543c75d_grep_n_xrdsec-err

[veprbl]

And what about something like

readelf -d /nix/store/6pzl0ws9jhn09cvg86fakqqglyaivxi0-xrootd-5.5.0/{bin,lib}/* | grep RPATH

?

[ShamrockLee]

Using the readelf executable from package binutils, the result of

readelf -d "$(nix path-info .#xrootd.bin)/bin"/* | grep RPATH
readelf -d "$(nix path-info .#xrootd.out)/lib"*/* | grep RPATH

is empty. (with only warnings that sub-directories like systemd and tempfiles.d aren't regular files)

[veprbl]

Maybe it's "RUNPATH"?

[ShamrockLee]

https://gist.github.com/ShamrockLee/1d8975be628b943ff04fd8a0439f2dfc#file-xrdcp_pr_200830_543c75d_grep_n_xrdsec-err
https://gist.github.com/ShamrockLee/1d8975be628b943ff04fd8a0439f2dfc#file-xrdcp_pr_200830_543c75d_lib_readelf_grep_runpath-out

The xrootd library path is there. It can even be seen directly by viewing the executable inside Vim and type slash to search.

[veprbl]

I see that there are few lines where the xrootd is not in RUNPATH. I wonder which binaries they correspond to.

Perhaps it might be fixed by switching from RUNPATH to a global RPATH by passing a --disable-new-dtags compilation flag. Or, a more proper fix would be to ensure that there is no missing entry in RUNPATH.

[veprbl]

reply to @ShamrockLee #285505 (comment)

I'll test locally/on LXPLUS these 2 days.

I tried PyXRootD today. It largely works with LD_LIBRARY_PATH set to ${xrootd.out}/lib. However, I haven't got a way to write a Python-based xrdcp alternative for passthru.tests.

About LD_LIBRARY_PATH, I did not need it this time. What is a trigger for it? Is there a reproducer with a public URL?

[ShamrockLee]

What is a trigger for it?

It can be triggered by accessing root://eospublic.cern.ch//eos/opendata/alice/2010/LHC10h/000138275/ESD/0000/AliESDs.root with XRootD.client.File(). The iconic Auth failed: Could not load authentication handler. error message can be seen when LD_LIBRARY_PATH does not contain the library path of xrootd.

The following reproducer uses the experimental features nix-command and flakes, but classical-Nix equivalence should be easy to construct.

Preparation:

$ cd path/to/nixpkgs
$ git switch -d 3d695c171d8dbed8c7f80a23a56572f01c8304f1

$ nix run --impure --expr "let lib = import ./lib; pkgs = import ./. { }; in pkgs.python3.withPackages (ps: with ps; [ xrootd ])"
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import XRootD.client as client
>>> file = client.File()
>>> file.open("root://eospublic.cern.ch//eos/opendata/alice/2010/LHC10h/000138275/ESD/0000/AliESDs.root", flags=client.flags.OpenFlags.READ)
(<status: 3, code: 204, errno: 0, message: '[FATAL] Auth failed: Could not load authentication handler.', shellcode: 52, error: True, fatal: True, ok: False>, None)
>>> file.close()
(<status: 3, code: 204, errno: 0, message: '[FATAL] Auth failed: Could not load authentication handler.', shellcode: 52, error: True, fatal: True, ok: False>, None)
>>>

$ LD_LIBRARY_PATH="$(nix build --print-out-paths .#xrootd.out)/lib" nix run --impure --expr "let lib = import ./lib; pkgs = import ./. { }; in pkgs.python3.withPackages (ps: with ps; [ xrootd ])"
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import XRootD.client as client
>>> file = client.File()
>>> file.open("root://eospublic.cern.ch//eos/opendata/alice/2010/LHC10h/000138275/ESD/0000/AliESDs.root", flags=client.flags.OpenFlags.READ)
(<status: 0, code: 0, errno: 0, message: '[SUCCESS] ', shellcode: 0, error: False, fatal: False, ok: True>, None)
>>> file.close()
(<status: 0, code: 0, errno: 0, message: '[SUCCESS] ', shellcode: 0, error: False, fatal: False, ok: True>, None)

[veprbl]

# readelf -d /nix/store/c01mwq8kij00iglcd5irmjpcnrwx68zd-xrootd-5.6.6/lib/libXrdCl.so.3 | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/c01mwq8kij00iglcd5irmjpcnrwx68zd-xrootd-5.6.6/lib:/nix/store/h24v1xs4i9l3k18ygxvs62361rl77804-openssl-3.0.13/lib:/nix/store/ilr7br1ck7i6wcgfkynwpnjp7n964h38-zlib-1.3.1/lib:/nix/store/q9snqijfg4v4pjfabspysxszd25w4lcn-util-linux-minimal-2.39.3-lib/lib:/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib:/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib]

but LD_DEBUG=libs gives

...
    160257:	
    160257:	find library=libXrdSec-5.so [0]; searching
    160257:	search path=/nix/store/h24v1xs4i9l3k18ygxvs62361rl77804-openssl-3.0.13/lib:/nix/store/m0nicpa4d724rljadlmw1m3qv0ja4syy-systemd-255.2/lib		(RUNPATH from file /nix/store/c01mwq8kij00iglcd5irmjpcnrwx68zd-xrootd-5.6.6/lib/libXrdCl.so.3)
    160257:	 trying file=/nix/store/h24v1xs4i9l3k18ygxvs62361rl77804-openssl-3.0.13/lib/libXrdSec-5.so
    160257:	 trying file=/nix/store/m0nicpa4d724rljadlmw1m3qv0ja4syy-systemd-255.2/lib/libXrdSec-5.so
    160257:	search path=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib		(system search path)
    160257:	 trying file=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/libXrdSec-5.so
    160257:	search path=/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib		(RUNPATH from file python3)
    160257:	 trying file=/nix/store/1s98ricsglmfjjqkfnpvywnip5z7gp9q-gcc-13.2.0-lib/lib/libXrdSec-5.so
    160257:	search cache=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/etc/ld.so.cache
    160257:	search path=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib:/nix/store/f5my15qww10swmf66ns13l24yp6j5dmq-xgcc-13.2.0-libgcc/lib		(system search path)
    160257:	 trying file=/nix/store/cyrrf49i2hm1w7vn2j945ic3rrzgxbqs-glibc-2.38-44/lib/libXrdSec-5.so
    160257:	 trying file=/nix/store/f5my15qww10swmf66ns13l24yp6j5dmq-xgcc-13.2.0-libgcc/lib/libXrdSec-5.so
    160257:	
(<status: 3, code: 204, errno: 0, message: '[FATAL] Auth failed: Could not load authentication handler.', shellcode: 52, error: True, fatal: True, ok: False>, None)
    160257:	
...