R does not work after relocation

[ebagrenrut]

Steps to reproduce

$ docker run -it --rm spack/ubuntu-bionic:latest
:~# spack mirror add v0.21.0-build_systems https://binaries.spack.io/v0.21.0/build_systems
:~# spack buildcache keys --install --trust
:~# spack install --use-buildcache=only /eqteloqhjzix6ta373ruzt5imvvbcesc
:~# spack load r
:~# R
> help()
> install.packages('bit64')

Notice that R commands do not work because the default packages are not found.

Error message

:~# R

R version 4.3.0 (2023-04-21) -- "Already Tomorrow"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
> help()
Error in help() : could not find function "help"
> install.packages('bit64')

Error in install.packages("bit64") :
  could not find function "install.packages"

Information on your system

# spack debug report
* **Spack:** 0.21.0.dev0 (910190f55bb5467305dd75a4dac8c60f1f51e283)
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-skylake_avx512
* **Concretizer:** clingo

After struggling to understand why R installed from a buildcache was not working on our local systems (running CentOS 7), I reproduced the issue using Spack's own distributed ubuntu-bionic Docker image and Spack's own R binary from binaries.spack.io.

If I force Spack to build the same R from source (i.e. spack install --use-buildcache=package:never,dependencies:only /eqteloqhjzix6ta373ruzt5imvvbcesc), then R does work as expected. Some part of the relocation process seems to be missing or silently failing when the binary is used.

Here is what one might expect for a properly-working R:

root@6949bf78224a:~# spack install --use-buildcache=package:never,dependencies:only /eqteloqhjzix6ta373ruzt5imvvbcesc
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/bzip2-1.0.8-5mghoqz4ngtwq3v67ewyun64xv4o53m3
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libiconv-1.17-ogz4kf67zest4ajc6dtwxiokntbip24x
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/nghttp2-1.57.0-roxxir4emd7bmj7qv2733unvagl2xlg6
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/zlib-ng-2.1.4-syxyq4lwjyg6jcwdmueu3pcwz6vv4imc
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/gmake-4.4.1-s7ksjmgajvb7sjqiee7boqyapewgzirk
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/openjdk-11.0.20.1_1-emyzuxuhihfdjtxtq7raapcw2qufuym2
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/openssl-3.1.3-i7dl3uq4lad3dchztaghe2ae5vwiyloh
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libunistring-1.1-iuznve7ejcwrfepyqq2f5icu7l4oeuc4
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/ncurses-6.4-cod5aaaoh43yzbzkf4ljb3nfvjr637hh
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/berkeley-db-18.1.40-dzr37itwibvfx66orqzx2oya5fj4jits
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/xz-5.4.1-q2vsdwtcjmoirhf7wt4ffijey7buklby
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/which-2.21-3j5ymd5gybivcdmkxy53tdgq5vzuk5ys
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/pcre2-10.42-quubhlacmkwpz43ygmqn3vnwslxpgdjy
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/icu4c-67.1-gw4x3xgvtys4b2aahemlcpqh2v7zmwiz
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/zstd-1.5.5-dnn2xfy6g4mybb4aqijdyxt66auzn6vf
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/pigz-2.7-5v3za7qyrp6mt3bcapyy4ofoolslr2hp
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libidn2-2.3.4-uh2kdjvixrfblju2czgx3yzhnxp4xpkm
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/readline-8.2-abh5mvvbkpfvn6wrxf5n3m633t76b6kw
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libxml2-2.10.3-hzocw3suyawfzi6ufnf7kkvso4zey5tk
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/tar-1.34-oolmzjtck3er63v7mqv2x2dqdfdffxq3
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/curl-8.4.0-5oclhntmwoiecmbssryqc5nsdtjrfp2z
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/gdbm-1.23-puhkytr3sslzfdbwpcbunu5orlqa34if
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/gettext-0.22.3-q4bymsxue2tqaq6sj4d32vu6v7rrvviv
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/perl-5.38.0-7lnnnoavjsd2ov5326dv7x3hrtquydyb
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/texinfo-7.0.3-zora4oj5z4bfjkz3oai324wgeam5eagn
==> Installing r-4.3.0-eqteloqhjzix6ta373ruzt5imvvbcesc [26/26]
==> Using cached archive: /opt/spack/var/spack/cache/_source-cache/archive/45/45dcc48b6cf27d361020f77fde1a39209e997b81402b3663ca1c010056a6a609.tar.gz
==> No patches needed for r
==> r: Executing phase: 'autoreconf'
==> r: Executing phase: 'configure'
==> r: Executing phase: 'build'
==> r: Executing phase: 'install'
==> r: Successfully installed r-4.3.0-eqteloqhjzix6ta373ruzt5imvvbcesc
  Stage: 1.49s.  Autoreconf: 0.00s.  Configure: 33.87s.  Build: 2m 36.27s.  Install: 1.65s.  Post-install: 0.62s.  Total: 3m 14.02s
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/r-4.3.0-eqteloqhjzix6ta373ruzt5imvvbcesc
root@6949bf78224a:~# spack load /eqteloqhjzix6ta373ruzt5imvvbcesc
root@6949bf78224a:~# R

R version 4.3.0 (2023-04-21) -- "Already Tomorrow"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

Notice R gives no warnings about missing default packages.

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[haampie]

I think the issue is that R compresses the original install prefix and decompresses at runtime, strace shows it tries to execute <old prefix>/bin/which for example.

[haampie]

Okay, this is what happens:

R stores an absolute path to <prefix>/bin/which in byte code, and that byte code is compressed. It can be retrieved like this:

$ ./bin/R --quiet -e 'readRDS("./rlib/R/library/base/R/base.rdx")$variables$Sys.which' 2>/dev/null
> readRDS("./rlib/R/library/base/R/base.rdx")$variables$Sys.which
[1] 133838   2044

meaning the serialized stuff for Sys.which is stored at offset 133838 with length 2044 in base.rdb. That itself starts with a 4 byte integer, so those have to be skipped. Then a zlib header follows:

$ dd if=./rlib/R/library/base/R/base.rdb bs=1 skip=133842 count=2044 status=none | zlib-flate -uncompress
...
/home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/which-2.21-3j5ymd5gybivcdmkxy53tdgq5vzuk5ys/bin/which
...

So in short: a string we need to relocate is stored as compressed, serialized byte code in some sort of database for serialized objects. In principle it's possible to decompress the blocks in base.rdb, rewrite strings, store again, and update the offsets in base.rdx -- but that's pretty involved.

Hopefully R can use which from PATH instead of an absolute path?

[haampie]

Hopefully R can use which from PATH instead of an absolute path?

AC_PATH_PROGS(WHICH, which, which)
if test "${WHICH}" = which ; then
  ## needed to build and run R
  ## ends up hard-coded in the utils package
  AC_MSG_ERROR([[which is required but missing]])
fi

looks like we can't.

Another alternative is to look if we can disable compression of byte code. (Edit: looks like you can, but now idk where the source code for Sys.which is coming from...)

[haampie]

Disabling byte compilation doesn't help, it still stores something compressed & serialized in base.rdb. The file is smaller, so I guess there's a difference between byte code and compiled byte code.

Further options:

• Easiest is to patch the R build system to allow WHICH=which and expect it to be in PATH.
• Disabling compression of rdb entries (probably a bad idea)
• Ask R people to create a symlink to which in their prefix, have them register that in their R sources, so no absolute paths are registered in bytecode.

[haampie]

Submitted a small patch to R that uses indirection through a symlink to make relocation of which work.

[ebagrenrut]

@haampie I can't thank you enough for getting to the root cause. I've tested installing R from a buildcache to a new location and the default packages are now successfully found at start-up. Also, with the addition of gcc-runtime, R is, as far as I can tell, fully relocatable! Thanks!