rpy2 3.5.7 ignores R_HOME for some reason (and prints warning), when rpy2 3.5.6 just works #982

ekungurov · 2023-01-20T15:26:16Z

Describe the issue or bug

rpy2 3.5.7 prints the following warning:
situation.py:266: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME

And then doesn't work because it can't find libR.so:
OSError: cannot load library '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so': /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so: cannot open shared object file: No such file or directory

rpy2 3.5.6 doesn't print the warning and works.

To Reproduce

Actually I use "jupyterlab_code_formatter" package to reproduce. The code is simple import

import jupyterlab_code_formatter

Pre-requisites:
libR.so installed in a non-standard directory

Steps to reproduce:

Install rpy2=3.5.7, jupyterlab-code-formatter=1.5.3,
Execute "export R_HOME=/usr/frog/R/4.1.2-foss-2018a/lib64/R/" assuming that library is located in /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib/libR.so
Run command line python
Execute "import jupyterlab_code_formatter" line of code

Python 3.8.15, R 4.1.2

Expected behavior
rpy2=3.5.6 searches libR in $R_HOME/lib/
rpy2=3.5.7 is searching in $R_HOME/lib64/
Is that a correct behaviour?

Error
The "import jupyterlab_code_formatter" fails with the following stack trace:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
  File "/.../.../.../lib/python3.8/site-packages/jupyterlab_code_formatter/__init__.py", line 4, in <module>
    from .handlers import setup_handlers
  File "/.../.../.../lib/python3.8/site-packages/jupyterlab_code_formatter/handlers.py", line 8, in <module>
    from .formatters import SERVER_FORMATTERS
  File "/.../.../.../lib/python3.8/site-packages/jupyterlab_code_formatter/formatters.py", line 16, in <module>
    import rpy2.robjects
  File "/.../.../.../lib/python3.8/site-packages/rpy2/robjects/__init__.py", line 15, in <module>
    import rpy2.rinterface as rinterface
  File "/.../.../.../lib/python3.8/site-packages/rpy2/rinterface.py", line 16, in <module>
    from rpy2.rinterface_lib import openrlib
  File "/.../.../.../lib/python3.8/site-packages/rpy2/rinterface_lib/openrlib.py", line 58, in <module>
    rlib = _dlopen_rlib(R_HOME)
  File "/.../.../.../lib/python3.8/site-packages/rpy2/rinterface_lib/openrlib.py", line 51, in _dlopen_rlib
    rlib = ffi.dlopen(lib_path)
OSError: cannot load library '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so': /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so: cannot open shared object file: No such file or directory

The text was updated successfully, but these errors were encountered:

ekungurov · 2023-01-20T19:58:01Z

When using rpy2=3.5.6, the libR.so is being searched in:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib/libR.so

When using rpy2=3.5.7, the libR.so is being searched in:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so

The output of python -m rpy2.situation when not working (rpy2=3.5.7):

rpy2 version:
3.5.7
Python version:
3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:46:39)
[GCC 10.4.0]
Looking for R's HOME:
    Environment variable R_HOME: /usr/frog/R/4.1.2-foss-2018a/lib64/R/
rpy2.situation: Unable to determine R home: [Errno 2] No such file or directory: 'R'
    Calling `R RHOME`: None
    Environment variable R_LIBS_USER: None
    Warning: There is no R in the PATH and no R_HOME defined.
R's additions to LD_LIBRARY_PATH:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib:/usr/frog/GCCcore/6.4.0/lib/gcc/x86_64-pc-linux-gnu/6.4.0:/usr/frog/GCCcore/6.4.0/lib64:/usr/frog/GCCcore/6.4.0/lib:/cm/shared/apps/uge/current/lib/lx-amd64
/usr/frog/scicomp/pythonds/develop/pyds_develop/lib/python3.8/site-packages/rpy2/situation.py:266: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME
  warnings.warn(msg)
R version:
rpy2.situation: Unable to determine the R version: [Errno 2] No such file or directory: 'R'
    In the PATH: None
    Loading R library from rpy2: cannot load library '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so': /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so: cannot open shared object file: No such file or directory
Additional directories to load R packages from:
None
C extension compilation:
/usr/frog/scicomp/pythonds/develop/pyds_develop/lib/python3.8/site-packages/rpy2/situation.py:266: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME
  warnings.warn(msg)
  include:
  ['/usr/frog/R/4.1.2-foss-2018a/lib64/R/include']
  libraries:
  ['R', 'pcre', 'lzma', 'bz2', 'z', 'rt', 'dl', 'm', 'm', 'pthread', 'icuuc', 'icui18n']
  library_dirs:
  ['/usr/frog/Java/1.8.0_162/lib', '/usr/frog/FFTW/3.3.7-gompi-2018a/lib', '/usr/frog/ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20/lib', '/usr/frog/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib', '/usr/frog/GCCcore/6.4.0/lib64', '/usr/frog/GCCcore/6.4.0/lib', '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib']
  extra_compile_args:
  ['-std=c99']
  extra_link_args:
  ['-Wl,--export-dynamic', '-fopenmp']
Directory for the R shared library:
lib64

The output of python -m rpy2.situation when working (rpy2=3.5.6):

rpy2 version:
3.5.6
Python version:
3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:46:39)
[GCC 10.4.0]
Looking for R's HOME:
    Environment variable R_HOME: /usr/frog/R/4.1.2-foss-2018a/lib64/R/
rpy2.situation: Unable to determine R home: [Errno 2] No such file or directory: 'R'
    Calling `R RHOME`: None
    Environment variable R_LIBS_USER: None
    Warning: There is no R in the PATH and no R_HOME defined.
R's additions to LD_LIBRARY_PATH:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib:/usr/frog/GCCcore/6.4.0/lib/gcc/x86_64-pc-linux-gnu/6.4.0:/usr/frog/GCCcore/6.4.0/lib64:/usr/frog/GCCcore/6.4.0/lib:/cm/shared/apps/uge/current/lib/lx-amd64
R version:
rpy2.situation: Unable to determine the R version: [Errno 2] No such file or directory: 'R'
    In the PATH: None
    Loading R library from rpy2: OK
Additional directories to load R packages from:
None
C extension compilation:
/usr/frog/scicomp/pythonds/develop/pyds_develop/lib/python3.8/site-packages/rpy2/situation.py:242: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME
  warnings.warn(msg)
  include:
  ['/usr/frog/R/4.1.2-foss-2018a/lib64/R/include']
  libraries:
  ['R', 'pcre', 'lzma', 'bz2', 'z', 'rt', 'dl', 'm', 'm', 'pthread', 'icuuc', 'icui18n']
  library_dirs:
  ['/usr/frog/Java/1.8.0_162/lib', '/usr/frog/FFTW/3.3.7-gompi-2018a/lib', '/usr/frog/ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20/lib', '/usr/frog/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib', '/usr/frog/GCCcore/6.4.0/lib64', '/usr/frog/GCCcore/6.4.0/lib', '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib']
  extra_compile_args:
  []
  extra_link_args:
  ['-Wl,--export-dynamic', '-fopenmp']

The difference seeems to be in those lines:

Directory for the R shared library:
lib64

dagonzalezfo · 2023-01-27T16:45:01Z

Hi @ekungurov, I got the same issue with this version. To fix it I modify this line:

rpy2/rpy2/situation.py

Line 192 in 77dac44

lib_path = os.path.join(r_home, get_r_libnn(r_home), 'libR.so')

by changing get_r_libnn(r_home) to lib and it work smootly. Do you mind to test?

lgautier · 2023-01-29T16:00:31Z

Hi, if this looks like a possible fix it is easier for me to have the proposed change as a PR to review.

ekungurov · 2023-01-29T18:37:28Z

@dagonzalezfo The changed behaviour is definitely introduced by pull request #969 "Build with -Wl,-rpath".

However, I don't fully understand what RPATH is, what LIBnn is, and I am not sure what supposed behaviour should be.

Our version of R is built with some 'easybuild-easyconfigs' recipes. The
R CMD config LIBnn
returns "lib64" indeed, however as I wrote in the first message, the .so file is in ./R/4.1.2-foss-2018a/lib64/R/lib/libR.so

dagonzalezfo · 2023-01-30T15:30:21Z

Hi @ekungurov ,

Regarding LIBnn it is defined in the configuration process of R installation, that is not used by default in an easybuild installation. That's why R CMD config LIBnn do not return the proper value for your installation. However, you can tune easybuild-easyconfig configopts to change this behavior as explainted in R installation guidelines. https://cran.r-project.org/doc/manuals/r-patched/R-admin.html

To cover manual/standard easybuild installations I create a PR that should fix it .

Correction: LIBnn is not related with shared library location.

lgautier · 2023-02-04T20:20:08Z

@dagonzalezfo - Can you point out the exact point in the R documentation where the installation setup is described? R CMD config LIBnn should return the correct directory name for R's shared libraries (https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Compile-and-load-flags).

Fixes issue #982.

lgautier · 2023-02-11T14:28:32Z

@dagonzalezfo The changed behaviour is definitely introduced by pull request #969 "Build with -Wl,-rpath".

However, I don't fully understand what RPATH is, what LIBnn is, and I am not sure what supposed behaviour should be.

@ekungurov : this additional flags includes the path to the shared library libR.so to the Python C-extension in rpy2. The intent was to try solving the dynamic library loading issue present when the R installation does not take care of informing the system's linker about where the R library is. The resolution is to set LD_LIBRARY_PATH (see the README) at the root of the rpy2 source tree, but a few users trip on this. Unfortunately in practice it does not quite work as well as I hoped as an R installation might have dependent C libraries that are not system libraries (e.g., libRblas.so or libRlapacks.so). With -Wl,-rpath not providing the expected benefits but apparently creating trouble I'll probably revert that for rpy2-3.5.9.

Our version of R is built with some 'easybuild-easyconfigs' recipes. The R CMD config LIBnn returns "lib64" indeed, however as I wrote in the first message, the .so file is in ./R/4.1.2-foss-2018a/lib64/R/lib/libR.so

The output of LIBnn was not correctly used to build the path to the R shared libraries. PR #990 aims at fixing the regression.

Fixes issue #982.

dagonzalezfo · 2023-02-11T15:30:50Z

@dagonzalezfo - Can you point out the exact point in the R documentation where the installation setup is described? R CMD config LIBnn should return the correct directory name for R's shared libraries (https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Compile-and-load-flags).

@lgautier My mistake, LIBnn is used to form the prefix for r_home:

prefix/LIBnn/R or libdir/R
all the rest (libraries, on-line help system, …). Here LIBnn is usually ‘lib’, but may be ‘lib64’ on some 64-bit Linux systems. This is known as the R home directory

(https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installation)

I test with some configurations, and in each time the shared libraries are at prefix/LIBnn/R/lib.

Thanks for fixing it in #990.

ekungurov added the bug Something isn't working label Jan 20, 2023

dagonzalezfo mentioned this issue Jan 30, 2023

Add eb-support #988

Closed

lgautier added a commit that referenced this issue Feb 11, 2023

Incorrect use of LIBnn to build path to R shared library libR.so.

93c17b7

Fixes issue #982.

lgautier mentioned this issue Feb 11, 2023

Incorrect use of LIBnn to build path to R shared library libR.so. #990

Merged

lgautier added a commit that referenced this issue Feb 11, 2023

Incorrect use of LIBnn to build path to R shared library libR.so. (#990)

6a0b683

Fixes issue #982.

lgautier closed this as completed Feb 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpy2 3.5.7 ignores R_HOME for some reason (and prints warning), when rpy2 3.5.6 just works #982

rpy2 3.5.7 ignores R_HOME for some reason (and prints warning), when rpy2 3.5.6 just works #982

ekungurov commented Jan 20, 2023 •

edited

Loading

ekungurov commented Jan 20, 2023

dagonzalezfo commented Jan 27, 2023

lgautier commented Jan 29, 2023

ekungurov commented Jan 29, 2023 •

edited

Loading

dagonzalezfo commented Jan 30, 2023 •

edited

Loading

lgautier commented Feb 4, 2023

lgautier commented Feb 11, 2023

dagonzalezfo commented Feb 11, 2023

rpy2 3.5.7 ignores R_HOME for some reason (and prints warning), when rpy2 3.5.6 just works #982

rpy2 3.5.7 ignores R_HOME for some reason (and prints warning), when rpy2 3.5.6 just works #982

Comments

ekungurov commented Jan 20, 2023 • edited Loading

ekungurov commented Jan 20, 2023

dagonzalezfo commented Jan 27, 2023

lgautier commented Jan 29, 2023

ekungurov commented Jan 29, 2023 • edited Loading

dagonzalezfo commented Jan 30, 2023 • edited Loading

lgautier commented Feb 4, 2023

lgautier commented Feb 11, 2023

dagonzalezfo commented Feb 11, 2023

ekungurov commented Jan 20, 2023 •

edited

Loading

ekungurov commented Jan 29, 2023 •

edited

Loading

dagonzalezfo commented Jan 30, 2023 •

edited

Loading