Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpy2 3.5.7 ignores R_HOME for some reason (and prints warning), when rpy2 3.5.6 just works #982

Closed
ekungurov opened this issue Jan 20, 2023 · 8 comments
Labels
bug Something isn't working

Comments

@ekungurov
Copy link

ekungurov commented Jan 20, 2023

Describe the issue or bug

rpy2 3.5.7 prints the following warning:
situation.py:266: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME

And then doesn't work because it can't find libR.so:
OSError: cannot load library '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so': /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so: cannot open shared object file: No such file or directory

rpy2 3.5.6 doesn't print the warning and works.

To Reproduce

Actually I use "jupyterlab_code_formatter" package to reproduce. The code is simple import

import jupyterlab_code_formatter

Pre-requisites:
libR.so installed in a non-standard directory

Steps to reproduce:

  1. Install rpy2=3.5.7, jupyterlab-code-formatter=1.5.3,
  2. Execute "export R_HOME=/usr/frog/R/4.1.2-foss-2018a/lib64/R/" assuming that library is located in /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib/libR.so
  3. Run command line python
  4. Execute "import jupyterlab_code_formatter" line of code

Python 3.8.15, R 4.1.2

Expected behavior
rpy2=3.5.6 searches libR in $R_HOME/lib/
rpy2=3.5.7 is searching in $R_HOME/lib64/
Is that a correct behaviour?

Error
The "import jupyterlab_code_formatter" fails with the following stack trace:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
  File "/.../.../.../lib/python3.8/site-packages/jupyterlab_code_formatter/__init__.py", line 4, in <module>
    from .handlers import setup_handlers
  File "/.../.../.../lib/python3.8/site-packages/jupyterlab_code_formatter/handlers.py", line 8, in <module>
    from .formatters import SERVER_FORMATTERS
  File "/.../.../.../lib/python3.8/site-packages/jupyterlab_code_formatter/formatters.py", line 16, in <module>
    import rpy2.robjects
  File "/.../.../.../lib/python3.8/site-packages/rpy2/robjects/__init__.py", line 15, in <module>
    import rpy2.rinterface as rinterface
  File "/.../.../.../lib/python3.8/site-packages/rpy2/rinterface.py", line 16, in <module>
    from rpy2.rinterface_lib import openrlib
  File "/.../.../.../lib/python3.8/site-packages/rpy2/rinterface_lib/openrlib.py", line 58, in <module>
    rlib = _dlopen_rlib(R_HOME)
  File "/.../.../.../lib/python3.8/site-packages/rpy2/rinterface_lib/openrlib.py", line 51, in _dlopen_rlib
    rlib = ffi.dlopen(lib_path)
OSError: cannot load library '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so': /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so: cannot open shared object file: No such file or directory
@ekungurov ekungurov added the bug Something isn't working label Jan 20, 2023
@ekungurov
Copy link
Author

When using rpy2=3.5.6, the libR.so is being searched in:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib/libR.so

When using rpy2=3.5.7, the libR.so is being searched in:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so

The output of python -m rpy2.situation when not working (rpy2=3.5.7):

rpy2 version:
3.5.7
Python version:
3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:46:39)
[GCC 10.4.0]
Looking for R's HOME:
    Environment variable R_HOME: /usr/frog/R/4.1.2-foss-2018a/lib64/R/
rpy2.situation: Unable to determine R home: [Errno 2] No such file or directory: 'R'
    Calling `R RHOME`: None
    Environment variable R_LIBS_USER: None
    Warning: There is no R in the PATH and no R_HOME defined.
R's additions to LD_LIBRARY_PATH:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib:/usr/frog/GCCcore/6.4.0/lib/gcc/x86_64-pc-linux-gnu/6.4.0:/usr/frog/GCCcore/6.4.0/lib64:/usr/frog/GCCcore/6.4.0/lib:/cm/shared/apps/uge/current/lib/lx-amd64
/usr/frog/scicomp/pythonds/develop/pyds_develop/lib/python3.8/site-packages/rpy2/situation.py:266: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME
  warnings.warn(msg)
R version:
rpy2.situation: Unable to determine the R version: [Errno 2] No such file or directory: 'R'
    In the PATH: None
    Loading R library from rpy2: cannot load library '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so': /usr/frog/R/4.1.2-foss-2018a/lib64/R/lib64/libR.so: cannot open shared object file: No such file or directory
Additional directories to load R packages from:
None
C extension compilation:
/usr/frog/scicomp/pythonds/develop/pyds_develop/lib/python3.8/site-packages/rpy2/situation.py:266: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME
  warnings.warn(msg)
  include:
  ['/usr/frog/R/4.1.2-foss-2018a/lib64/R/include']
  libraries:
  ['R', 'pcre', 'lzma', 'bz2', 'z', 'rt', 'dl', 'm', 'm', 'pthread', 'icuuc', 'icui18n']
  library_dirs:
  ['/usr/frog/Java/1.8.0_162/lib', '/usr/frog/FFTW/3.3.7-gompi-2018a/lib', '/usr/frog/ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20/lib', '/usr/frog/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib', '/usr/frog/GCCcore/6.4.0/lib64', '/usr/frog/GCCcore/6.4.0/lib', '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib']
  extra_compile_args:
  ['-std=c99']
  extra_link_args:
  ['-Wl,--export-dynamic', '-fopenmp']
Directory for the R shared library:
lib64

The output of python -m rpy2.situation when working (rpy2=3.5.6):

rpy2 version:
3.5.6
Python version:
3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:46:39)
[GCC 10.4.0]
Looking for R's HOME:
    Environment variable R_HOME: /usr/frog/R/4.1.2-foss-2018a/lib64/R/
rpy2.situation: Unable to determine R home: [Errno 2] No such file or directory: 'R'
    Calling `R RHOME`: None
    Environment variable R_LIBS_USER: None
    Warning: There is no R in the PATH and no R_HOME defined.
R's additions to LD_LIBRARY_PATH:
/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib:/usr/frog/GCCcore/6.4.0/lib/gcc/x86_64-pc-linux-gnu/6.4.0:/usr/frog/GCCcore/6.4.0/lib64:/usr/frog/GCCcore/6.4.0/lib:/cm/shared/apps/uge/current/lib/lx-amd64
R version:
rpy2.situation: Unable to determine the R version: [Errno 2] No such file or directory: 'R'
    In the PATH: None
    Loading R library from rpy2: OK
Additional directories to load R packages from:
None
C extension compilation:
/usr/frog/scicomp/pythonds/develop/pyds_develop/lib/python3.8/site-packages/rpy2/situation.py:242: UserWarning: R emitting a warning: WARNING: ignoring environment value of R_HOME
  warnings.warn(msg)
  include:
  ['/usr/frog/R/4.1.2-foss-2018a/lib64/R/include']
  libraries:
  ['R', 'pcre', 'lzma', 'bz2', 'z', 'rt', 'dl', 'm', 'm', 'pthread', 'icuuc', 'icui18n']
  library_dirs:
  ['/usr/frog/Java/1.8.0_162/lib', '/usr/frog/FFTW/3.3.7-gompi-2018a/lib', '/usr/frog/ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20/lib', '/usr/frog/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib', '/usr/frog/GCCcore/6.4.0/lib64', '/usr/frog/GCCcore/6.4.0/lib', '/usr/frog/R/4.1.2-foss-2018a/lib64/R/lib']
  extra_compile_args:
  []
  extra_link_args:
  ['-Wl,--export-dynamic', '-fopenmp']

The difference seeems to be in those lines:

Directory for the R shared library:
lib64

@dagonzalezfo
Copy link

Hi @ekungurov, I got the same issue with this version. To fix it I modify this line:

lib_path = os.path.join(r_home, get_r_libnn(r_home), 'libR.so')

by changing get_r_libnn(r_home) to lib and it work smootly. Do you mind to test?

@lgautier
Copy link
Member

Hi, if this looks like a possible fix it is easier for me to have the proposed change as a PR to review.

@ekungurov
Copy link
Author

ekungurov commented Jan 29, 2023

@dagonzalezfo The changed behaviour is definitely introduced by pull request #969 "Build with -Wl,-rpath".

However, I don't fully understand what RPATH is, what LIBnn is, and I am not sure what supposed behaviour should be.

Our version of R is built with some 'easybuild-easyconfigs' recipes. The
R CMD config LIBnn
returns "lib64" indeed, however as I wrote in the first message, the .so file is in ./R/4.1.2-foss-2018a/lib64/R/lib/libR.so

@dagonzalezfo
Copy link

dagonzalezfo commented Jan 30, 2023

Hi @ekungurov ,

Regarding LIBnn it is defined in the configuration process of R installation, that is not used by default in an easybuild installation. That's why R CMD config LIBnn do not return the proper value for your installation. However, you can tune easybuild-easyconfig configopts to change this behavior as explainted in R installation guidelines. https://cran.r-project.org/doc/manuals/r-patched/R-admin.html

To cover manual/standard easybuild installations I create a PR that should fix it .

Correction: LIBnn is not related with shared library location.

@lgautier
Copy link
Member

lgautier commented Feb 4, 2023

@dagonzalezfo - Can you point out the exact point in the R documentation where the installation setup is described? R CMD config LIBnn should return the correct directory name for R's shared libraries (https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Compile-and-load-flags).

@lgautier
Copy link
Member

@dagonzalezfo The changed behaviour is definitely introduced by pull request #969 "Build with -Wl,-rpath".

However, I don't fully understand what RPATH is, what LIBnn is, and I am not sure what supposed behaviour should be.

@ekungurov : this additional flags includes the path to the shared library libR.so to the Python C-extension in rpy2. The intent was to try solving the dynamic library loading issue present when the R installation does not take care of informing the system's linker about where the R library is. The resolution is to set LD_LIBRARY_PATH (see the README) at the root of the rpy2 source tree, but a few users trip on this. Unfortunately in practice it does not quite work as well as I hoped as an R installation might have dependent C libraries that are not system libraries (e.g., libRblas.so or libRlapacks.so). With -Wl,-rpath not providing the expected benefits but apparently creating trouble I'll probably revert that for rpy2-3.5.9.

Our version of R is built with some 'easybuild-easyconfigs' recipes. The R CMD config LIBnn returns "lib64" indeed, however as I wrote in the first message, the .so file is in ./R/4.1.2-foss-2018a/lib64/R/lib/libR.so

The output of LIBnn was not correctly used to build the path to the R shared libraries. PR #990 aims at fixing the regression.

@dagonzalezfo
Copy link

@dagonzalezfo - Can you point out the exact point in the R documentation where the installation setup is described? R CMD config LIBnn should return the correct directory name for R's shared libraries (https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Compile-and-load-flags).

@lgautier My mistake, LIBnn is used to form the prefix for r_home:

prefix/LIBnn/R or libdir/R
all the rest (libraries, on-line help system, …). Here LIBnn is usually ‘lib’, but may be ‘lib64’ on some 64-bit Linux systems. This is known as the R home directory

(https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installation)

I test with some configurations, and in each time the shared libraries are at prefix/LIBnn/R/lib.

Thanks for fixing it in #990.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants