Support for Lmod/EESSI module environments: Injecting environment variables into the consoles

[ruben-canas-ivi]

Hi Positron team! First of all, amazing work with the IDE. I'm testing it on a server environment, but I've hit a structural roadblock regarding how Positron (and specifically the Ark kernel) handles environment variables in HPC/Server setups.

I am working on a Linux server utilizing EESSI and Lmod. In these environments, R and its dependencies (compilers, external C/C++ libraries) are loaded dynamically via shell modules (module load ...). This relies heavily on injecting paths into LD_LIBRARY_PATH, CPATH, LIBRARY_PATH, etc.

I have used a custom initScript for the environmentModules setting and a list of different modules I usually load on sessions (e.g. in Open OnDemand) in order for R to work with all packages:

"positron.environmentModules.initScript": "/soft/.positron_eessi_init.sh",
	"positron.environmentModules.environments": {
		"R 4.4.1 EESSI" : {
			"languages" : ["r"],
			"modules": [
            "GCC/13.2.0",
            "Szip/2.1.1-GCCcore-13.2.0",
            "CMake/3.27.6-GCCcore-13.2.0",
            "pkgconf/2.0.3-GCCcore-13.2.0",
			"FFTW/3.3.10-GCC-13.2.0",
            "R/4.4.1-gfbf-2023b"
        ]
		},
		"R 4.5.2 EESSI" : {
			"languages" : ["r"],
			"modules": [
            "GCC/13.2.0",
            "Szip/2.1.1-GCCcore-13.2.0",
            "CMake/3.27.6-GCCcore-13.2.0",
            "pkgconf/2.0.3-GCCcore-13.2.0",
            "R/4.5.2-gfbf-2023b"
        ]
		}

Due to Positron console execution policy, it bypasses or drops the environment variables set by the Lmod shell.
As a result:

1. Dynamic linking fails: Packages that require external shared libraries (e.g., libfftw3.so.3) crash at runtime because Ark doesn't inherit the LD_LIBRARY_PATH from the EESSI module.
2. Compilation fails: Source packages cannot find the loaded GCC/Intel compilers or system headers.

Feature Request:

A supported mechanism to inject or inherit environment variables (such as LD_LIBRARY_PATH) into the Ark kernel before the R session starts. It believe something such as an envVars object on the positron.environmentModules.environments could do the trick.

Thank you in advance.
Regards,

[jmcphers]

Positron console execution policy, it bypasses or drops the environment variables set by the Lmod shell.

Can you say more about this? We don't have any explicit policy set that does this, and anecdotally have a number of people using environment modules that set LD_LIBRARY_PATH without incident.

What version of Positron are you using? We have recently made some improvements to the environment module support. E.g. if your R installation was in a well known location, you may have been hitting #13936.

[ruben-canas-ivi]

I would like to share something that I cannot wrap my head around. I have been twitching with different ways to inject variables into the Ark R environment, and I realised that this terminal would read the .Renviron variables. So I changed my init script to this:

#!/bin/bash
export EESSI_SILENT=1
export LMOD_QUIET=1
export LMOD_REDIRECT=yes
source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash > /dev/null 2>&1
module use /soft/EESSI_custom/modules/all > /dev/null 2>&1


module load GCC/13.2.0 Szip/2.1.1-GCCcore-13.2.0 CMake/3.27.6-GCCcore-13.2.0 pkgconf/2.0.3-GCCcore-13.2.0 FFTW/3.3.10-GCC-13.2.0
export LD_LIBRARY_PATH="${LIBRARY_PATH}:${EBROOTGCC}/lib64:${EBROOTFFTW}/lib:${EBROOTSZIP}/lib:${LD_LIBRARY_PATH}"
cat > "${HOME}/.Renviron" <<EOF
PATH="${PATH}"
LD_LIBRARY_PATH="\${R_HOME}/lib:${LD_LIBRARY_PATH}"
LIBRARY_PATH="${LIBRARY_PATH}"
CPATH="${CPATH}"
PKG_CONFIG_PATH="${PKG_CONFIG_PATH}"
EOF

This allows to R from Ark to correctly read the LD_LIBRARY_PATH:

> stringr::str_split(string = Sys.getenv("LD_LIBRARY_PATH"), pattern = ":")
[[1]]
 [1] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/R/4.4.1-gfbf-2023b/lib64/R/lib"     
 [2] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/FFTW/3.3.10-GCC-13.2.0/lib"         
 [3] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/pkgconf/2.0.3-GCCcore-13.2.0/lib"   
 [4] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/libarchive/3.7.2-GCCcore-13.2.0/lib"
 [5] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/cURL/8.3.0-GCCcore-13.2.0/lib"      
 [6] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/OpenSSL/1.1/lib"                    
 [7] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/Szip/2.1.1-GCCcore-13.2.0/lib"      
 [8] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/GCCcore/13.2.0/lib64"               
 [9] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/FFTW/3.3.10-GCC-13.2.0/lib"         
[10] "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/Szip/2.1.1-GCCcore-13.2.0/lib"      
[11] ""

I can see that the FFTW library path is loaded (twice). Then the other ldd system command give me this result:

> system("ldd /home/rcanas/R/x86_64-pc-linux-gnu-library/4.4/KSgeneral/libs/KSgeneral.so")
	linux-vdso.so.1 (0x00007350084f0000)
	libfftw3.so.3 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/FFTW/3.3.10-GCC-13.2.0/lib/libfftw3.so.3 (0x0000735008200000)
	libR.so => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/R/4.4.1-gfbf-2023b/lib64/R/lib/libR.so (0x0000735007c00000)
	libstdc++.so.6 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/GCCcore/13.2.0/lib64/libstdc++.so.6 (0x0000735007800000)
	libm.so.6 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/libm.so.6 (0x0000735008120000)
	libgcc_s.so.1 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/GCCcore/13.2.0/lib64/libgcc_s.so.1 (0x0000735008498000)
	libc.so.6 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/libc.so.6 (0x000073500762f000)
	libflexiblas.so.3 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/FlexiBLAS/3.3.1-GCC-13.2.0/lib64/libflexiblas.so.3 (0x0000735007200000)
	libgfortran.so.5 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/GCCcore/13.2.0/lib64/libgfortran.so.5 (0x0000735006e00000)
	libreadline.so.8 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libreadline.so.8 (0x00007350080c8000)
	libpcre2-8.so.0 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/PCRE2/10.42-GCCcore-13.2.0/lib64/libpcre2-8.so.0 (0x0000735007b64000)
	libdeflate.so.0 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/libdeflate/1.19-GCCcore-13.2.0/lib/../lib64/libdeflate.so.0 (0x00007350080b6000)
	liblzma.so.5 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/liblzma.so.5 (0x0000735008086000)
	libbz2.so.1 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libbz2.so.1 (0x0000735007b51000)
	libz.so.1 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libz.so.1 (0x0000735007b37000)
	libtirpc.so.3 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libtirpc.so.3 (0x0000735007b0e000)
	libicuuc.so.74 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/ICU/74.1-GCCcore-13.2.0/lib/../lib64/libicuuc.so.74 (0x0000735006a00000)
	libicui18n.so.74 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/ICU/74.1-GCCcore-13.2.0/lib/../lib64/libicui18n.so.74 (0x0000735006600000)
	libgomp.so.1 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/GCCcore/13.2.0/lib64/libgomp.so.1 (0x0000735007abc000)
	/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x00007350084f2000)
	libquadmath.so.0 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/GCCcore/13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/../../../../lib64/libquadmath.so.0 (0x0000735007a75000)
	libtinfow.so.6 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/lib64/libtinfow.so.6 (0x00007350075ee000)
	libicudata.so.74 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/cascadelake/software/ICU/74.1-GCCcore-13.2.0/lib/libicudata.so.74 (0x0000735004800000)

Which I thought it was great news, since it could find all the libraries.
However, I keep getting the same error:

> library("KSgeneral")
Error:
! package or namespace load failed for ‘KSgeneral’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/rcanas/R/x86_64-pc-linux-gnu-library/4.4/KSgeneral/libs/KSgeneral.so':
  libfftw3.so.3: cannot open shared object file: No such file or directory

Hide Traceback
    ▆
 1. └─base::library("KSgeneral")
 2.   └─base::tryCatch(...)
 3.     └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 4.       └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 5.         └─value[[3L]](cond)

I tried to remove the package and reinstall it in this state. The package installed perfectly (which wasn't happening before, the installation was made on a regular R terminal) but the same error persists.

[ruben-canas-ivi]

Another note, as a temporary (?) solution I injected the paths directly on the binaries of the library with patchelf --set-rpath "${EBROOTGCC}/lib64:${EBROOTFFTW}/lib" /home/rcanas/R/x86_64-pc-linux-gnu-library/4.4/KSgeneral/libs/KSgeneral.so.

This way, the library can be loaded on the Ark R environment in Positron. I find the solution a bit complicated, and if I want to propagate the solution across the different users I believe I should create a custom Makevars for the EESSI environment, which is a hole I do not want to dig in.

[jmcphers]

The good news is that thanks to these logs I was able to pinpoint the problem... the bad news is that it's a bug in Positron without a workaround better than what you're doing now. We'll get this fixed soon! #15191

[ruben-canas-ivi]

Thanks you for the update and for opening the issue!
Looking forward to the fix, thanks again for the great work!

[jmcphers]

I did some more detailed analysis and discovered that the analysis I did in #15191 was not correct. :-/ But the good news is that this still may be a problem over which you have some control!

The goal of the "init script" setting to make the module command available, etc. I wonder if this isn't happening for some reason.

If you remove the init script setting and let Positron auto detect the init script, does it work?

If not, could you run the Environment Modules: Generate Diagnostics Report command and share the results?

If your goal is to set additional env vars in the consoles, there are a few other routes you might investigate:

• in Settings > Environment Variables, you can define variables to set in all consoles & terminals
• you can create your own environment module named e.g. positron and add the appropriate vars there, then include it in the set of modules to load in the array you define in Positron's settings