Config: Many platform variants of the same package version may be confusing

[DraTeots]

From the user (Florian), who uses cvmfs spack installation on ifarm:

I am little bit confused aobut all the different packages and architectures. I need to run 1.3.8 to verify my plots for the Yellow Report. So which of these should I pick:

   63cvnpm g4e@1.3.8%gcc@9.3.0 arch=linux-rhel7-x86_64
    s277bkq g4e@1.3.8%gcc@9.3.0 arch=linux-rhel7-x86_64
    3ph635f g4e@1.3.8%gcc@9.3.0 arch=linux-rhel7-x86_64
    ef7rk23 g4e@1.3.8%gcc@9.3.0 arch=linux-centos7-x86_64
    4w74oew g4e@1.3.8%gcc@9.3.0 arch=linux-centos8-x86_64
    kteqden g4e@1.3.8%gcc@9.3.0 arch=linux-rhel7-x86_64
    ekarbs2 g4e@1.3.8%gcc@9.3.0 arch=linux-ubuntu18.04-x86_64
    7aqb4wy g4e@1.3.8%gcc@9.3.0 arch=linux-ubuntu20.04-x86_64
    2loaofp g4e@1.3.8%gcc@10.2.0 arch=linux-ubuntu20.10-x86_6

I know that Spack tends to use many versions, that is what we have to live with. But what is confusing for me here is that at JLab we have different nodes with various operating system versions (rhel7, rhel8, centos7, centos8). And it is not clear how much versions are compatible and if a user should write a script in order to load g4e for batch jobs (very inconvenient), or they are compatible. And if compatible, could we reduce their number or not.

Kind regards

[wdconinc]

Not a solution, but in the short term

spack load g4e os=centos7

is a workaround.

[wdconinc]

The longer answer is: we need to get lmod/tcl to work. At this point there is a restriction in spack that prevents the lmod modulefiles from being added to the MODULEPATH when both lmod and tcl are available.

[wdconinc]

This page https://spack-tutorial.readthedocs.io/en/latest/tutorial_modules.html is what I need to spend some time with...

[wdconinc]

The ultimate answer is that Florian is encouraged to use module load. You'll see that now (after some more work) there is only 1 entry for g4e under module avail. It is still somewhat unfortunately named (g4e-1.3.8-gcc-9.3.0-ef7rk23) and that's for now a bit in flux. The ultimate intended behavior is that module load g4e would be the main way of loading g4e. There will be a few things needed before we get to that:

• hierarchical modules: so we can support differnet gcc versions, so you would do module load gcc/10.2.0 and then module load g4e would automatically only load the one loaded with gcc@10.2.0.
• hiding the dependencies: right now all dependencies are showing up as modules as well, but no one would ever load pcre-8.44-gcc-9.3.0-cg22j7n explicitly. But, if we use the current system of installing escalate then g4e is as much a dependency as pcre. So, we can't quite hide the dependencies just yet. We first need to modify the environments to install the components of the escalate dummy package.

[wdconinc]

Another response to the original question: the intent is that module load g4e (when it is supported in the future) on any supported system loads the exact same version tree, no matter which system you are on. So, you should be able to run that on centos7, centos8, or an ubuntu machine. This does NOT mean it is compiled with the same compiler (we will prefer a more recent system compiler).

[hauenst]

I am happy to use module in the future. I would just suggest to have different versions like module load g4e/1.3.8 or module load g4e/1.4.2. CLAS12 is doing this on the farm and various versions with correct dependencies are available which are quite easy to use and fail safe from the use of wrong/missing dependencies

[wdconinc]

@hauenst Thanks for the feedback. We are still navgating how best to provide multiple versions in a way that doesn't blow our filesystem allocation (we are using cvmfs, which comes with some considerations on total installed size since it is replicated for multiple operating systems to other local mirror servers around the world).

For end-point packages like g4e where any individual version is of a fairly trivial size, that's not as much of a concern (as long as dependency versions are not forced to change), so I will see what we can do about this.