support for multiple CVMFS repositories in `eessi_container.sh` #618

trz42 · 2024-06-17T13:12:36Z

Support for multiple CVMFS repositories is needed for dev.eessi.io.

Ready for review. Example test:

# ./eessi_container.sh --repository software.eessi.io --repository dev.eessi.io,access=rw
Using /tmp/eessi.jLuBerqoFu as tmp directory (to resume session add '--resume /tmp/eessi.jLuBerqoFu').
Pulling container image from docker://ghcr.io/eessi/build-node:debian11 to /tmp/eessi.jLuBerqoFu/ghcr.io_eessi_build_node_debian11.sif
Launching container with command (next line):
singularity -q shell  --fusemount container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch --fusemount container:cvmfs2 software.eessi.io /cvmfs/software.eessi.io --fusemount container:cvmfs2 dev.eessi.io /cvmfs_ro/dev.eessi.io --fusemount container:fuse-overlayfs -o lowerdir=/cvmfs_ro/dev.eessi.io -o upperdir=/tmp/dev.eessi.io/overlay-upper -o workdir=/tmp/dev.eessi.io/overlay-work /cvmfs/dev.eessi.io /tmp/eessi.jLuBerqoFu/ghcr.io_eessi_build_node_debian11.sif
INFO:    Environment variable SINGULARITY_BIND is set, but APPTAINER_BIND is preferred
INFO:    Environment variable SINGULARITY_HOME is set, but APPTAINER_HOME is preferred
CernVM-FS: pre-mounted on file descriptor 3
CernVM-FS: pre-mounted on file descriptor 3
CernVM-FS: pre-mounted on file descriptor 3
Apptainer> CernVM-FS: loading Fuse module... done
CernVM-FS: loading Fuse module... CernVM-FS: loading Fuse module... done
done

Apptainer> touch /cvmfs/dev.eessi.io/foo
Apptainer> touch /cvmfs/software.eessi.io/foo
touch: cannot touch '/cvmfs/software.eessi.io/foo': Permission denied

Partially fixes #616 (doesn't provide repository-specific bind mounts for host_injections).

eessi-bot · 2024-06-17T13:12:40Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

eessi-bot · 2024-06-17T13:12:41Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-software, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-compat

… polishing

Neves-P · 2024-06-27T16:14:09Z

Tried the container with @trz42's instructions from slack

# Step 1: checkout repo (and PR)
git clone https://github.com/EESSI/software-layer
cd software-layer
git fetch origin pull/618/head:PR618
git checkout PR618
# Step 2: launch interactive job, for example
srun --time=60 --pty bash
# Step 3: launch container with read access to software.eessi.io and write access to dev.eessi.io
./eessi_container.sh -r software.eessi.io -r dev.eessi.io,access=rw
# Step 4: init EESSI and build environment
source /cvmfs/software.eessi.io/versions/2023.06/init/bash
export EESSI_PROJECT_INSTALL=/cvmfs/dev.eessi.io/dev_test/versions/2023.06/software/linux/${EESSI_SOFTWARE_SUBDIR}
module load EESSI-extend/2023.06-easybuild
# Step 5: build some software (e.g., recently merged imbalanced-learn-0.12.3-gfbf-2023a.eb)
eb --robot --from-pr 20848 imbalanced-learn-0.12.3-gfbf-2023a.eb
# Step 6: show, load and test module
module show imbalanced-learn/0.12.3-gfbf-2023a
module load imbalanced-learn/0.12.3-gfbf-2023a
python -c "import imblearn"

Works out of the box! The only thing extra I had to do was: mkdir -p /cvmfs/dev.eessi.io/dev_test/versions/2023.06/software/linux/${EESSI_SOFTWARE_SUBDIR}.

Otherwise I get:

{EESSI 2023.06} Apptainer> module load EESSI-extend/2023.06-easybuild
-- Using /tmp/$USER as a temporary working directory for installations, you can override this by setting the
environment variable WORKING_DIR and reloading the module (e.g., /dev/shm is a common option)
Lmod has detected the following error:  The location of EESSI_PROJECT_INSTALL
(/cvmfs/dev.eessi.io/dev_test/versions/2023.06/software/linux/aarch64/neoverse_v1) does not exist or is not a folder
While processing the following module(s):
    Module fullname                 Module Filename
    ---------------                 ---------------
    EESSI-extend/2023.06-easybuild  /cvmfs/software.eessi.io/versions/2023.06/software/linux/aarch64/neoverse_v1/modules/all/EESSI-extend/2023.06-easybuild.lua

Once I first created the directory then I was able to load EESSI-extend in such a way that newly installed modules would land in dev.eessi.io, which is the intended behaviour.

I might be missing a step, but if not perhaps there is a way to take care of creating the necessary subdirectories automatically (i.e., cvmfs/somerepo/right_path/${EESSI_SOFTWARE_SUBDIR}?

Neves-P

Quoting my message from slack:

I can't see anything wrong in the code, but it might be a good idea to have someone who is more familiar with setting up CVMFS repositories than I am to also take a look since some changes may have implications that I'm not aware of. Beyond that, looks good to me, and works!

eessi_container.sh

…-layer into multiple_cvmfs_repos_eessi_container

bedroge

Looks good to me. Tested it with various scenarios, and it worked like a charm!

Neves-P

LGTM!

bedroge · 2024-07-04T13:36:11Z

The "tests for eessi_container.sh script / eessi_container_script (listrepos_custom)" CI was actually already broken: it was configuring a "EESSI/20AB.CD" and "EESSI/20HT.TP", but then running a grep for ""[EESSI/2023.02]". Not only is that version not available, also the regex itself is wrong: the square brackets should have been escaped, otherwise they will just try to match any of those characters. Because of the wrong regex, the check still passed with this wrong EESSI version.

I've fixed it in aa9895e. Also, the other check now doesn't look for "EESSI" anymore (which used to be the default repo name), but instead for "software.eessi.io". This was needed because of the changes in this PR.

Neves-P

Regex is fixed and the CI passed without issues 🎉

boegel · 2024-07-04T19:49:30Z

eessi_container.sh

-    BIND_PATHS="${BIND_PATHS},${EESSI_TMPDIR}/repos_cfg/${src}:${target}"
-  done
-  export EESSI_VERSION_OVERRIDE=${repo_version}
-  export EESSI_CVMFS_REPO_OVERRIDE="/cvmfs/${repo_name}"


Seems like this was important, it shouldn't have been removed.

I'm now seeing errors like this in the build job for #632:

ERROR: software.eessi.io/versions/2023.06/compat/linux/aarch64 does not exist!

Note the missing /cvmfs/...

Or maybe we're missing a fix in bot/build.sh, which now has this:

export EESSI_CVMFS_REPO_OVERRIDE=$(cfg_get_value "repository" "repo_name")

I guess this should be:

export EESSI_CVMFS_REPO_OVERRIDE=/cvmfs/$(cfg_get_value "repository" "repo_name")

Ah, yes, good catch, looks like that needs to be changed (and I think it makes more sense that these things are (only) set there and not in the container script).

see:

add missing /cvmfs/ in $EESSI_CVMFS_REPO_OVERRIDE defined by bot/build.sh and bot/test.sh #633

first steps to support multiple CVMFS repositories

e8c2d06

trz42 added the enhancement New feature or request label Jun 17, 2024

trz42 marked this pull request as draft June 17, 2024 13:12

truib added 3 commits June 18, 2024 15:05

further changes to support multiple repositories

c7a1ca2

configure multiple fusemounts

934059a

revert back changes related to BIND_PATHS and SINGULARITY_BIND + some…

32f4133

… polishing

trz42 marked this pull request as ready for review June 19, 2024 14:09

trz42 changed the title ~~[WIP] support for multiple CVMFS repositories in eessi_container.sh~~ support for multiple CVMFS repositories in eessi_container.sh Jun 19, 2024

truib added 3 commits June 20, 2024 09:07

store repository settings under tmp

07de700

removed unnecessary variable

d9c1fb1

add more information about repository-specific access modes

e25cd1e

Neves-P previously approved these changes Jun 27, 2024

View reviewed changes

bedroge reviewed Jul 1, 2024

View reviewed changes

eessi_container.sh Outdated Show resolved Hide resolved

bedroge reviewed Jul 1, 2024

View reviewed changes

eessi_container.sh Outdated Show resolved Hide resolved

bedroge added 2 commits July 4, 2024 13:22

Merge branch '2023.06-software.eessi.io' of github.com:EESSI/software…

ac6984b

…-layer into multiple_cvmfs_repos_eessi_container

remove TODO for using separate cache dirs for each repo

c895bf1

bedroge dismissed Neves-P’s stale review via c895bf1 July 4, 2024 11:24

fix usage line for access permissions

417f4dd

bedroge previously approved these changes Jul 4, 2024

View reviewed changes

Neves-P previously approved these changes Jul 4, 2024

View reviewed changes

bedroge added 2 commits July 4, 2024 13:41

check for software.eessi.io in the default --list-repos output

d8cf6a6

fix regex to find the custom EESSI version in the output of --list-repos

aa9895e

bedroge dismissed stale reviews from Neves-P and themself via aa9895e July 4, 2024 13:28

Neves-P approved these changes Jul 4, 2024

View reviewed changes

Neves-P merged commit a9b4233 into EESSI:2023.06-software.eessi.io Jul 4, 2024
34 checks passed

boegel mentioned this pull request Jul 4, 2024

{2023.06,a64fx}[foss/2023a] ParaView 5.11.2 #632

Merged

boegel reviewed Jul 4, 2024

View reviewed changes

boegel mentioned this pull request Jul 4, 2024

add missing /cvmfs/ in $EESSI_CVMFS_REPO_OVERRIDE defined by bot/build.sh and bot/test.sh #633

Merged

This was referenced Jul 5, 2024

{2023.06}[GCCcore/12.3.0-GCC/12.3.0-gompi/2023a] bio-packages #634

Merged

Fix path to overlay upper directory in create_tarball.sh #635

Merged

trz42 mentioned this pull request Aug 19, 2024

use upper directory of former 'rw' session in 'ro' mode #674

Merged

bedroge mentioned this pull request Sep 18, 2024

Broken/weird filenames for tarballs created by eessi_container.sh script #712

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for multiple CVMFS repositories in `eessi_container.sh` #618

support for multiple CVMFS repositories in `eessi_container.sh` #618

trz42 commented Jun 17, 2024 •

edited

Loading

eessi-bot bot commented Jun 17, 2024

eessi-bot bot commented Jun 17, 2024

Neves-P commented Jun 27, 2024

Neves-P left a comment

bedroge left a comment

Neves-P left a comment

bedroge commented Jul 4, 2024

Neves-P left a comment

boegel Jul 4, 2024 •

edited

Loading

boegel Jul 4, 2024

bedroge Jul 4, 2024

boegel Jul 4, 2024 •

edited

Loading

support for multiple CVMFS repositories in eessi_container.sh #618

support for multiple CVMFS repositories in eessi_container.sh #618

Conversation

trz42 commented Jun 17, 2024 • edited Loading

eessi-bot bot commented Jun 17, 2024

eessi-bot bot commented Jun 17, 2024

Neves-P commented Jun 27, 2024

Neves-P left a comment

Choose a reason for hiding this comment

bedroge left a comment

Choose a reason for hiding this comment

Neves-P left a comment

Choose a reason for hiding this comment

bedroge commented Jul 4, 2024

Neves-P left a comment

Choose a reason for hiding this comment

boegel Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

boegel Jul 4, 2024

Choose a reason for hiding this comment

bedroge Jul 4, 2024

Choose a reason for hiding this comment

boegel Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

support for multiple CVMFS repositories in `eessi_container.sh` #618

support for multiple CVMFS repositories in `eessi_container.sh` #618

trz42 commented Jun 17, 2024 •

edited

Loading

boegel Jul 4, 2024 •

edited

Loading

boegel Jul 4, 2024 •

edited

Loading