Repository navigation
Using dev container on HPC
We can grab the same container that is used on NOAA-EMC/GEODE’s GitHub Actions CI for development on NOAA RDHPCS Ursa (or wherever else you like to use containers)
To use the container on Ursa:
singularity shell -B /scratch3 -B /scratch4 -e /scratch3/NCEPDEV/da/Cory.R.Martin/geode/containers/geode_env_latest.sifThat’s it! You can confirm by running something like
Apptainer> which show_queries.x
/usr/local/bin/show_queries.x
to verify that you are in the container and have the prebuilt executables/libraries.
Note, you may want to source your ~/.bash_profile since inside the container things won’t be “pretty”, such as ‘ls’, and you will not have any of your aliases defined. But BE CAREFUL, if you have a lot of “stuff” in your login environment this could be problematic. Just another reason why one should be cautious on what they put in their .bashrc/.bash_profile, etc.
Now, say you want to pull another container onto Ursa. It’s easy:
export APPTAINER_CACHEDIR=/path/on/scratch/$USER/apptainer-cache
mkdir -p "$APPTAINER_CACHEDIR"
cd /path/to/where/you/want/container/saved
srun -n 1 --cpus-per-task=8 --mem=200G --time=30 --partition=u1-service singularity pull <name_of_container>.sif docker://ghcr.io/<org>/<repo>:<tag>
Where <name_of_container> is what name it will be saved to on disk, <org> is the GitHub organization or username, <repo> is the repository name and <tag> is the tag (such as latest).
Note, you may want to pull a container from Docker Hub instead of GitHub, such as those from EPIC:
https://hub.docker.com/u/noaaepic/ or JCSDA https://hub.docker.com/u/jcsda/
Note, you will likely need to modify the srun command for other machines/environments and this assumes that you already have default accounts defined in your environment for srun. It may not be necessary to submit a batch job to do this, but this is the better practice to be a good neighbor for other users on the HPC login nodes.