Singularity Images

[ax3l]

@ax3l BTW i created different singularity definition files which could eventually allow any user to run WarpX on any HPC system which provide singularity/apptainer as container technique.
Defintiion files for CPUs and GPU (AMD+ROCM) are provided.

• https://git.gsi.de/d.bertini/pp-containers

May be interesting for WarpX user?

Originally posted by @denisbertini in #3967 (comment)

[ax3l]

Thank you @denisbertini , that is awesome! We have containers planned for users but did not yet get to it.

I would suggest the following workflow: we could build singularity containers via GitHub actions cronjob nightly and on every release.

There are a few nitpicks with containers we need to be aware of:

• we cannot tune for CPU microarchitectures, which according to our tests usually leaves about 2.5% performance on the table, but this gap might increase a bit in the future
• we need to build at least for OpenMPI and MPICH, to be compatible with many host MPIs.

Do you have a good workflow for publishing the generated images? GitHub has a registry, but I think the storage is very limited (we can ask for more, but not clear if we get it). Is there something like Dockerhub for Singularity these days?

Small note: your containers do not yet build the RZ geometry, so we can add that, too :)
-DWarpX_DIMS="1;2;RZ;3"

[denisbertini]

• you are right about tuning on microarchitectures, using container instead of working directly on bare metal nodes is a compromise. One can discuss about pros and cons for long and i do no thing it is relevant here.
  One should take this just as another way to run warpx on HPC clusters.

• I did no think for now about publishing the images, but yes Dockerhub could be a possibility, but certainly coming with storage limitations. Best is to have your own local registry with controllable storage capabilties.

• About implementing other warpx options: of course one can add other option to the building procedure in the .def files.

[ax3l]

One should take this just as another way to run warpx on HPC clusters.

Oh yes, fully agree. Was just listing for completeness. I think it is a great addition and we can add a comment in the docs to be transparent. I think this will be super useful for many workflows, including AI/ML, optimization runs, etc.

own local registry with controllable storage capabilties

Ok, gotta check. We might be able to use NERSC's GitLab as well.

About implementing other warpx options: of course one can add other option to the building procedure in the .def files.

Yup, that's what I was thinking, too 👍