Roadmap - SingularityCE 4.2

[dtrudg]

SingularityCE 4.2 is targeted for release in the period July-September 2024.

Please use this thread to comment and suggest features for inclusion in 4.2. We'll update this top post as items are suggested and discussed.

The focus of SingularityCE 4.2 will be continued improvements for complex workflows and interoperability in OCI-mode.

4.2 Planned Features

Drop accommodations for RHEL7 / SLES12

#2849

RHEL7 is mainstream EOL in July 2024, followed by SLES12 in October 2024. These distributions have very old kernels and basic userspace libraries that require workarounds and restrictions on SingularityCE functionality. We can clean up code and reduce the number of paths that are taken by dropping support for these distributions.

Push from OCI-SIF to OCI registries in native OCI tar layer format

#2848

Allow OCI-SIF images to be trivially pushed to an OCI registry in the standard OCI format with tar layers. This will allow direct consumption by non-singularity runtimes.

Overlays in images for OCI-mode

#1480

OCI-mode does not support using an overlay that is an extfs/squashfs file inside a SIF. This should be addressed. Will support be included for non OCI-SIF images with an embedded overlay? How will ro / rw concerns be handled? An overlay may conceptually be a new layer in an image, but OCI layers are read only.

Image Binds from OCI-SIF

#1729

In native mode, SIF files can be created that hold data (which is not in the form of an embedded overlay but a data partition), and this data can be mounted out of the SIF to a container via e.g. --bind my.sif:/my-dest:image-src=/.

This needs to be supported in OCI-Mode. Presumably we should be able to mount non-OCI-SIF data partitions, but also there should be a structure for keeping data in an OCI-SIF file.

Dockerfile Build Optimisation

Issue: #2308

The Dockerfile build process uses an intermediate streaming of output to a tar, which consumes disk space and I/O bandwidth. This can become an issue when building larger containers (e.g. from tensorflow base images etc). Use a streaming approach into SIF conversion, and identify and implement any other easy performance optimisations.

4.2 Under Consideration

Instances for OCI-mode

OCI-mode doesn't support background containers, which are known as instances in the native runtime. We may wish to either:

• Emulate the native mode instance lifecycle in OCI mode, for CLI compatibility with native mode.
• Support background containers following the OCI lifecycle. This would not be truly compatible with that native mode instances.

CDI Support in native mode

Issue: #1395

OCI-Mode implements the Container Device Interface standard for exposing devices inside a container via the --device flag. We

Expose cgroups namespace as an option for native singularity runtimes

The cgroups namespace will be addressed for the OCI runtime in 3.10. Consider allowing it to be requested for the native singularity runtime.