Apptainer 1.3.6

v1.3.6 - [2024-12-02]

• Avoid using kernel overlayfs when the lower layer is a sandbox on an incompatible filesystem type such as GPFS or Lustre. For those cases use fuse-overlayfs instead. This fixes a regression introduced in 1.3.0. The regression didn't much impact Lustre because kernel overlayfs refused to try to use it and Apptainer proceeded to use fuse-overlayfs anyway, but with GPFS the kernel overlayfs allowed mounting but returned stale file handle errors.

Apptainer 1.3.5

v1.3.5 - [2024-10-30]

• Fix a regression introduced in 1.3.4 that overwrote existing standard /.singularity.d files such as runscript in container images even if they had been modified.
• Skip attempting to bind inaccessible mount points when handling the mount hostfs = yes configuration option.
• Support parsing nested variables defined inside %arguments section of definition files.
• Ignore invalid environment variables when pulling oci/docker containers.

Apptainer 1.3.4

v1.3.4 - [2024-09-04]

• Fixed sif-embedded overlay partitions for containers that are larger than 2 gigabytes.
• Fixed the apparmor profile that was added in v1.3.3 but didn't work. An apparmor profile is applied in all Debian-based apptainer packaging, but is only needed to enable user namespaces for apptainer on a default-configured Ubuntu 23.10 or newer.
• Fixed the failure when starting apptainer with instance --fakeroot.
• apptainer build -B ... can now be used to mount custom resolv.conf and hosts files from non-standard outside locations. This can be used to run apptainer build in a nix-build sandbox that has no /etc/resolv.conf.
• Fixed failing builds from local images that have symbolic links for paths that are part of the base container environment (e.g. /var/tmp -> /tmp).
• Show info messages suggesting to use enable underlay = preferred or the --underlay flag when overlay is implied for bind mounts but the kernel is too old to support fuse mounts in user namespaces and so tries to use fusermount.
• When someone uses a yum bootstrap to build a container without using subuid-based fakeroot or root, warn that it is unlikely to work.
• Allow a writable --overlay to be used with --nvccli instead of --writable-tmpfs.
• If an error "no descriptor found for reference" is seen while getting an oci container, retry the operation up to five times.
• Make fakeroot Recommended for SUSE rpms instead of Required.
• Allow bind mounts onto existing files on r/o NFS filesystems.
• If an error is seen in the %post section when building a container using fakeroot mode 3 (with the fakeroot command) then show a message suggesting using --ignore-fakeroot-command and referring to the documentation about how to install and use it inside the container definition file.
• Show a more helpful error message when using fakeroot in suid mode and there's an /etc/subuid mapping even though user namespaces are not available (user namespaces are required for /etc/subuid mapping).

Apptainer 1.3.3

v1.3.3 - [2024-07-03]

• Updated the minimum golang version to 1.21.
• Removed support for EL7.
• Added libcudadebugger.so to nvliblist.conf to support cuda-gdb in CUDA 12+.
• Ensure opened/kept file descriptors in stage 1 are not closed during the Go garbage collection to avoid "bad file descriptor" errors at startup.
• Fixed a segmentation violation issue when running Apptainer checkpoint.
• Added apparmor profiles for ubuntu 24.04 or higher distros.
• Fixed an issue that Apptainer won't read default docker credentials.

v1.3.2

v1.3.2 - [2024-05-28]

Security fix

• Included a fix for CVE-2024-3727 in a dependent library which describes a flaw that can allow attackers to trigger unexpected authenticated registry accesses due to object digest values not being validated in all cases.

Other Changes

• Fixed the issue when nesting apptainer instance start inside a container on cgroups-v2 capable host.
• Fixed the issue that oras download progress bar gets stuck when downloading large images.

v1.3.1

v1.3.1 - [2024-04-24]

• Make 'apptainer build' work with signed Docker containers.
• Fixed regression introduced in 1.3.0 that prevented closing cryptsetup and the corresponding loop device after running an encrypted sif container file in suid mode.
• Stopped binding over the default timezone in the container with the host's timezone, which led to unexpected behavior if the application changed timezones.
• Added progress bars for oras:// push and pull.
• Hide Instance stats will not be available message under --sharens mode.
• Fix problem where credentials locally stored with registry login command were not usable in some execution flows. Run registry login again with latest version to ensure credentials are stored correctly.
• Make runscript timeout configurable.
• Return invalid bind path mount options during bind path parsing.
• Make the INFO message more helpful when a running background process at exit time causes a FUSE mount to not shut down cleanly.
• Fixed the wrong mediaType in the oras push manifest.

Apptainer 1.3.0

v1.3.0 - [2024-03-12]

Changes since v1.2.5

Security

This release fixes two moderate severity denial of service vulnerabilities by upgrading a dependent library: CVE-2024-28176 and CVE-2024-28180.

Changed defaults / behaviours

• FUSE mounts are now supported in setuid mode, enabling full functionality even when kernel filesystem mounts are insecure due to unprivileged users having write access to raw filesystems in containers.

  When allow setuid-mount extfs = no (the default) in apptainer.conf, then the fuse2fs image driver will be used to mount ext3 images in setuid mode instead of the kernel driver (ext3 images are primarily used for the --overlay feature), restoring functionality that was removed by default in Apptainer 1.1.8 because of the security risk.

  The allow setuid-mount squashfs configuration option in apptainer.conf now has a new default called iflimited which allows kernel squashfs mounts only if there is at least one limit container option set or if Execution Control Lists are activated in ecl.toml. If kernel squashfs mounts are are not allowed, then the squashfuse image driver will be used instead.
  iflimited is the default because if one of those limits are used the system administrator ensures that unprivileged users do not have write access to the containers, but on the other hand using FUSE would enable a user to theoretically bypass the limits via ptrace() because the FUSE process runs as that user.

  The fuse-overlayfs image driver will also now be tried in setuid mode if the kernel overlayfs driver does not work (for example if one of the layers is a FUSE filesystem).

  In addition, if allow setuid-mount encrypted = no then the unprivileged gocryptfs format will be used for encrypting SIF files instead of the kernel device-mapper. If a SIF file was encrypted using the gocryptfs format, it can now be mounted in setuid mode in addition to non-setuid mode.

• The four dependent FUSE programs for various reasons all now need to be compiled from source and included in Apptainer installations and packages.
  Scripts are provided to make this easy; see the updated instructions in INSTALL.md. The bundled squashfuse_ll is updated to version 0.5.1.

• Change the default in user namespace mode to use either kernel overlayfs or fuse-overlayfs instead of the underlay feature for the purpose of adding bind mount points. That was already the default in setuid mode; this change makes it consistent. The underlay feature can still be used with the --underlay option, but it is deprecated because the implementation is complicated and measurements have shown that the performance of underlay is similar to overlayfs and fuse-overlayfs.
  For now the underlay feature can be made the default again with a new preferred value on the enable underlay configuration option.
  Also the --underlay option can be used in setuid mode or as the root user, although it was ignored previously.

• Prefer again to use kernel overlayfs over fuse-overlayfs when a lower layer is FUSE and there's no writable upper layer, undoing the change from 1.2.0. Another workaround was found for the problem that change addressed. This applies in both setuid mode and in user namespace mode (except the latter not on CentOS7 where it isn't supported).

• --cwd is now the preferred form of the flag for setting the container's working directory, though --pwd is still supported for compatibility.

• When building RPM, we will now use /var/lib/apptainer (rather than /var/apptainer) to store local state files.

• The way --home is handled when running as root (e.g. sudo apptainer) or with --fakeroot has changed. Previously, we were only modifying the HOME environment variable in these cases, while leaving the container's /etc/passwd file unchanged (with its homedir field pointing to /root, regardless of the value passed to --home). With this change, both value of HOME and the contents of /etc/passwd in the container will reflect the value passed to --home if the container is readonly. If the container is writable, the /etc/passwd file is left alone because it can interfere with commands that want to modify it.

• The --vm and related flags to start apptainer inside a VM have been removed. This functionality was related to the retired Singularity Desktop / SyOS projects.

• The keyserver-related commands that were under remote have been moved to their own, dedicated keyserver command. Run apptainer help keyserver for more information.

• The commands related to OCI/Docker registries that were under remote have been moved to their own, dedicated registry command. Run apptainer help registry for more information.

• The the remote list subcommand now outputs only remote endpoints (with keyservers and OCI/Docker registries having been moved to separate commands), and the output has been streamlined.

• Adding a new remote endpoint using the apptainer remote add command will now set the new endpoint as default. This behavior can be suppressed by supplying the --no-default (or -n) flag to remote add.

• Skip parsing build definition file template variables after comments beginning with a hash symbol.

• Improved the clarity of apptainer key list output.

• The global /tmp directory is no longer used for gocryptfs mountpoints.

• Updated minimum go version to 1.20

New Features & Functionality

• The remote status command will now print the username, realname, and email of the logged-in user, if available.
• Add monitoring feature support, which requires the usage of an additional tool named apptheus, this tool will put apptainer starter into a newly created cgroup and collect system metrics.
• A new --no-pid flag for apptainer run/shell/exec disables the PID namespace inferred by --containall and --compat.
• Added --config option tokeyserver commands.
• Honor an optional remoteName argument to the keyserver list command.
• Added the APPTAINER_ENCRYPTION_PEM_DATA env var to allow for encrypting and running encrypted containers without a PEM file.
• Adding --sharens mode for apptainer exec/run/shell, which enables to run multiple apptainer instances created by the same parent using the same image in the same user namespace.

Developer / API

• Changes in pkg/build/types.Definition struct. New .FullRaw field introduced, which always contains the raw data for the entire definition file. Behavior of .Raw field has changed: for multi-stage builds parsed with pkg/build/types/parser.All(), .Raw contains the raw content of a single build stage. Otherwise, it is equal to .FullRaw.

Bug fixes

• Don't bind /var/tmp on top of /tmp in the container, where /var/tmp resolves to same location as /tmp.
• Support parentheses in test / [ commands in container startup scripts, via dependency update of mvdan.cc/sh.
• Fix regression introduced in v1.2.0 that led to an empty user's shell field in the /etc/passwd file.
• Prevent container builds from failing when $HOME points to a non-readable directory.
• Fix the use of nvidia-container-cli on Ubuntu 22.04 where an ldconfig wrapper script gets in the way. Instead, we use ldconfig.real directly.
• Run image drivers with CAP_DAC_OVERRIDE in user namespace mode. This fixes --nvccli with NVIDIA_DRIVER_CAPABILITIES=graphics, which previously failed when using fuse-overlayfs.

Release change

• Releases will generate apptainer Docker images for the Linux amd64 and arm64 architectures at ghcr.io/apptainer/apptainer.

Apptainer 1.3.0 Release Candidate 2

v1.3.0-rc.2 - [2024-02-15]

Changes since v1.3.0-rc.1

• Change the default in user namespace mode to use either kernel overlayfs or fuse-overlayfs instead of the underlay feature for the purpose of adding bind mount points. That was already the default in setuid mode; this change makes it consistent. The underlay feature can still be used with the --underlay option, but it is deprecated because the implementation is complicated and measurements have shown that the performance of underlay is similar to overlayfs and fuse-overlayfs. For now the underlay feature can be made the default again with a new preferred value on the enable underlay configuration option. Also the --underlay option can be used in setuid mode or as the root user, although it was ignored previously.
• Prefer again to use kernel overlayfs over fuse-overlayfs when a lower layer is FUSE and there's no writable upper layer, undoing the change from 1.2.0. Another workaround was found for the problem that change addressed. This applies in both setuid mode and in user namespace mode (except the latter not on CentOS7 where it isn't supported).
• Fix the use of an overlay ext3 filesystem in SIF files.
• Fix --sharens failure on EL8.
• Fix Harbor registry login failure.
• Prevent container builds from failing when $HOME points to a non-readable directory.

Apptainer 1.3.0 Release Candidate 1

v1.3.0-rc.1 - [2024-01-10]

Changes since v1.2.5

Changed defaults / behaviours

• FUSE mounts are now supported in setuid mode, enabling full functionality even when kernel filesystem mounts are insecure due to unprivileged users having write access to raw filesystems in containers.

  When allow setuid-mount extfs = no (the default) in apptainer.conf, then the fuse2fs image driver will be used to mount ext3 images in setuid mode instead of the kernel driver (ext3 images are primarily used for the --overlay feature), restoring functionality that was removed by default in Apptainer 1.1.8 because of the security risk.

  The allow setuid-mount squashfs configuration option in apptainer.conf now has a new default called iflimited which allows kernel squashfs mounts only if there is at least one limit container option set or if Execution Control Lists are activated in ecl.toml. If kernel squashfs mounts are are not allowed, then the squashfuse image driver will be used instead. iflimited is the default because if one of those limits are used the system administrator ensures that unprivileged users do not have write access to the containers, but on the other hand using FUSE would enable a user to theoretically bypass the limits via ptrace() because the FUSE process runs as that user.

  The fuse-overlayfs image driver will also now be tried in setuid mode if the kernel overlayfs driver does not work (for example if one of the layers is a FUSE filesystem).

  In addition, if allow setuid-mount encrypted = no then the unprivileged gocryptfs format will be used for encrypting SIF files instead of the kernel device-mapper. If a SIF file was encrypted using the gocryptfs format, it can now be mounted in setuid mode in addition to non-setuid mode.

• The four dependent FUSE programs for various reasons all now need to be compiled from source and included in Apptainer installations and packages. Scripts are provided to make this easy; see the updated instructions in INSTALL.md.

• --cwd is now the preferred form of the flag for setting the container's working directory, though --pwd is still supported for compatibility.

• When building RPM, we will now use /var/lib/apptainer (rather than /var/apptainer) to store local state files.

• The way --home is handled when running as root (e.g. sudo apptainer) or with --fakeroot has changed. Previously, we were only modifying the HOME environment variable in these cases, while leaving the container's /etc/passwd file unchanged (with its homedir field pointing to /root, regardless of the value passed to --home). With this change, both value of HOME and the contents of /etc/passwd in the container will reflect the value passed to --home if the container is readonly. If the container is writable, the /etc/passwd file is left alone because it can interfere with commands that want to modify it.

• The --vm and related flags to start apptainer inside a VM have been removed. This functionality was related to the retired Singularity Desktop / SyOS projects.

• The keyserver-related commands that were under remote have been moved to their own, dedicated keyserver command. Run apptainer help keyserver for more information.

• The commands related to OCI/Docker registries that were under remote have been moved to their own, dedicated registry command. Run apptainer help registry for more information.

• The the remote list subcommand now outputs only remote endpoints (with keyservers and OCI/Docker registries having been moved to separate commands), and the output has been streamlined.

• Adding a new remote endpoint using the apptainer remote add command will now set the new endpoint as default. This behavior can be suppressed by supplying the --no-default (or -n) flag to remote add.

• Skip parsing build definition file template variables after comments beginning with a hash symbol.

• Improved the clarity of apptainer key list output.

• The global /tmp directory is no longer used for gocryptfs mountpoints.

• Updated minimum go version to 1.20

New Features & Functionality

• The remote status command will now print the username, realname, and email of the logged-in user, if available.
• Add monitoring feature support, which requires the usage of an additional tool named apptheus, this tool will put apptainer starter into a newly created cgroup and collect system metrics.
• A new --no-pid flag for apptainer run/shell/exec disables the PID namespace inferred by --containall and --compat.
• Added --config option tokeyserver commands.
• Honor an optional remoteName argument to the keyserver list command.
• Added the APPTAINER_ENCRYPTION_PEM_DATA env var to allow for encrypting and running encrypted containers without a PEM file.
• Adding --sharens mode for apptainer exec/run/shell, which enables to run multiple apptainer instances created by the same parent using the same image in the same user namespace.

Developer / API

• Changes in pkg/build/types.Definition struct. New .FullRaw field introduced, which always contains the raw data for the entire definition file. Behavior of .Raw field has changed: for multi-stage builds parsed with pkg/build/types/parser.All(), .Raw contains the raw content of a single build stage. Otherwise, it is equal to .FullRaw.

Bug fixes

• Don't bind /var/tmp on top of /tmp in the container, where /var/tmp resolves to same location as /tmp.
• Support parentheses in test / [ commands in container startup scripts, via dependency update of mvdan.cc/sh.
• Fix regression introduced in v1.2.0 that led to an empty user's shell field in the /etc/passwd file.

Release change

• Releases will generate apptainer Docker images for the Linux amd64 and arm64 architectures.

Apptainer 1.2.5

v1.2.5 - [2023-11-21]

• Added libnvidia-nvvm to nvliblist.conf. Newer NVIDIA Drivers (known with >= 525.85.05) require this lib to compile OpenCL programs against NVIDIA GPUs, i.e. libnvidia-opencl depends on libnvidia-nvvm.
• Disable the usage of cgroup in instance creation when --fakeroot is passed.
• Disable the usage of cgroup in instance creation when hidepid mount option on /proc is set.