instance start failed with --fakeroot

[yucancook]

Version of Apptainer

What version of Apptainer (or Singularity) are you using?
$ apptainer --version
apptainer version 1.2.2-1.el7

Expected behavior

What did you expect to see when you do...?
instance should start with --fakeroot

Actual behavior

What actually happened? Why was it incorrect?

Steps to reproduce this behavior

How can others reproduce this issue/problem?

$ apptainer instance start --fakeroot docker://alpine a1
INFO: Using cached SIF image
INFO: User not listed in /etc/subuid, trying root-mapped namespace
INFO: Using cached SIF image
INFO: Using fakeroot command combined with root-mapped namespace
ERROR: container cleanup failed: no instance found with name a1
FATAL: container creation failed: while applying cgroups config: Interactive authentication required.

FATAL: while executing starter: failed to start instance: while running /usr/libexec/apptainer/bin/starter: exit status 255

What OS/distro are you running

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

How did you install Apptainer

RPM

$ rpm -qa|egrep apptainer
apptainer-1.2.2-1.el7.x86_64

[GodloveD]

I'm able to replicate this error with v1.2.4 so I don't think it has been fixed by recent cgroups PRs.

$ apptainer --version
apptainer version 1.2.4

$ apptainer instance start --fakeroot docker://alpine a1
INFO:    Using cached SIF image
ERROR:   container cleanup failed: no instance found with name a1
FATAL:   container creation failed: while applying cgroups config: unable to start unit "apptainer-890424.scope" (properties [{Name:Description Value:"libcontainer container 890424"} {Name:Slice Value:"system.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [890424]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Interactive authentication required.

FATAL:   while executing starter: failed to start instance: while running /opt/apptainer/1.2.4/libexec/apptainer/bin/starter: exit status 255

[DrDaveD]

Yes and this problem doesn't happen in 1.1.9, but then there's a different problem of losing track of the instance (because the instance info goes into ~/.apptainer/instances/app/$(uname -n)/root instead of using the user's login name).

@JasonYangShadow can you please look into this?

[JasonYangShadow]

hmm, I tried investigating this issue.

1. It looks like that it is related to creation failure (systemd related thing) as shown in the error message,

RROR   [U=0,P=426377]     Master()                      container cleanup failed: no instance found with name a1
FATAL   [U=0,P=426377]     Master()                      container creation failed: while applying cgroups config: unable to start unit "apptainer-426378.scope" (properties [{Name:Description Value:"libcontainer container 426378"} {Name:Slice Value:"system.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [426378]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Interactive authentication required.
VERBOSE [U=1000,P=426369]  wait_child()                  instance exited with status 255

FATAL   [U=1000,P=426363]  instanceAction()              while executing starter: failed to start instance: while running /usr/local/libexec/apptainer/bin/starter: exit status 255

I also tried printing out the configurations (common.config & engine.config) (paste it to json viewer)
engine.config:

{"jsonConfig":{"unixSocketPair":[0,0],"image":"/home/vagrant/test/alpine.sif","imageArg":"alpine.sif","configdir":"/home/vagrant/.apptainer","cgroupsJSON":"{}","homedir":"/home/vagrant","homeDest":"/root","cwd":"/home/vagrant/test","configurationFile":"/usr/local/etc/apptainer/apptainer.conf","instance":true,"fakeroot":true,"restoreUmask":true,"umask":2,"dmtcpConfig":{},"xdgRuntimeDir":"/run/user/1000","dbusSessionBusAddress":"unix:path=/run/user/1000/bus","userInfo":{}},"ociConfig":{"ociVersion":"","process":{"user":{"uid":0,"gid":0},"args":["/.singularity.d/actions/start"],"env":["APPTAINER_CONTAINER=/home/vagrant/test/alpine.sif","SINGULARITY_CONTAINER=/home/vagrant/test/alpine.sif","APPTAINER_NAME=alpine.sif","SINGULARITY_NAME=alpine.sif","APPTAINER_BIND=","SINGULARITY_BIND=","APPTAINER_INSTANCE=a1","SINGULARITY_INSTANCE=a1","SHELL=/bin/bash","COLORTERM=truecolor","TERM_PROGRAM_VERSION=1.83.1","GPG_TTY=/dev/pts/1","PWD=/home/vagrant/test","LOGNAME=vagrant","XDG_SESSION_TYPE=tty","VSCODE_GIT_ASKPASS_NODE=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/node","MOTD_SHOWN=pam","LANG=C.UTF-8","LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:","GIT_ASKPASS=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/extensions/git/dist/askpass.sh","SSH_CONNECTION=192.168.0.103 45562 10.0.2.15 22","VSCODE_GIT_ASKPASS_EXTRA_ARGS=","LESSCLOSE=/usr/bin/lesspipe %s %s","XDG_SESSION_CLASS=user","TERM=xterm-256color","LESSOPEN=| /usr/bin/lesspipe %s","USER=vagrant","VSCODE_GIT_IPC_HANDLE=/run/user/1000/vscode-git-36252595d0.sock","SHLVL=2","XDG_SESSION_ID=3","XDG_RUNTIME_DIR=/run/user/1000","SSH_CLIENT=192.168.0.103 45562 22","VSCODE_GIT_ASKPASS_MAIN=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/extensions/git/dist/askpass-main.js","XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop","BROWSER=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/helpers/browser.sh","DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus","OLDPWD=/home/vagrant","TERM_PROGRAM=vscode","VSCODE_IPC_HOOK_CLI=/run/user/1000/vscode-ipc-311bc2ca-4078-4a4f-a445-8b660733751d.sock","_=/usr/local/bin/apptainer","USER_PATH=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/remote-cli:/home/vagrant/.local/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/go/bin:/home/vagrant/go/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin","HOME=/root","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","APPTAINER_APPNAME=","SINGULARITY_APPNAME="],"cwd":"/home/vagrant/test"},"linux":{"namespaces":[{"type":"pid"},{"type":"user"}]}},"fileConfig":{"AllowSetuid":true,"AllowPidNs":true,"ConfigPasswd":true,"ConfigGroup":true,"ConfigResolvConf":true,"MountProc":true,"MountSys":true,"MountDevPts":true,"MountHome":true,"MountTmp":true,"MountHostfs":false,"UserBindControl":true,"EnableFusemount":true,"EnableUnderlay":true,"MountSlave":true,"AllowContainerSIF":true,"AllowContainerEncrypted":true,"AllowContainerSquashfs":true,"AllowContainerExtfs":true,"AllowContainerDir":true,"AllowSetuidMountEncrypted":true,"AllowSetuidMountSquashfs":true,"AllowSetuidMountExtfs":false,"AlwaysUseNv":false,"UseNvCCLI":false,"AlwaysUseRocm":false,"SharedLoopDevices":false,"MaxLoopDevices":256,"SessiondirMaxSize":64,"MountDev":"yes","EnableOverlay":"try","BindPath":["/etc/localtime","/etc/hosts"],"LimitContainerOwners":[],"LimitContainerGroups":[],"LimitContainerPaths":[],"AllowNetUsers":[],"AllowNetGroups":[],"AllowNetNetworks":[],"RootDefaultCapabilities":"full","MemoryFSType":"tmpfs","CniConfPath":"","CniPluginPath":"","BinaryPath":"/usr/local/libexec/apptainer/bin:/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/remote-cli:/home/vagrant/.local/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","SuidBinaryPath":"/usr/local/libexec/apptainer/bin:/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/remote-cli:/home/vagrant/.local/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","MksquashfsProcs":0,"MksquashfsMem":"","ImageDriver":"fuseapps","DownloadConcurrency":3,"DownloadPartSize":5242880,"DownloadBufferSize":32768,"SystemdCgroups":true}}

common.config

{"engineName":"apptainer","containerID":"a1","engineConfig":{"jsonConfig":{"unixSocketPair":[0,0],"image":"/home/vagrant/test/alpine.sif","imageArg":"alpine.sif","configdir":"/home/vagrant/.apptainer","cgroupsJSON":"{}","homedir":"/home/vagrant","homeDest":"/root","cwd":"/home/vagrant/test","configurationFile":"/usr/local/etc/apptainer/apptainer.conf","instance":true,"fakeroot":true,"restoreUmask":true,"umask":2,"dmtcpConfig":{},"xdgRuntimeDir":"/run/user/1000","dbusSessionBusAddress":"unix:path=/run/user/1000/bus","userInfo":{}},"ociConfig":{"ociVersion":"","process":{"user":{"uid":0,"gid":0},"args":["/.singularity.d/actions/start"],"env":["APPTAINER_CONTAINER=/home/vagrant/test/alpine.sif","SINGULARITY_CONTAINER=/home/vagrant/test/alpine.sif","APPTAINER_NAME=alpine.sif","SINGULARITY_NAME=alpine.sif","APPTAINER_BIND=","SINGULARITY_BIND=","APPTAINER_INSTANCE=a1","SINGULARITY_INSTANCE=a1","SHELL=/bin/bash","COLORTERM=truecolor","TERM_PROGRAM_VERSION=1.83.1","GPG_TTY=/dev/pts/1","PWD=/home/vagrant/test","LOGNAME=vagrant","XDG_SESSION_TYPE=tty","VSCODE_GIT_ASKPASS_NODE=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/node","MOTD_SHOWN=pam","LANG=C.UTF-8","LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:","GIT_ASKPASS=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/extensions/git/dist/askpass.sh","SSH_CONNECTION=192.168.0.103 45562 10.0.2.15 22","VSCODE_GIT_ASKPASS_EXTRA_ARGS=","LESSCLOSE=/usr/bin/lesspipe %s %s","XDG_SESSION_CLASS=user","TERM=xterm-256color","LESSOPEN=| /usr/bin/lesspipe %s","USER=vagrant","VSCODE_GIT_IPC_HANDLE=/run/user/1000/vscode-git-36252595d0.sock","SHLVL=2","XDG_SESSION_ID=3","XDG_RUNTIME_DIR=/run/user/1000","SSH_CLIENT=192.168.0.103 45562 22","VSCODE_GIT_ASKPASS_MAIN=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/extensions/git/dist/askpass-main.js","XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop","BROWSER=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/helpers/browser.sh","DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus","OLDPWD=/home/vagrant","TERM_PROGRAM=vscode","VSCODE_IPC_HOOK_CLI=/run/user/1000/vscode-ipc-311bc2ca-4078-4a4f-a445-8b660733751d.sock","_=/usr/local/bin/apptainer","USER_PATH=/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/remote-cli:/home/vagrant/.local/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/go/bin:/home/vagrant/go/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin","HOME=/root","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","APPTAINER_APPNAME=","SINGULARITY_APPNAME="],"cwd":"/home/vagrant/test"},"linux":{"namespaces":[{"type":"pid"},{"type":"user"}]}},"fileConfig":{"AllowSetuid":true,"AllowPidNs":true,"ConfigPasswd":true,"ConfigGroup":true,"ConfigResolvConf":true,"MountProc":true,"MountSys":true,"MountDevPts":true,"MountHome":true,"MountTmp":true,"MountHostfs":false,"UserBindControl":true,"EnableFusemount":true,"EnableUnderlay":true,"MountSlave":true,"AllowContainerSIF":true,"AllowContainerEncrypted":true,"AllowContainerSquashfs":true,"AllowContainerExtfs":true,"AllowContainerDir":true,"AllowSetuidMountEncrypted":true,"AllowSetuidMountSquashfs":true,"AllowSetuidMountExtfs":false,"AlwaysUseNv":false,"UseNvCCLI":false,"AlwaysUseRocm":false,"SharedLoopDevices":false,"MaxLoopDevices":256,"SessiondirMaxSize":64,"MountDev":"yes","EnableOverlay":"try","BindPath":["/etc/localtime","/etc/hosts"],"LimitContainerOwners":[],"LimitContainerGroups":[],"LimitContainerPaths":[],"AllowNetUsers":[],"AllowNetGroups":[],"AllowNetNetworks":[],"RootDefaultCapabilities":"full","MemoryFSType":"tmpfs","CniConfPath":"","CniPluginPath":"","BinaryPath":"/usr/local/libexec/apptainer/bin:/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/remote-cli:/home/vagrant/.local/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","SuidBinaryPath":"/usr/local/libexec/apptainer/bin:/home/vagrant/.vscode-server/bin/f1b07bd25dfad64b0167beb15359ae573aecd2cc/bin/remote-cli:/home/vagrant/.local/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/go/bin:/home/vagrant/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","MksquashfsProcs":0,"MksquashfsMem":"","ImageDriver":"fuseapps","DownloadConcurrency":3,"DownloadPartSize":5242880,"DownloadBufferSize":32768,"SystemdCgroups":true}},"plugin":null}

I could not find suspicious values (might miss something)

2. It has not reached to write instance json into disk yet, it errors before calling PostStartProcess (

   apptainer/internal/pkg/runtime/engine/apptainer/process_linux.go

   Line 335 in 73e06fc

   func (e *EngineOperations) PostStartProcess(ctx context.Context, pid int) error {

   ), it fails in this method starterInstance (

   apptainer/internal/pkg/runtime/launch/launcher_linux.go

   Line 1176 in 73e06fc

   cmdErr := starter.Run(

   )

[JasonYangShadow]

continue from previous post

it fails here
https://github.com/apptainer/apptainer/blob/73e06fc047c89bfba1b3c51a3dac75136a1f2d4e/internal/pkg/cgroups/manager_linux.go#L365C19-L365C19

with some useful values here
https://github.com/apptainer/apptainer/blob/73e06fc047c89bfba1b3c51a3dac75136a1f2d4e/internal/pkg/cgroups/manager_linux.go#L360C16-L360C16
spec:

{[] <nil> <nil> <nil> <nil> [] <nil> map[] map[]}

group:

system.slice:apptainer:434164

systemd:

true

when I changed the systemd=false
it'll show the following errors

ERROR   [U=0,P=435284]     Master()                      container cleanup failed: no instance found with name a1
FATAL   [U=0,P=435284]     Master()                      container creation failed: while applying cgroups config: mkdir /sys/fs/cgroup/user.slice/user-1000.slice/system.slice:apptainer:435285: permission denied
VERBOSE [U=1000,P=435274]  wait_child()                  instance exited with status 255

FATAL   [U=1000,P=435266]  instanceAction()              while executing starter: failed to start instance: while running /usr/local/libexec/apptainer/bin/starter: exit status 255

I feel like is it caused by opencontainers package upgrades? or should it by default fail because of not enough permission?

[DrDaveD]

It doesn't need to be an opencontainers package upgrade, because running an instance in a cgroup by default is a new 1.2.0 feature. Perhaps using a cgroup just needs to be avoided when using the --fakeroot option. You might also want to try to see if the same issue happens in singularity-ce, because this feature was imported from that project.

[JasonYangShadow]

It doesn't need to be an opencontainers package upgrade, because running an instance in a cgroup by default is a new 1.2.0 feature. Perhaps using a cgroup just needs to be avoided when using the --fakeroot option. You might also want to try to see if the same issue happens in singularity-ce, because this feature was imported from that project.

singularity works fine by default (see below comment)

vagrant@vagrant:~$ singularity --version
singularity-ce version 4.0.1-jammy
vagrant@vagrant:~$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
vagrant@vagrant:~$ singularity instance start --fakeroot docker://alpine a1
INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...
INFO:    instance started successfully
vagrant@vagrant:~$ singularity instance list
INSTANCE NAME    PID     IP    IMAGE
a1               1828          /tmp/rootfs-1074526444/root
vagrant@vagrant:~$ singularity instance stop a1
INFO:    Stopping a1 instance of /tmp/rootfs-1074526444/root (PID=1828)

[JasonYangShadow]

it's working in suid installation mode or userns enabled mode.

If I build apptainer with --with-suid, apptainer works fine the same as singularity does

vagrant@ubuntu:~/test$ apptainer instance start --fakeroot alpine.sif a1
INFO:    instance started successfully
vagrant@ubuntu:~/test$ apptainer instance stop a1
INFO:    Stopping a1 instance of /home/vagrant/test/alpine.sif (PID=535383)

also works with --userns mode

vagrant@ubuntu:~/test$ apptainer instance start --userns alpine.sif a1
INFO:    instance started successfully
vagrant@ubuntu:~/test$ apptainer instance stop a1
INFO:    Stopping a1 instance of /home/vagrant/test/alpine.sif (PID=612353)

compiled with --with-suid but disabled suid will also fail

vagrant@ubuntu:~/test$ apptainer instance start --ignore-subuid --fakeroot alpine.sif a1
INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    Using fakeroot command combined with root-mapped namespace
ERROR:   container cleanup failed: no instance found with name a1
FATAL:   container creation failed: while applying cgroups config: unable to start unit "apptainer-642505.scope" (properties [{Name:Description Value:"libcontainer container 642505"} {Name:Slice Value:"system.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [642505]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Interactive authentication required.

FATAL:   while executing starter: failed to start instance: while running /usr/local/libexec/apptainer/bin/starter: exit status 255

Also I compiled singularity with --without-suid and singularity also has the same issue

vagrant@vagrant:~$ singularity version
4.0.1
vagrant@vagrant:~$ singularity instance start --fakeroot alpine.sif a1
INFO:    Converting SIF file to temporary sandbox...
INFO:    Cleaning up image...
ERROR:   Container cleanup failed: no instance found with name a1
FATAL:   container creation failed: while applying cgroups config: unable to start unit "singularity-24974.scope" (properties [{Name:Description Value:"libcontainer container 24974"} {Name:Slice Value:"system.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [24974]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Interactive authentication required.

FATAL:   while executing starter: failed to start instance: while running /usr/local/libexec/singularity/bin/starter: exit status 255

[DrDaveD]

compiled with --with-suid but disabled suid will also fail

vagrant@ubuntu:~/test$ apptainer instance start --ignore-subuid --fakeroot alpine.sif a1

--ignore-subuid is not the same thing as disabled suid. It just disables /etc/subuid. But it's interesting that it has the same failure as being compiled --without-suid even when there is no /etc/subuid configured (as reported in the original description).

[JasonYangShadow]

compiled with --with-suid but disabled suid will also fail
vagrant@ubuntu:~/test$ apptainer instance start --ignore-subuid --fakeroot alpine.sif a1

--ignore-subuid is not the same thing as disabled suid. It just disables /etc/subuid. But it's interesting that it has the same failure as being compiled --without-suid even when there is no /etc/subuid configured (as reported in the original description).

thanks for pointing out the wrong part. Yeah, I've misunderstood the --ignore-subuid flag.

[JasonYangShadow]

another update after doing more investigation. And thanks to some clues shared by @cclerget

1. This issue is related to cgroup v2, (v1 does not have such issue, non-suid apptainer)

vagrant@vagrant:~$ apptainer version
1.2.4
vagrant@vagrant:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
vagrant@vagrant:~$ ls
alpine_latest.sif  apptainer_1.2.4_amd64.deb
vagrant@vagrant:~$ apptainer instance start --fakeroot alpine_latest.sif a1
INFO:    Instance stats will not be available - requires cgroups v2 with systemd as manager.
INFO:    instance started successfully
vagrant@vagrant:~$ apptainer instance list
INSTANCE NAME    PID      IP    IMAGE
a1               11406          /home/vagrant/alpine_latest.sif
vagrant@vagrant:~$ apptainer instance stop a1
INFO:    Stopping a1 instance of /home/vagrant/alpine_latest.sif (PID=11406)

2. I found a similar ticket in moby community, Docker Rootless Ubuntu 22.04 - Unable to apply cgroup configuration (...: Interactive authentication required.: unknown.) moby/moby#45014, the fix is somehow unrelated to container, but cgroup v2.
3. I could not solve this issue by following some docs on google like setting several env vars

XDG_RUNTIME_DIR=/run/user/$(id -u)
DBUS_SESSION_BUS_ADDRESS=unix:path=${XDG_RUNTIME_DIR}/bus
export DBUS_SESSION_BUS_ADDRESS
export XDG_RUNTIME_DIR

[JasonYangShadow]

continuous updates

This issue is related to cgroup v2, (v1 does not have such issue, non-suid apptainer)

on cgroup v1, apptainer instance start --fakeroot actually does not use cgroup at all. And that's the reason why this command works on ubuntu 20.04.

In this PR
#1773
I tried aligning behavior for command apptainer instance start --fakeroot on cgroup v1 and v2. Ignoring the usage of cgroup when --fakeroot is passed.