Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

oci: cgroup error in bare ssh session with crun only #1538

Closed
dtrudg opened this issue Apr 11, 2023 · 0 comments · Fixed by #1539
Closed

oci: cgroup error in bare ssh session with crun only #1538

dtrudg opened this issue Apr 11, 2023 · 0 comments · Fixed by #1539
Assignees
@dtrudg
Labels
bug Something isn't working

Comments

@dtrudg
Copy link
Member

dtrudg commented Apr 11, 2023

Version of Singularity

3.11.1

Describe the bug

When using crun, if singularity is started from a bare ssh login, which will be in a session.scope cgroup owned by root, the container cgroup is not successfully set up by crun:

03:04 pm $ singularity run --oci docker://alpine
2023/04/11 15:04:09  info unpack layer: sha256:f56be85fc22e46face30e2c3de3f7fe7c15f8fd7c4e5add29d7f64b87abdaa09
2023-04-11T14:04:10.121315Z: error creating systemd unit `crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope`: got `failed`

03:04 pm $ systemctl --user status -l --no-pager crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope
× crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope - libcrun container
     Loaded: loaded (/run/user/1001/systemd/transient/crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope; transient)
  Transient: yes
     Active: failed (Result: resources)
        CPU: 0

Apr 11 15:04:10 mini systemd[3306]: crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope: Couldn't move process 134245 to requested cgroup '/user.slice/user-1001.slice/user@1001.service/app.slice/crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope' (directly or via the system bus): Permission denied
Apr 11 15:04:10 mini systemd[3306]: crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope: Failed to add PIDs to scope's control group: Permission denied
Apr 11 15:04:10 mini systemd[3306]: crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope: Failed with result 'resources'.
Apr 11 15:04:10 mini systemd[3306]: Failed to start crun-ba95266e-f39e-4736-9f7b-43211f8a8808.scope - libcrun container.

03:04 pm $ cat /proc/self/cgroup 
0::/user.slice/user-1001.slice/session-8.scope

This does not happen with runc - which is able to create the container cgroup without any problem from a bare ssh login.

Starting a user scope, and running from within, is a workaround for crun...

dtrudg-sylabs@mini:~
03:07 pm $ systemd-run --user --scope singularity run --oci docker://alpine
Running scope as unit: run-rff8e8aeb43024c83852bf6bb260a5927.scope
2023/04/11 15:08:00  info unpack layer: sha256:f56be85fc22e46face30e2c3de3f7fe7c15f8fd7c4e5add29d7f64b87abdaa09
~ $

it appears that crun doesn't work around a non-user owned common ancestor cgroup, while runc does?
@dtrudg dtrudg added the bug Something isn't working label Apr 11, 2023
@dtrudg dtrudg self-assigned this Apr 11, 2023
dtrudg added a commit to dtrudg/singularity that referenced this issue Apr 11, 2023
@dtrudg
fix: oci: enter cgroup before executing crun as non-root
90cb5a1 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs#1538
dtrudg added a commit to dtrudg/singularity that referenced this issue Apr 11, 2023
@dtrudg
fix: oci: enter cgroup before executing crun as non-root
61871c7 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs#1538
@dtrudg dtrudg mentioned this issue Apr 11, 2023
Merged
dtrudg added a commit to dtrudg/singularity that referenced this issue Apr 11, 2023
@dtrudg
fix: oci: enter cgroup before executing crun as non-root
ef8647a 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs#1538
dtrudg added a commit to dtrudg/singularity that referenced this issue Apr 11, 2023
@dtrudg
fix: oci: enter cgroup before executing crun as non-root
e3dbc53 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs#1538
dtrudg added a commit to dtrudg/singularity that referenced this issue Apr 12, 2023
@dtrudg
fix: oci: enter cgroup before executing crun as non-root (release-3.11)
c5c46b0 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs#1538
@dtrudg dtrudg mentioned this issue Apr 12, 2023
Merged
dtrudg added a commit to dtrudg/singularity that referenced this issue Apr 13, 2023
@dtrudg
fix: oci: enter cgroup before executing crun as non-root (release-3.11)
7361cf4 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs#1538
edytuk pushed a commit to vzokay/apptainer that referenced this issue Apr 21, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
c06a69e 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue May 24, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
a25b03b 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue May 24, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
7d1dc19 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jun 14, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
734bc60 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jun 14, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
cb128e4 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jun 14, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
8f8aa22 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jun 16, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
33505d1 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jul 4, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
569b626 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jul 11, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
956ab65 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jul 21, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
d7e7efe 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
edytuk pushed a commit to vzokay/apptainer that referenced this issue Jul 24, 2023
@dtrudg @edytuk
fix: oci: enter cgroup before executing crun as non-root
1bb1254 
When executed from a root-owned cgroup, such as the session scope
resulting from a bare ssh login, crun will fail to create our
requested container cgroup.

If we are running as non-root, create and move into a user-owned
cgroup, so that there's a common user-owned ancestor. This avoids the
`crun` error.

Note that no workaround is needed for `runc` as it is able to create
the requested container cgroup without any issue.

Fixes sylabs/singularity#1538

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dtrudg dtrudg

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: oci: enter cgroup before executing crun as non-root dtrudg/singularity
1 participant
@dtrudg