utils: ignore ENOTSUP when chmod a symlink #1309

giuseppe · 2023-09-22T09:37:07Z

commit 5d1f903f75a80daa4dfb3d84e114ec8ecbf29956 in the kernel, present in a release since Linux 6.6 doesn't allow anymore to change the mode of a symlink, so just ignore the failure.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d1f903f75a80daa4dfb3d84e114ec8ecbf29956

Closes: #1308

commit 5d1f903f75a80daa4dfb3d84e114ec8ecbf29956 in the kernel, present in a release since Linux 6.6 doesn't allow anymore to change the mode of a symlink, so just ignore the failure. Closes: containers#1308 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

src/libcrun/utils.c

flouthoc

LGTM , just a small question above.

Luap99 · 2023-09-22T10:00:21Z

Did you ask on the kernel list to revert or at least not backport this change?
I am asking because the linked commit asked for backports to stable releases which means this will start breaking on older LTS releases unless crun gets updated there which seems unlikely.

giuseppe · 2023-09-22T10:26:24Z

no, I've not asked it yet.

@brauner even if the issue is in user space (as in our case), do you think the mentioned patch "attr: block mode changes of symlinks" should be reverted on LTS releases to prevent such kind of errors when the kernel is updated?

brauner · 2023-09-22T12:06:05Z

no, I've not asked it yet.

@brauner even if the issue is in user space (as in our case), do you think the mentioned patch "attr: block mode changes of symlinks" should be reverted on LTS releases to prevent such kind of errors when the kernel is updated?

I would leave it in the LTS as well to make it consistent what you did above could fail before btw. It depends on the fstype whether this breaks and whether posix acls are enabled or not. That's why this was blocked in the first place as it only works sometimes.

The code you added in there isn't guaranteed to work and why both glibc and musl removed it. If you want to insist on it getting reverted from the LTS than you can probably get Greg to do this but if you can live with fixing this up here we're all better off imho.

kolyshkin

LGTM

NB1: ENOTSUP == EOPNOTSUPP in glibc (but not in the kernel).

NB2: The code to handle ENOTSUP was added in 2019 in commit 805e01d.

Later in 2020, glibc added its own handling of AT_SYMLINK_NOFOLLOW to its fchmodat, and it returns EOPNOTSUPP in case of a symlink. This was added by https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=a492b1e5ef7ab50c6fdd4e4e9879ea5569ab0a6c (see also https://sourceware.org/bugzilla/show_bug.cgi?id=14578) and is there since glibc-2.32.

I think musl did a similar change but haven't took a closer look.

edsantiago · 2023-09-26T15:42:29Z

@giuseppe do you have a guess on when this will hit bodhi?

giuseppe · 2023-09-26T16:01:51Z

today! :)

...from f38 + f37. Requires one minor e2e test change, to handle an error logging change in conmon 2.1.8. Also, this is important, requires crun-1.9.1 because of a kernel symlink change; see containers/crun#1309 The VM images here were carefully built to include that. By the time the next VM images get built, it should be default. Signed-off-by: Ed Santiago <santiago@redhat.com>

Ref: containers/crun#1309 Signed-off-by: Chris Evich <cevich@redhat.com>

...from f38 + f37. Requires one minor e2e test change, to handle an error logging change in conmon 2.1.8. Also, this is important, requires crun-1.9.1 because of a kernel symlink change; see containers/crun#1309 The VM images here were carefully built to include that. By the time the next VM images get built, it should be default. Since we've bumped crun, remove two obsolete skips And, skip a flaky pasta test, containers#20170 Signed-off-by: Ed Santiago <santiago@redhat.com>

Ref: containers/crun#1309 Signed-off-by: Chris Evich <cevich@redhat.com>

...from f38 + f37. Requires one minor e2e test change, to handle an error logging change in conmon 2.1.8. Also, this is important, requires crun-1.9.1 because of a kernel symlink change; see containers/crun#1309 The VM images here were carefully built to include that. By the time the next VM images get built, it should be default. Since we've bumped crun, remove two obsolete skips And, skip a flaky pasta test, containers#20170 Signed-off-by: Ed Santiago <santiago@redhat.com>

aki-k · 2023-10-05T07:23:53Z

This issue occurs with kernel 6.5.5 too (podman container with systemd)

adworacz · 2023-10-09T14:46:42Z

This issue is now effecting Debian stable, which updated its kernel to 6.1.0-13 (6.1.55), which includes the backported change, as you can see in this change log.

This is breaking some of my podman containers on my Debian install.

https://metadata.ftp-master.debian.org/changelogs//main/l/linux-signed-amd64/linux-signed-amd64_6.1.55+1_changelog

aki-k · 2023-10-16T03:05:52Z

I received an update crun-1.9.2-1.fc37 (Fedora 37) today and now systemd in a podman container works again.

Kernel 6.5.6-100.fc37.x86_64

christianhorn · 2023-10-28T11:20:11Z

I see this on Fedora39 with all updates as of today, on aarch64 and x86_64.
kernels 6.5.6-300.fc39.x86_64 and 6.5.6-403.asahi.fc39.aarch64+16k
podman and crun versions are same on both systems:

[chris@asahi ~]$ podman --version
podman version 4.7.0
[chris@asahi ~]$ crun --version
crun version 1.9
commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

Reproduces in a newly deployed Fed39 KVM guest on x86_64: I used 'podman build [..]' to build an image (based on Debian Bookworm in my case) and then 'podman run':

[chris@fed39 ~]$ podman run --privileged -p 127.0.0.1:8084:80 --name mastodon -d \
   --security-opt seccomp=unconfined      localhost/mastodon /lib/systemd/systemd
Error: OCI runtime error: crun: chmod `run/shm`: Operation not supported

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 23.10 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Ales Musil <amusil@redhat.com>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 23.10 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Ales Musil <amusil@redhat.com> Signed-off-by: 0-day Robot <robot@bytheb.org>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 23.10 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Ales Musil <amusil@redhat.com>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 23.10 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Ales Musil <amusil@redhat.com> Acked-by: Dumitru Ceara <dceara@redhat.com> Signed-off-by: Numan Siddique <numans@ovn.org>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 23.10 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Ales Musil <amusil@redhat.com> Acked-by: Dumitru Ceara <dceara@redhat.com> Signed-off-by: Numan Siddique <numans@ovn.org> (cherry picked from commit 891cf58)

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com> Signed-off-by: 0-day Robot <robot@bytheb.org>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com> Signed-off-by: 0-day Robot <robot@bytheb.org>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com> Signed-off-by: Dumitru Ceara <dceara@redhat.com>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com> Signed-off-by: Dumitru Ceara <dceara@redhat.com> (cherry picked from commit cbf4fc8)

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 23.10 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Ales Musil <amusil@redhat.com> Acked-by: Dumitru Ceara <dceara@redhat.com> Signed-off-by: Numan Siddique <numans@ovn.org> (cherry picked from commit 891cf58) Signed-off-by: Dumitru Ceara <dceara@redhat.com>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com> Signed-off-by: Dumitru Ceara <dceara@redhat.com> (cherry picked from commit cbf4fc8)

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 23.10 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Ales Musil <amusil@redhat.com> Acked-by: Dumitru Ceara <dceara@redhat.com> Signed-off-by: Numan Siddique <numans@ovn.org> (cherry picked from commit 891cf58) Signed-off-by: Dumitru Ceara <dceara@redhat.com>

There are some issues with older crun and newer kernel combination [0]. Unfortunately Ubuntu 22.04 falls into this category and the jobs will often time fail. Temoprarily use newer crun from git that has the fix [1]. [0] containers/crun#1309 [1] https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ Signed-off-by: Mohammad Heib <mheib@redhat.com> Signed-off-by: Dumitru Ceara <dceara@redhat.com> (cherry picked from commit cbf4fc8)

giuseppe mentioned this pull request Sep 22, 2023

container with systemd don't start #1308

Closed

giuseppe force-pushed the ignore-ENOTSUP-chmod branch from 0b2bf5e to 66d6cd7 Compare September 22, 2023 09:37

giuseppe marked this pull request as draft September 22, 2023 09:41

giuseppe force-pushed the ignore-ENOTSUP-chmod branch from 66d6cd7 to 57262a2 Compare September 22, 2023 09:42

giuseppe marked this pull request as ready for review September 22, 2023 09:42

flouthoc reviewed Sep 22, 2023

View reviewed changes

src/libcrun/utils.c Show resolved Hide resolved

flouthoc approved these changes Sep 22, 2023

View reviewed changes

kolyshkin approved these changes Sep 22, 2023

View reviewed changes

giuseppe merged commit be16ee7 into containers:main Sep 25, 2023
22 of 38 checks passed

edsantiago mentioned this pull request Sep 27, 2023

CI VMs: bump to f39 + f38 containers/podman#20162

Merged

cevich added a commit to cevich/automation_images that referenced this pull request Sep 27, 2023

Update rawhide image for crun-1.9.1-3

7b1039c

Ref: containers/crun#1309 Signed-off-by: Chris Evich <cevich@redhat.com>

cevich mentioned this pull request Sep 27, 2023

Update windows CI VMs for hyper-v machine testing containers/automation_images#309

Merged

cevich added a commit to cevich/automation_images that referenced this pull request Sep 28, 2023

Update rawhide image for crun-1.9.1-3

92503fe

Ref: containers/crun#1309 Signed-off-by: Chris Evich <cevich@redhat.com>

jwillikers mentioned this pull request Nov 12, 2023

Container fails to start after attempting to chmod a symlink when run via Kubernetes YAML containers/podman#20666

Closed

tnqn mentioned this pull request Feb 22, 2024

Kind with podman fails to create cluster on Ubuntu 20.04.6 (kernel 5.15.0-1056-azure and likely more versions) kubernetes-sigs/kind#3526

Closed

mafalb mentioned this pull request Mar 2, 2024

Incompatible kernel breaks crun on github runner mafalb/workflows#55

Closed

ubergeek42 mentioned this pull request Mar 25, 2024

privileged podman ps broken after reboot containers/podman#22159

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

utils: ignore ENOTSUP when chmod a symlink #1309

utils: ignore ENOTSUP when chmod a symlink #1309

giuseppe commented Sep 22, 2023 •

edited

flouthoc left a comment

Luap99 commented Sep 22, 2023

giuseppe commented Sep 22, 2023

brauner commented Sep 22, 2023

kolyshkin left a comment

edsantiago commented Sep 26, 2023

giuseppe commented Sep 26, 2023

aki-k commented Oct 5, 2023

adworacz commented Oct 9, 2023

aki-k commented Oct 16, 2023 •

edited

christianhorn commented Oct 28, 2023 •

edited

utils: ignore ENOTSUP when chmod a symlink #1309

utils: ignore ENOTSUP when chmod a symlink #1309

Conversation

giuseppe commented Sep 22, 2023 • edited

flouthoc left a comment

Choose a reason for hiding this comment

Luap99 commented Sep 22, 2023

giuseppe commented Sep 22, 2023

brauner commented Sep 22, 2023

kolyshkin left a comment

Choose a reason for hiding this comment

edsantiago commented Sep 26, 2023

giuseppe commented Sep 26, 2023

aki-k commented Oct 5, 2023

adworacz commented Oct 9, 2023

aki-k commented Oct 16, 2023 • edited

christianhorn commented Oct 28, 2023 • edited

giuseppe commented Sep 22, 2023 •

edited

aki-k commented Oct 16, 2023 •

edited

christianhorn commented Oct 28, 2023 •

edited