s6-mkfifo: fatal: unable to mkfifo

[amscanne]

@wmuizelaar hrm - thanks for the reply :) @amscanne I'm using nightly gVisor (runsc --version returns runsc version release-20190806.1-329-g1c480abc39b9) and s6-overlay 1.22.0 and 1.22.1 and get this issue with both versions of s6:

s6-mkfifo: fatal: unable to mkfifo /var/run/s6/services/s6-fdholderd/supervise/control: Operation not permitted

I've tried mounting an EmptyDir after looking at the tmpfs stuff at /var/run and get the same issue - Here is a Kubernetes example: https://kubesail.com/template/erulabs/sonarr/1 (Running that on KubeSail reproduces the error - KubeSail uses gVisor under the hood 💃)

Let me know if that's helpful - I'll keep digging on my side. Thanks!

Originally posted by @erulabs in #139 (comment)

[erulabs]

Some additional debugging attempts / info:

Mounting /var as an emptydir (emptyDir: {} and emptyDir.medium: Memory) in an attempt to get a tmpfs under /var/run/s6/services/s6-fdholderd/supervise/control doesn't seem to have any effect.

[fvoznika]

emptyDir.medium: Memory doesn't translate to a tmpfs mount inside gVisor. The annotations added in fc746ef do not work out of the box with Kubernetes and would require adding an admission webhook to make it work. Not sure if this is something that you could do.

/tmp is a tmpfs mount inside the sandbox, you can try to symlink /var/run/s6 => /tmp/s6 and see if it fixes the problem.

[erulabs]

@fvoznika I could probably add an admission webhook if that's required - mounting tmpfs easily without requiring additional steps inside the container is the goal - unfortunately in this situation I don't have a lot of control over the images themselves. Any link to said admission controller / hints?

Edit: Not sure i'm clear on the annotation / webhook relation - the annotation doesn't work without a webhook? Or, are you suggesting I use an admission controller to force the annotation? Sorry if im being a bit thick here :P

[fvoznika]

@erulabs it'll be better to just allow namedpipes to be created on any mount, as you don't really have control over the location where namedpipes can be created for all applications.

Can you check if #1135 fixes the issue for you? I get further running docker run --rm --runtime=runsc -ti kubesail/sonarr. I get error s6-applyuidgid: fatal: unable to exec mono: No such file or directory, but I also get the same error with vanilla docker.

[wmuizelaar]

When trying to setup a local minikube + knative + gvisor setup, with an s6-overlay based container, I ran into this issue as well. Tried to see what #1135 did for me, and the mkfifo-error disappeared, but I got a new one instead:
s6-svwait: fatal: unable to subscribe to events for /var/run/s6/services/php-fpm: Invalid argument

Going to figure out how to enable debug-logging in this setup, and try to get a minimal reproducable setup.

[prattmic]

I only briefly looked at the s6 code, but there are a bunch of uses of MSG_NOSIGNAL (https://github.com/skarnet/skalibs/search?q=sendmsg&unscoped_q=sendmsg), which gVisor doesn't support, which could be the cause of this.

cc @iangudger

[iangudger]

We effectively do support MSG_NOSIGNAL as we always do the MSG_NOSIGNAL behavior. We never send signals in the cases that MSG_NOSIGNAL affects.

[prattmic]

Oops, apologies. I read

gvisor/pkg/sentry/syscalls/linux/sys_socket.go

Line 941 in 9194aab

if flags & ^(linux.MSG_DONTWAIT|linux.MSG_EOR|linux.MSG_MORE|linux.MSG_NOSIGNAL) != 0 {

backwards and thought we were rejecting MSG_NOSIGNAL.