Ensure the seccomp pipe is being read while exporting bpf (regression in rc93) #2871

danail-branekov · 2021-03-24T17:51:03Z

Not reading from the seccomp pipe might get the ExportBRF function stuck
on writing to the seccomp pipe if its buffer is full (i.e. the writer
wrote more than 65K of data). Reading from the read end of the pipe
asynchronously prevents this from happening.

We have seen the issue in real environments when the system is under heavy load. No idea why it does not always happen, but as demonstrated by the unit test, the issue is real.

(Potentially) related issue:
#2530 (comment)
#2865

cc @cloudfoundry/cf-garden

kolyshkin · 2021-03-24T21:10:31Z

libcontainer/seccomp/patchbpf/enosys_linux.go

+	readerBuffer := bytes.NewBuffer([]byte{})
+	errChan := make(chan error)
+	go func() {
+		defer close(errChan)


no need to use defer here, you can just close the channel at the end

kolyshkin · 2021-03-24T21:12:07Z

nit: typo in commit message: ExportBRF

kolyshkin · 2021-03-24T21:14:49Z

glitch in CI; restarted

libcontainer/seccomp/patchbpf/enosys_linux.go

kieron-dev · 2021-03-25T10:56:46Z

Thanks for the comments. I've pushed a change addressing them.

libcontainer/seccomp/patchbpf/enosys_linux_test.go

addressed

kolyshkin · 2021-03-26T18:02:56Z

CI failure in Fedora is a known flake (#2805)
CI failure in CentOS is a relatively new flake (#2861)

Both unrelated; CI restarted.

kolyshkin · 2021-03-26T21:34:21Z

Test failure is a flake (#2756) which just got fixed (so I hope it's the last time I see it). CI restarted.

libcontainer/seccomp/patchbpf/enosys_linux.go

kolyshkin

Need to fix reading goroutine stuck (see above).

kieron-dev · 2021-03-29T08:49:26Z

Need to fix reading goroutine stuck (see above).

Hi @kolyshkin . Thanks for spotting that. Suggested changed pushed.

kolyshkin · 2021-03-29T20:49:26Z

Last commit should be merged into the first one.

Also, can you please explain changing the test case in more details (in its commit message)?

There is a potential deadlock where the ExportBPF method call writes to a pipe but the pipe is not read until after the method call returns. ExportBPF might fill the pipe buffer, in which case it will block waiting for a read on the other side which can't happen until the method returns. Here we concurrently read from the pipe into a buffer to ensure ExportBPF will always return. Co-authored-by: Kieron Browne <kbrowne@vmware.com> Co-authored-by: Danail Branekov <danailster@gmail.com> Signed-off-by: Kieron Browne <kbrowne@vmware.com> Signed-off-by: Danail Branekov <danailster@gmail.com>

TestPatchHugeSeccompFilterDoesNotBlock is only testing the disassembleFilter function. There is no need to invoke PatchAndLoad which has the side effect of loading a seccomp profile. Co-authored-by: Danail Branekov <danailster@gmail.com> Co-authored-by: Kieron Browne <kbrowne@vmware.com> Signed-off-by: Kieron Browne <kbrowne@vmware.com> Signed-off-by: Danail Branekov <danailster@gmail.com>

kieron-dev · 2021-03-30T11:45:49Z

Last commit should be merged into the first one.

Also, can you please explain changing the test case in more details (in its commit message)?

Yep - done and pushed.

cpuguy83 · 2021-03-30T16:46:26Z

This is interesting, so I suppose sometimes the pipe is filling up and sometimes not? Does that seem right? I guess another question here is why are we generating a profile at all for cri?

cpuguy83 · 2021-03-30T16:49:45Z

And why would it only fill up sometimes if the profile is constant?

cpuguy83

This LGTM and seems to fix the issue seen by runc.

kolyshkin

LGTM

kolyshkin · 2021-03-31T21:21:00Z

@AkihiroSuda @mrunalp @cyphar PTAL

cpuguy83 · 2021-04-01T18:42:21Z

I think this is fairly high priority to get into a release. We are seeing reports across several repos that seem to point to the issue being fixed here.

AKS is reverting to use rc92 for the timing being because this is really easy to trigger during scaling. Thankfully our kernel doesn't have CAP_PERFMON yet 😄

deadok22 · 2021-04-09T07:53:16Z

After several days of debugging I found the root cause of some of my apps not starting, which happens to be addressed in this pull request. Thanks! :)

Here's a stack trace of the issue on runc 1.0.0-rc93:

SIGQUIT: quitsyscall, 39 minutes, locked to thread]:
runtime.cgocall(, 0xc000148d00, 0x0)
        runtime/cgocall.go +0x5d0xc000148c980x560223dd5a2d, 0x4f sp= pc=(0x6, 0x0github.com/opencontainers/runc/vendor/github.com/seccomp/libseccomp-golang/seccomp.go930 +0x69 fp=0xc000148d40 sp= pc=0x56022421a4b9
github.com/opencontainers/runc/vendor/github.com/seccomp/libseccomp-golang.(*ScmpFilter).ExportBPF(, 0xc0000102000x0)
        github.com/opencontainers/runc/vendor/github.com/seccomp/libseccomp-golang/seccomp.go930 +0xb5 fp=0xc000148da8 sp= pc=0x560224218565
(0x0, 0x00x00x0)
        : +0xc000148e90 pc=github.com/opencontainers/runc/libcontainer/seccomp/patchbpf.enosysPatchFilter(0xc000065480, 0xc00000e1600x00xc0001b0c400x0)
        :531 +0x42 fp=0xc000148f600xc000148e90 pc=0x560224224002github.com/opencontainers/runc/libcontainer/seccomp/patchbpf.PatchAndLoad0xc000065480, 0xc00000e160, 0x0, 0x0)
        :603 + fp=0xc000148fc8 sp=0xc000148f60 pc=0x5602242247ab
github.com/opencontainers/runc/libcontainer/seccomp.InitSeccomp(0xc000065480, , 0x0)
        :77 + fp=0xc000149048 sp= pc=0x5602242251e6
github.com/opencontainers/runc/libcontainer.(*linuxStandardInit).Init(0xc00011ae70, 0x00x0)
         + fp=0xc000149300 pc=
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization0x0, 0x0)
        github.com/opencontainers/runc/libcontainer/factory_linux.go:396 +0x392 fp=0xc0001494e8 sp=0xc000149300 pc=0x56022423bb42, 0x0 +0x4d fp=0xc000149540 sp=0xc0001494e8 pc=0x5602242be31d(0xc0000acb00, 0xc0000acb00, 0x0)
        github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:5230xc0 fp= pc=0x5602242796c00x5602242c7e270x40x00x00x0, 0x00x5602242e7b52, , 0x0, ...       github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go: +0x51e fp=0xc000149890 sp=0xc000149568 pc=0x56022427a42e
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc00009e8c0, 0xc00000e080, 0x2, 0x2, 0x00x0)
        github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:276 +0x71a fp=0xc000149890 pc=0x56022427776a
main.main()
        github.com/opencontainers/runc/main.go160 +0xbba sp=0xc000149bd80x5602242b3b3aruntime.mainruntime/proc.go0x20e fp=0xc000149fe0 sp=0xc000149f60 pc=0x560223e026ae
()
         +0x1 fp=0xc000149fe8 sp= pc=0x560223e2e791

goroutine 7 [syscall, 39 minutes]:
(0x0)
runtime/sigqueue.go
os/signal.loop(: +
os/signal/signal_unix.go:0x43
0x1000rbx    0x56022516bab00x7f39a247e6fd
0x6
0x560225215de0
rbp    0x0
rsp    0x7fffe3ae2350
r9     
r10    0x7fffe3ae1760
r11    0x293
r12    
r13    
r14    
r15    0x0
0x7f39a247e6fd

fs     0x00x0

deadok22 · 2021-04-14T06:31:56Z

if its buffer is full (i.e. the writer wrote more than 65K of data)

In my case the blocked writer only attempted to write just under 8KB of data. When the soft limit on the number of pages used for pipe buffers is reached (16K pages) and the process doesn't have either CAP_SYS_RESOURCE or CAP_SYS_ADMIN, the kernel uses the default pipe buffer size of just 1 page - see this code

danail-branekov · 2021-04-14T08:29:35Z

^ This probably explains why we were only seeing the issue when the system is under load (apparently IO load)

This change adds patches from opencontainers/runc#2871 to 1.0.8.

This change adds patches from [0] to mitigate an issue customers were seeing where pods could not be scheduled because allocated IP addresses were not being released. [0]- opencontainers/runc#2871

rishikeshdevsot · 2021-11-24T04:16:34Z

After several days of debugging I found the root cause of some of my apps not starting, which happens to be addressed in this pull request. Thanks! :)

Here's a stack trace of the issue on runc 1.0.0-rc93:

SIGQUIT: quitsyscall, 39 minutes, locked to thread]:
runtime.cgocall(, 0xc000148d00, 0x0)
        runtime/cgocall.go +0x5d0xc000148c980x560223dd5a2d, 0x4f sp= pc=(0x6, 0x0github.com/opencontainers/runc/vendor/github.com/seccomp/libseccomp-golang/seccomp.go930 +0x69 fp=0xc000148d40 sp= pc=0x56022421a4b9
github.com/opencontainers/runc/vendor/github.com/seccomp/libseccomp-golang.(*ScmpFilter).ExportBPF(, 0xc0000102000x0)
        github.com/opencontainers/runc/vendor/github.com/seccomp/libseccomp-golang/seccomp.go930 +0xb5 fp=0xc000148da8 sp= pc=0x560224218565
(0x0, 0x00x00x0)
        : +0xc000148e90 pc=github.com/opencontainers/runc/libcontainer/seccomp/patchbpf.enosysPatchFilter(0xc000065480, 0xc00000e1600x00xc0001b0c400x0)
        :531 +0x42 fp=0xc000148f600xc000148e90 pc=0x560224224002github.com/opencontainers/runc/libcontainer/seccomp/patchbpf.PatchAndLoad0xc000065480, 0xc00000e160, 0x0, 0x0)
        :603 + fp=0xc000148fc8 sp=0xc000148f60 pc=0x5602242247ab
github.com/opencontainers/runc/libcontainer/seccomp.InitSeccomp(0xc000065480, , 0x0)
        :77 + fp=0xc000149048 sp= pc=0x5602242251e6
github.com/opencontainers/runc/libcontainer.(*linuxStandardInit).Init(0xc00011ae70, 0x00x0)
         + fp=0xc000149300 pc=
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization0x0, 0x0)
        github.com/opencontainers/runc/libcontainer/factory_linux.go:396 +0x392 fp=0xc0001494e8 sp=0xc000149300 pc=0x56022423bb42, 0x0 +0x4d fp=0xc000149540 sp=0xc0001494e8 pc=0x5602242be31d(0xc0000acb00, 0xc0000acb00, 0x0)
        github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:5230xc0 fp= pc=0x5602242796c00x5602242c7e270x40x00x00x0, 0x00x5602242e7b52, , 0x0, ...       github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go: +0x51e fp=0xc000149890 sp=0xc000149568 pc=0x56022427a42e
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc00009e8c0, 0xc00000e080, 0x2, 0x2, 0x00x0)
        github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:276 +0x71a fp=0xc000149890 pc=0x56022427776a
main.main()
        github.com/opencontainers/runc/main.go160 +0xbba sp=0xc000149bd80x5602242b3b3aruntime.mainruntime/proc.go0x20e fp=0xc000149fe0 sp=0xc000149f60 pc=0x560223e026ae
()
         +0x1 fp=0xc000149fe8 sp= pc=0x560223e2e791

goroutine 7 [syscall, 39 minutes]:
(0x0)
runtime/sigqueue.go
os/signal.loop(: +
os/signal/signal_unix.go:0x43
0x1000rbx    0x56022516bab00x7f39a247e6fd
0x6
0x560225215de0
rbp    0x0
rsp    0x7fffe3ae2350
r9     
r10    0x7fffe3ae1760
r11    0x293
r12    
r13    
r14    
r15    0x0
0x7f39a247e6fd

fs     0x00x0

@deadok22 Could you tell how you obtained this stack trace for runc?

deadok22 · 2021-11-24T05:00:50Z

@rishikeshdevsot I don't have this mentioned in my investigation notes, unfortunately. I must have got it from either strace output or from something reading runc's stdout/stderr. That was one deep rabbit hole :)

kolyshkin reviewed Mar 24, 2021

View reviewed changes

kolyshkin requested a review from cyphar March 24, 2021 21:16

kolyshkin previously requested changes Mar 24, 2021

View reviewed changes

libcontainer/seccomp/patchbpf/enosys_linux.go Outdated Show resolved Hide resolved

kolyshkin added this to the 1.0.0-rc94 milestone Mar 24, 2021

kieron-dev force-pushed the pr-seccomp-pipe-hang branch from c115d5d to 2bd0fb6 Compare March 25, 2021 10:53

cyphar reviewed Mar 25, 2021

View reviewed changes

libcontainer/seccomp/patchbpf/enosys_linux_test.go Outdated Show resolved Hide resolved

cyphar requested a review from kolyshkin March 25, 2021 12:23

kieron-dev force-pushed the pr-seccomp-pipe-hang branch from 16f9b68 to ca7b78a Compare March 25, 2021 14:08

kolyshkin added the area/seccomp label Mar 26, 2021

kolyshkin reviewed Mar 26, 2021

View reviewed changes

libcontainer/seccomp/patchbpf/enosys_linux.go Show resolved Hide resolved

kolyshkin requested changes Mar 26, 2021

View reviewed changes

kieron-dev force-pushed the pr-seccomp-pipe-hang branch 3 times, most recently from ee1fc29 to 700febc Compare March 27, 2021 11:21

cyphar mentioned this pull request Mar 30, 2021

loading seccomp filter: invalid argument #2865

Closed

danail-branekov and others added 2 commits March 30, 2021 12:29

danail-branekov force-pushed the pr-seccomp-pipe-hang branch from 700febc to 08b5279 Compare March 30, 2021 09:37

cpuguy83 approved these changes Mar 30, 2021

View reviewed changes

kolyshkin approved these changes Mar 31, 2021

View reviewed changes

mrunalp approved these changes Apr 1, 2021

View reviewed changes

mrunalp merged commit 8c4dca8 into opencontainers:master Apr 1, 2021

kolyshkin added the impact/changelog label Apr 1, 2021

This was referenced Apr 2, 2021

runc init blocked opening exec.fifo #2828

Closed

rc94 discussion (mid-April 2021?) #2790

Closed

divinerapier mentioned this pull request Apr 2, 2021

using bytes.Buffer cross goroutines #2884

Closed

zmrow added a commit to zmrow/bottlerocket that referenced this pull request Apr 23, 2021

TEST: Add runc patch to 1.0.8

3eba03b

This change adds patches from opencontainers/runc#2871 to 1.0.8.

champtar mentioned this pull request Apr 29, 2021

PSA: runc 1.0.0-rc93 / containerd 1.4.4 bug kubernetes-sigs/kubespray#7574

Closed

jakommo mentioned this pull request Apr 30, 2021

Pin containerd version to <=1.4.3-1 to avoid containers getting stuck in "Created" state elastic/ansible-elastic-cloud-enterprise#121

Closed

kolyshkin changed the title ~~Ensure the seccomp pipe is being read while exporting bpf~~ Ensure the seccomp pipe is being read while exporting bpf (regression in rc93) May 3, 2021

ionutleca mentioned this pull request May 11, 2021

Pods stuck in ContainerCreating state rancher/rke2#973

Closed

dresnick-sf mentioned this pull request May 19, 2021

docker hang with container in created status moby/moby#38064

Open

DaneEveritt mentioned this pull request May 24, 2021

Another power action is being processed: bug in runc causing container lockup pterodactyl/panel#2475

Closed

cyphar mentioned this pull request May 30, 2021

runc init hang #2530

Closed

This was referenced Jun 4, 2021

Fix cleanup context of teardownPodNetwork containerd/containerd#5569

Merged

Pods stuck in RunContainerError status with error: no IP addresses available antrea-io/antrea#2244

Closed

zq-david-wang mentioned this pull request Dec 7, 2021

docker daemon hang for futex moby/moby#42921

Open

jakommo mentioned this pull request Oct 27, 2022

Update CentoOS/RHEL containerd version to 1.5 to avoid docker "hang" issues elastic/ansible-elastic-cloud-enterprise#167

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure the seccomp pipe is being read while exporting bpf (regression in rc93) #2871

Ensure the seccomp pipe is being read while exporting bpf (regression in rc93) #2871

danail-branekov commented Mar 24, 2021

kolyshkin Mar 24, 2021

kolyshkin commented Mar 24, 2021

kolyshkin commented Mar 24, 2021

kieron-dev commented Mar 25, 2021

kolyshkin commented Mar 26, 2021

kolyshkin commented Mar 26, 2021

kolyshkin left a comment

kieron-dev commented Mar 29, 2021

kolyshkin commented Mar 29, 2021

kieron-dev commented Mar 30, 2021

cpuguy83 commented Mar 30, 2021

cpuguy83 commented Mar 30, 2021

cpuguy83 left a comment

kolyshkin left a comment

kolyshkin commented Mar 31, 2021

cpuguy83 commented Apr 1, 2021

deadok22 commented Apr 9, 2021 •

edited

Loading

deadok22 commented Apr 14, 2021 •

edited

Loading

danail-branekov commented Apr 14, 2021

rishikeshdevsot commented Nov 24, 2021

deadok22 commented Nov 24, 2021

Ensure the seccomp pipe is being read while exporting bpf (regression in rc93) #2871

Ensure the seccomp pipe is being read while exporting bpf (regression in rc93) #2871

Conversation

danail-branekov commented Mar 24, 2021

kolyshkin Mar 24, 2021

Choose a reason for hiding this comment

kolyshkin commented Mar 24, 2021

kolyshkin commented Mar 24, 2021

kieron-dev commented Mar 25, 2021

kolyshkin commented Mar 26, 2021

kolyshkin commented Mar 26, 2021

kolyshkin left a comment

Choose a reason for hiding this comment

kieron-dev commented Mar 29, 2021

kolyshkin commented Mar 29, 2021

kieron-dev commented Mar 30, 2021

cpuguy83 commented Mar 30, 2021

cpuguy83 commented Mar 30, 2021

cpuguy83 left a comment

Choose a reason for hiding this comment

kolyshkin left a comment

Choose a reason for hiding this comment

kolyshkin commented Mar 31, 2021

cpuguy83 commented Apr 1, 2021

deadok22 commented Apr 9, 2021 • edited Loading

deadok22 commented Apr 14, 2021 • edited Loading

danail-branekov commented Apr 14, 2021

rishikeshdevsot commented Nov 24, 2021

deadok22 commented Nov 24, 2021

deadok22 commented Apr 9, 2021 •

edited

Loading

deadok22 commented Apr 14, 2021 •

edited

Loading