runtime/cgo: pthread_create failed: Operation not permitted #467

michaeljs1990 · 2023-06-15T15:17:05Z

All of a sudden today in CI on the docker 1.20 image I am getting this error. In general the OS hasn't mattered to me since I'm not installing anything in the container however over the past week all of these bookworm upgrades have broken multiple things. This has been an absolute pain. I pinned back to bullseye to fix the issue.

runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f[29](https://gitlab.com/michaeljs1990/-/jobs/4480933541#L29)9d101ccc m=0 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: g 0: unknown pc 0x7f299d101ccc
stack: frame={sp:0x7ffc0e570a[30](https://gitlab.com/michaeljs1990/-/jobs/4480933541#L30), fp:0x0} stack=[0x7ffc0dd71eb0,0x7ffc0e570ec0)
0x00007ffc0e570930:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570940:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570950:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570960:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570970:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570980:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570990:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709a0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709b0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709c0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709d0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709e0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709f0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570a00:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570a10:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570a20:  0x0000000000000000  0x00007f299d101cbe 
0x00007ffc0e570a30: <0x0000000000000000  0x723d2808a3ffd900 
0x00007ffc0e570a40:  0x0000000000000006  0x00007f299d075b80 
0x00007ffc0e570a50:  0x0000000000000001  0x00007ffc0e570d00 
0x00007ffc0e570a60:  0x0000000000e69fa0  0x00007f299d0b2ef2 
0x00007ffc0e570a70:  0x00007f299d24ae70  0x00007f299d09d472 
0x00007ffc0e570a80:  0x0000000000000020  0x0000000000000000 
0x00007ffc0e570a90:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570aa0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ab0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ac0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ad0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ae0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570af0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570b00:  0x00007f299d075b80  0x00007f299d0f480a 
0x00007ffc0e570b10:  0x00007f299d24a840  0x723d2808a3ffd900 
0x00007ffc0e570b20:  0x00007f299d24a840  0x00007f299d24a840 
runtime: g 0: unknown pc 0x7f299d101ccc
stack: frame={sp:0x7ffc0e570a30, fp:0x0} stack=[0x7ffc0dd71eb0,0x7ffc0e570ec0)
0x00007ffc0e570930:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570940:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570950:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570960:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570970:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570980:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570990:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709a0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709b0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709c0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709d0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709e0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e5709f0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570a00:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570a10:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570a20:  0x0000000000000000  0x00007f299d101cbe 
0x00007ffc0e570a30: <0x0000000000000000  0x723d2808a3ffd900 
0x00007ffc0e570a40:  0x0000000000000006  0x00007f299d075b80 
0x00007ffc0e570a50:  0x0000000000000001  0x00007ffc0e570d00 
0x00007ffc0e570a60:  0x0000000000e69fa0  0x00007f299d0b2ef2 
0x00007ffc0e570a70:  0x00007f299d24ae70  0x00007f299d09d472 
0x00007ffc0e570a80:  0x0000000000000020  0x0000000000000000 
0x00007ffc0e570a90:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570aa0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ab0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ac0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ad0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570ae0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570af0:  0x0000000000000000  0x0000000000000000 
0x00007ffc0e570b00:  0x00007f299d075b80  0x00007f299d0f480a 
0x00007ffc0e570b10:  0x00007f299d24a840  0x723d2808a3ffd900 
0x00007ffc0e570b20:  0x00007f299d24a840  0x00007f299d24a840 
goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc000042780 sp=0xc000042778 pc=0x4672c0
runtime.main()
	/usr/local/go/src/runtime/proc.go:170 +0x6d fp=0xc0000427e0 sp=0xc000042780 pc=0x43942d
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000427e8 sp=0xc0000427e0 pc=0x4694e1
rax    0x0
rbx    0x1
rcx    0x7f299d101ccc
rdx    0x6
rdi    0x1
rsi    0x1
rbp    0x7f299d075b80
rsp    0x7ffc0e570a30
r8     0x0
r9     0x73
r10    0x8
r11    0x246
r12    0x6
r13    0x7ffc0e570d00
r14    0xe69fa0
r15    0x1
rip    0x7f299d101ccc
rflags 0x246
cs     0x[33](https://gitlab.com/michaeljs1990/-/jobs/4480933541#L33)
fs     0x0
gs     0x0

The text was updated successfully, but these errors were encountered:

yosifkit · 2023-06-15T17:30:21Z

I'd suggest updating docker and libseccomp on the host. Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

rizkyekoputra · 2023-06-16T04:36:01Z

got the same issue using golang:1.20 since yesterday, some of operation not permitted

#16 0.231 make: mkdir: Operation not permitted
#16 0.231 make: /bin/sh: Operation not permitted
#16 0.232 make: pwd: Operation not permitted
#16 0.232 make: cat: Operation not permitted
#16 0.232 make: date: Operation not permitted
#16 0.232 make: git: Operation not permitted
#16 0.232 make: /bin/sh: Operation not permitted
#16 0.232 make: pwd: Operation not permitted
#16 0.233 make: cat: Operation not permitted
#16 0.233 make: date: Operation not permitted
#16 0.233 make: git: Operation not permitted
#16 0.233 make: /bin/sh: Operation not permitted

jame-developer · 2023-06-16T06:53:04Z

same issue here with golang:1.20.5 but golang:1.20.5-bullseye is working

ffjlabo · 2023-06-19T07:30:09Z

Same issue occured with golang:1.19 👀

Maybe it's because of this fix?
cd60713

tianon · 2023-06-20T21:57:55Z

Yep, definitely seccomp failures:

I'd suggest updating docker and libseccomp on the host. Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

yosifkit · 2023-06-22T00:13:32Z

Root cause: it is Docker with libseccomp; a newer syscall used in Debian Bookworm packages/libs is being blocked.

libseccomp lets you configure allowed syscalls for a process. Docker sets a default seccomp profile for all containers such that only certain syscalls are allowed and everything else is blocked (so, newer syscalls that are not yet known to libseccomp or docker are blocked).

verify that it is libseccomp by running the Bookworm-based image with --security-opt seccomp=unconfined
- use only as a test; it is less secure to keep running them "unconfined"
- learn more: https://docs.docker.com/engine/security/seccomp/
one fix:
- update libseccomp and docker on the host running the containers
one workaround:
- switch to the *bullseye images (in the golang images, these will continue to be maintained/updated until the earliest of the respective Golang end of life or the next Debian release, Debian Trixie)

michaeljs1990 · 2023-06-23T23:09:06Z

I'd suggest updating docker and libseccomp on the host. Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

Sadly we were running a fairly recent OS everywhere except our CI machines. Guess it's time for an upgrade :) thanks for the explanation!

Since this is related to the host machine and not the docker image I'm not sure how relevant this ticket is. I'll leave it open but feel free to close it out since it doesn't seem like their is anything to be done on the docker image side.

wu0407 · 2023-06-28T04:41:40Z

sloved by upgrate to docker 20.10.24

ejain · 2023-07-19T22:17:58Z

Stuck with an older version of Docker, but switching to a "-bullseye" image (as suggested by @yosifkit above) solved the issue for us (for now).

nilaonai · 2023-11-01T07:05:42Z

it really useful wen is upgrade the docker engine version to the newest.
if you can't upgrade docker or other limitation, you can replace the runtime for the same result.

tianon · 2023-12-15T01:03:01Z

#467 (comment)

Had to switch to golang:1.20-bullseye base image and disable CGO in order to get the images to work on the target host when building on linux build hosts. See docker-library/golang#467

yosifkit mentioned this issue Jun 22, 2023

curl not working in golang:1.20.5 #470

Closed

zmarano mentioned this issue Jun 26, 2023

Use bullseye base containers due to incompatibilites with bookworm based containers. GoogleCloudPlatform/guest-test-infra#706

Merged

This was referenced Jul 12, 2023

[Enhancement]: 按照docker方式部署，容器启动失败 matrixorigin/matrixorigin.io.cn#508

Closed

[Bug]: docker install MO 0.8.0 failed matrixorigin/matrixone#10593

Closed

This comment was marked as spam.

Sign in to view

nikita-vanyasin mentioned this issue Aug 7, 2023

[Feature] Bump Go Version to 1.20.7 arangodb-helper/arangodb#376

Merged

jmattheis mentioned this issue Sep 1, 2023

runtime/cgo: pthread_create failed: Operation not permitted gotify/server#590

Closed

12 tasks

RainbowDashy mentioned this issue Oct 31, 2023

docker run with error "pthread_create failed: Operation not permitted" bytebase/bytebase#3096

Closed

This comment was marked as spam.

Sign in to view

phdelodder mentioned this issue Nov 24, 2023

2.24.2 docker image not working in rootless mode on Ubuntu 22.04 host deconz-community/deconz-docker#220

Open

llhuii mentioned this issue Dec 12, 2023

runtime/cgo: pthread_create failed: Operation not permitted llhuii/dive-into-k8s#29

Open

tianon closed this as completed Dec 15, 2023

korniltsev mentioned this issue Dec 15, 2023

golang_pull :The agent cannot be started grafana/pyroscope#2847

Open

willdurand mentioned this issue Jan 9, 2024

"runtime/cgo: pthread_create failed: Operation not permitted" during deploy mozilla-services/go-bouncer#374

Closed

pablomendezroyo mentioned this issue Jan 16, 2024

Add seccomp unconfined dappnode/DNP_IPFS#128

Merged

songkq mentioned this issue Jan 25, 2024

Docker run failed with runtime/cgo: pthread_create failed: Operation not permitted songquanpeng/one-api#966

Open

5 tasks

wangeguo mentioned this issue Jan 26, 2024

Error: runtime/cgo: pthread_create failed: Operation not permitted buildpacks-community/kpack#1500

Closed

jelemux mentioned this issue Jan 29, 2024

Update k3d version and k3s image cloudogu/ces-build-lib#120

Closed

vinayakankugoyal mentioned this issue Feb 6, 2024

Update securityContext values.yaml for kuberay-operator to safe defaults. ray-project/kuberay#1896

Merged

4 tasks

vinayakankugoyal mentioned this issue Feb 21, 2024

[Bug] CI/CD pipelines fail when we apply seccompProfile.type: RuntimeDefault to kuberay-operator container. ray-project/kuberay#1935

Closed

2 tasks

purelind mentioned this issue Feb 26, 2024

fix: update docker daemon verisonn to docker:20.10.24-dind PingCAP-QE/ci#2820

Merged

juan131 mentioned this issue Mar 6, 2024

[bitnami/airflow] error - runtime/cgo: pthread_create failed: Operation not permitted bitnami/containers#63489

Closed

ashiqueps mentioned this issue Apr 2, 2024

Updated the ruby image to overcome the thread error chef/chef-workstation#3220

Merged

11 tasks

oumkale mentioned this issue Apr 12, 2024

Cannot write to /opt/jfrog/router/app/router/router.pid: Permission denied jfrog/charts#1857

Open

bbernhard mentioned this issue Apr 18, 2024

Cannot start container when using version >= 0.82 bbernhard/signal-cli-rest-api#520

Closed

2 tasks

purelind mentioned this issue Apr 22, 2024

chore(tiflash): update dind image version PingCAP-QE/ci#2933

Draft

LeNguyenGiaBao mentioned this issue May 9, 2024

Operation Not Permitted mattermost/docker#128

Open

yosifkit mentioned this issue May 23, 2024

"operation not permitted", a libseccomp story docker-library/official-images#16829

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime/cgo: pthread_create failed: Operation not permitted #467

runtime/cgo: pthread_create failed: Operation not permitted #467

michaeljs1990 commented Jun 15, 2023 •

edited

Loading

yosifkit commented Jun 15, 2023

rizkyekoputra commented Jun 16, 2023

jame-developer commented Jun 16, 2023

ffjlabo commented Jun 19, 2023

tianon commented Jun 20, 2023

yosifkit commented Jun 22, 2023

michaeljs1990 commented Jun 23, 2023 •

edited

Loading

wu0407 commented Jun 28, 2023

ejain commented Jul 19, 2023

This comment was marked as spam.

nilaonai commented Nov 1, 2023

This comment was marked as spam.

tianon commented Dec 15, 2023

runtime/cgo: pthread_create failed: Operation not permitted #467

runtime/cgo: pthread_create failed: Operation not permitted #467

Comments

michaeljs1990 commented Jun 15, 2023 • edited Loading

yosifkit commented Jun 15, 2023

rizkyekoputra commented Jun 16, 2023

jame-developer commented Jun 16, 2023

ffjlabo commented Jun 19, 2023

tianon commented Jun 20, 2023

yosifkit commented Jun 22, 2023

michaeljs1990 commented Jun 23, 2023 • edited Loading

wu0407 commented Jun 28, 2023

ejain commented Jul 19, 2023

This comment was marked as spam.

nilaonai commented Nov 1, 2023

This comment was marked as spam.

tianon commented Dec 15, 2023

michaeljs1990 commented Jun 15, 2023 •

edited

Loading

michaeljs1990 commented Jun 23, 2023 •

edited

Loading