Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/cgo: pthread_create failed: Operation not permitted #1501

Closed
tommywo opened this issue Nov 10, 2021 · 11 comments
Closed

runtime/cgo: pthread_create failed: Operation not permitted #1501

tommywo opened this issue Nov 10, 2021 · 11 comments

Comments

@tommywo
Copy link

tommywo commented Nov 10, 2021

Hello

We started getting seccomp related errors when running

  docker run --rm quay.io/skopeo/stable:latest \
    sync \
    --src docker \
    --dest docker \
    --dest-creds=AWS:$ECR_PASSWORD \
    docker.io/$i \
    $DOCKER_MIRROR_DESTINATION

the errors appeared with 1.5.x, 1.4.1 is working fine.

09-Nov-2021 11:05:59 | runtime/cgo: pthread_create failed: Operation not permitted
09-Nov-2021 11:05:59 | SIGABRT: abort
09-Nov-2021 11:05:59 | PC=0x7fefc876b85c m=0 sigcode=18446744073709551610
09-Nov-2021 11:05:59 |  
09-Nov-2021 11:05:59 | goroutine 0 [idle]:
09-Nov-2021 11:05:59 | runtime: unknown pc 0x7fefc876b85c
09-Nov-2021 11:05:59 | stack: frame={sp:0x7ffd334a3290, fp:0x0} stack=[0x7ffd32aa47f0,0x7ffd334a3830)
09-Nov-2021 11:05:59 | 00007ffd334a3190: 0000000000000000 00007ffd334a31d8
09-Nov-2021 11:05:59 | 00007ffd334a31a0: 00005603509893fa <runtime.mmap.func1+90> 00007fefb3d11000
09-Nov-2021 11:05:59 | 00007ffd334a31b0: 0000000000001000 0000003200000003
09-Nov-2021 11:05:59 | 00007ffd334a31c0: 00000000ffffffff 00007fefb3d11000
09-Nov-2021 11:05:59 | 00007ffd334a31d0: 00007ffd334a3218 00007ffd334a3250
09-Nov-2021 11:05:59 | 00007ffd334a31e0: 000056035092a27e <runtime.mmap+158> 00007ffd334a3220
09-Nov-2021 11:05:59 | 00007ffd334a31f0: 00007ffd334a3240 00005603509404a5 <runtime.sysAlloc+261>
09-Nov-2021 11:05:59 | 00007ffd334a3200: 0000560351d23578 00007ffd334a3230
09-Nov-2021 11:05:59 | 00007ffd334a3210: 00007ffd334a3240 00005603513474a0
09-Nov-2021 11:05:59 | 00007ffd334a3220: 00005603509893a0 <runtime.mmap.func1+0> 00007ffd334a3250
09-Nov-2021 11:05:59 | 00007ffd334a3230: 00007ffd334a3260 00005603513474a0
09-Nov-2021 11:05:59 | 00007ffd334a3240: 00007fefc8766ef0 000000001c000004
09-Nov-2021 11:05:59 | 00007ffd334a3250: 00007fefc871e760 00007fefc8778e59
09-Nov-2021 11:05:59 | 00007ffd334a3260: 0000000000000000 0000000000000014
09-Nov-2021 11:05:59 | 00007ffd334a3270: 0000000000000160 0000000000000000
09-Nov-2021 11:05:59 | 00007ffd334a3280: 0000000000000150 00007fefc876b84e
09-Nov-2021 11:05:59 | 00007ffd334a3290: <0000000000000016 0000000000000400
09-Nov-2021 11:05:59 | 00007ffd334a32a0: 0000000000000000 0000000000000005
09-Nov-2021 11:05:59 | 00007ffd334a32b0: 0000003500000016 00007fefc87ee711
09-Nov-2021 11:05:59 | 00007ffd334a32c0: 00007fefa185f640 00007ffd334a35a0
09-Nov-2021 11:05:59 | 00007ffd334a32d0: 00007ffd334a33fe 00007ffd334a33ff
09-Nov-2021 11:05:59 | 00007ffd334a32e0: 00007fefa185f640 00007fefc8769785
09-Nov-2021 11:05:59 | 00007ffd334a32f0: 00007fefc88d6aa0 0000000000000150
09-Nov-2021 11:05:59 | 00007ffd334a3300: 00000000003d0f00 00007fefa185f910
09-Nov-2021 11:05:59 | 00007ffd334a3310: 00007fefa185f910 bdc3732eea09de00
09-Nov-2021 11:05:59 | 00007ffd334a3320: 00007fefc84f1840 0000000000000006
09-Nov-2021 11:05:59 | 00007ffd334a3330: 000056035149737e 00007ffd334a3660
09-Nov-2021 11:05:59 | 00007ffd334a3340: 0000560351481d86 00007fefc871e6b6
09-Nov-2021 11:05:59 | 00007ffd334a3350: 00007fefc88d8990 00007fefc87087d3
09-Nov-2021 11:05:59 | 00007ffd334a3360: 0000000000000020 00007fefa185f640
09-Nov-2021 11:05:59 | 00007ffd334a3370: 0000000000000000 0000000000000001
09-Nov-2021 11:05:59 | 00007ffd334a3380: 00007fefa185f640 00007fefc876a55c
09-Nov-2021 11:05:59 | runtime: unknown pc 0x7fefc876b85c
09-Nov-2021 11:05:59 | stack: frame={sp:0x7ffd334a3290, fp:0x0} stack=[0x7ffd32aa47f0,0x7ffd334a3830)
09-Nov-2021 11:05:59 | 00007ffd334a3190: 0000000000000000 00007ffd334a31d8
09-Nov-2021 11:05:59 | 00007ffd334a31a0: 00005603509893fa <runtime.mmap.func1+90> 00007fefb3d11000
09-Nov-2021 11:05:59 | 00007ffd334a31b0: 0000000000001000 0000003200000003
09-Nov-2021 11:05:59 | 00007ffd334a31c0: 00000000ffffffff 00007fefb3d11000
09-Nov-2021 11:05:59 | 00007ffd334a31d0: 00007ffd334a3218 00007ffd334a3250
09-Nov-2021 11:05:59 | 00007ffd334a31e0: 000056035092a27e <runtime.mmap+158> 00007ffd334a3220
09-Nov-2021 11:05:59 | 00007ffd334a31f0: 00007ffd334a3240 00005603509404a5 <runtime.sysAlloc+261>
09-Nov-2021 11:05:59 | 00007ffd334a3200: 0000560351d23578 00007ffd334a3230
09-Nov-2021 11:05:59 | 00007ffd334a3210: 00007ffd334a3240 00005603513474a0
09-Nov-2021 11:05:59 | 00007ffd334a3220: 00005603509893a0 <runtime.mmap.func1+0> 00007ffd334a3250
09-Nov-2021 11:05:59 | 00007ffd334a3230: 00007ffd334a3260 00005603513474a0
09-Nov-2021 11:05:59 | 00007ffd334a3240: 00007fefc8766ef0 000000001c000004
09-Nov-2021 11:05:59 | 00007ffd334a3250: 00007fefc871e760 00007fefc8778e59
09-Nov-2021 11:05:59 | 00007ffd334a3260: 0000000000000000 0000000000000014
09-Nov-2021 11:05:59 | 00007ffd334a3270: 0000000000000160 0000000000000000
09-Nov-2021 11:05:59 | 00007ffd334a3280: 0000000000000150 00007fefc876b84e
09-Nov-2021 11:05:59 | 00007ffd334a3290: <0000000000000016 0000000000000400
09-Nov-2021 11:05:59 | 00007ffd334a32a0: 0000000000000000 0000000000000005
09-Nov-2021 11:05:59 | 00007ffd334a32b0: 0000003500000016 00007fefc87ee711
09-Nov-2021 11:05:59 | 00007ffd334a32c0: 00007fefa185f640 00007ffd334a35a0
09-Nov-2021 11:05:59 | 00007ffd334a32d0: 00007ffd334a33fe 00007ffd334a33ff
09-Nov-2021 11:05:59 | 00007ffd334a32e0: 00007fefa185f640 00007fefc8769785
09-Nov-2021 11:05:59 | 00007ffd334a32f0: 00007fefc88d6aa0 0000000000000150
09-Nov-2021 11:05:59 | 00007ffd334a3300: 00000000003d0f00 00007fefa185f910
09-Nov-2021 11:05:59 | 00007ffd334a3310: 00007fefa185f910 bdc3732eea09de00
09-Nov-2021 11:05:59 | 00007ffd334a3320: 00007fefc84f1840 0000000000000006
09-Nov-2021 11:05:59 | 00007ffd334a3330: 000056035149737e 00007ffd334a3660
09-Nov-2021 11:05:59 | 00007ffd334a3340: 0000560351481d86 00007fefc871e6b6
09-Nov-2021 11:05:59 | 00007ffd334a3350: 00007fefc88d8990 00007fefc87087d3
09-Nov-2021 11:05:59 | 00007ffd334a3360: 0000000000000020 00007fefa185f640
09-Nov-2021 11:05:59 | 00007ffd334a3370: 0000000000000000 0000000000000001
09-Nov-2021 11:05:59 | 00007ffd334a3380: 00007fefa185f640 00007fefc876a55c
09-Nov-2021 11:05:59 |  
09-Nov-2021 11:05:59 | goroutine 1 [running]:
09-Nov-2021 11:05:59 | runtime.systemstack_switch()
09-Nov-2021 11:05:59 |  
09-Nov-2021 11:05:59 | runtime.main()
09-Nov-2021 11:05:59 |  
09-Nov-2021 11:05:59 | runtime.goexit()
09-Nov-2021 11:05:59 |  
09-Nov-2021 11:05:59 |  
09-Nov-2021 11:05:59 | rax 0x0
09-Nov-2021 11:05:59 | rbx 0x7fefc84f1840
09-Nov-2021 11:05:59 | rcx 0x7fefc876b85c
09-Nov-2021 11:05:59 | rdx 0x6
09-Nov-2021 11:05:59 | rdi 0x1
09-Nov-2021 11:05:59 | rsi 0x1
09-Nov-2021 11:05:59 | rbp 0x1
09-Nov-2021 11:05:59 | rsp 0x7ffd334a3290
09-Nov-2021 11:05:59 | r8 0x7ffd334a3360
09-Nov-2021 11:05:59 | r9 0x7fefc88924e0
09-Nov-2021 11:05:59 | r10 0x8
09-Nov-2021 11:05:59 | r11 0x246
09-Nov-2021 11:05:59 | r12 0x6
09-Nov-2021 11:05:59 | r13 0x7ffd334a3660
09-Nov-2021 11:05:59 | r14 0x560351481d86
09-Nov-2021 11:05:59 | r15 0x0
09-Nov-2021 11:05:59 | rip 0x7fefc876b85c
09-Nov-2021 11:05:59 | rflags 0x246
09-Nov-2021 11:05:59 | cs 0x33
09-Nov-2021 11:05:59 | fs 0x0
09-Nov-2021 11:05:59 | gs 0x0
09-Nov-2021 11:06:00 | runtime/cgo: pthread_create failed: Operation not permitted
09-Nov-2021 11:06:00 | SIGABRT: abort
09-Nov-2021 11:06:00 | PC=0x7fcebf66b85c m=0 sigcode=18446744073709551610
09-Nov-2021 11:06:00
```
@mtrmac
Copy link
Collaborator

mtrmac commented Nov 11, 2021

Thanks for your report. Is this moby/moby#42680 ?

@RamLavi
Copy link

RamLavi commented Nov 11, 2021

I also see it here
is there a current workaround?

@mtrmac
Copy link
Collaborator

mtrmac commented Nov 11, 2021

Preferably upgrade Docker, or change the seccomp policy, or I guess disable seccomp for these containers.

@RamLavi
Copy link

RamLavi commented Nov 14, 2021

@mtrmac thanks for the reply!
I want to understand the problem better, can you explain why this started happening from (I think) quay.io/skopeo/stable:v1.4.1? what syscall exactly was added? it's hard to follow by comparing the versions..

AFAIK we're using the latest docker for fedora. what version should I use?

[update] I meant quay.io/skopeo/stable:v1.5.*

@rhatdan
Copy link
Member

rhatdan commented Nov 15, 2021

Newer versions of the OS move to newer SYSCALLS, which older versions of Docker do not allow.

@RamLavi
Copy link

RamLavi commented Nov 16, 2021

I see, thank you for the response! :)

RamLavi added a commit to RamLavi/cluster-network-addons-operator that referenced this issue Nov 18, 2021
Recent skopeo releaeses updated the OS and moved to
use more advanced syscalls, ones that are currently
filtered by docker (issue [0]).
In order to move past this issue, moving to Podman.

[0] containers/skopeo#1501

Signed-off-by: Ram Lavi <ralavi@redhat.com>
RamLavi added a commit to RamLavi/project-infra that referenced this issue Nov 29, 2021
Recent skopeo releaeses updated the OS and moved to
use more advanced syscalls, ones that are currently
filtered by docker (issue [0]).
In order to move past this issue, updating to image
using fedora 34.

[0] containers/skopeo#1501

Signed-off-by: Ram Lavi <ralavi@redhat.com>
kubevirt-bot pushed a commit to kubevirt/project-infra that referenced this issue Nov 30, 2021
Recent skopeo releaeses updated the OS and moved to
use more advanced syscalls, ones that are currently
filtered by docker (issue [0]).
In order to move past this issue, updating to image
using fedora 34.

[0] containers/skopeo#1501

Signed-off-by: Ram Lavi <ralavi@redhat.com>
@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@EppO
Copy link

EppO commented May 17, 2022

@rhatdan and @mtrmac I get the same error on CentOS 7 with skopeo 1.7.0 using docker 20.10.16, so it doesn't seem that newer docker versions help here.

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 38
 Server Version: 20.10.16
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.62.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.7GiB
 Name: herecomestrouble
 ID: DUWQ:NYZ6:LQOV:L3DN:DH3Y:J4AJ:RFVL:TGE6:VCEB:5AP2:B5LT:OMXA
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[root@herecomestrouble ~]# docker run -it --rm  quay.io/containers/skopeo:v1.7.0 list-tags docker://quay.io/containers/skopeo
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7fe0b838759c m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7fe0b838759c
stack: frame={sp:0x7ffdba07e350, fp:0x0} stack=[0x7ffdb987f8b0,0x7ffdba07e8f0)
00007ffdba07e250:  0000000000000000  00007ffdba07e298
00007ffdba07e260:  0000557fc5c5085a <runtime.mmap.func1+90>  00007fe0a3919000
00007ffdba07e270:  0000000000001000  0000003200000003
00007ffdba07e280:  00000000ffffffff  00007fe0a3919000
00007ffdba07e290:  00007ffdba07e2d8  00007ffdba07e310
00007ffdba07e2a0:  0000557fc5bf129e <runtime.mmap+158>  00007ffdba07e2e0
00007ffdba07e2b0:  00007ffdba07e300  0000557fc5c074c5 <runtime.sysAlloc+261>
00007ffdba07e2c0:  0000557fc7051e58  0000000000210c10
00007ffdba07e2d0:  0000002200000003  0000557fc66488d0
00007ffdba07e2e0:  0000557fc5c50800 <runtime.mmap.func1+0>  0000557fc5c5dd7e <runtime.callCgoMmap+62>
00007ffdba07e2f0:  00007ffdba07e2f0  0000557fc66488d0
00007ffdba07e300:  00007fe0b8382bc0  000000001c000004
00007ffdba07e310:  00007fe0b833ae30  00007fe0b8394c59
00007ffdba07e320:  0000000000000000  0000000000000014
00007ffdba07e330:  0000000000000160  0000000000000000
00007ffdba07e340:  0000000000000150  00007fe0b838758e
00007ffdba07e350: <0000000000000016  0000000000000100
00007ffdba07e360:  0000000000000000  0000000000000005
00007ffdba07e370:  0000003500000016  00007fe0b8434171
00007ffdba07e380:  00007fe091467640  00007ffdba07e660
00007ffdba07e390:  00007ffdba07e4be  00007ffdba07e4bf
00007ffdba07e3a0:  00007fe091467640  00007fe0b8385485
00007ffdba07e3b0:  00007fe0b84ddc80  0000000000000150
00007ffdba07e3c0:  00000000003d0f00  00007fe091467910
00007ffdba07e3d0:  00007fe091467910  b9fa2ecec86cf700
00007ffdba07e3e0:  00007fe0b80f9840  0000000000000006
00007ffdba07e3f0:  0000557fc679f7de  00007ffdba07e720
00007ffdba07e400:  0000557fc6789d72  00007fe0b833ad86
00007ffdba07e410:  00007fe0b84dee90  00007fe0b830d833
00007ffdba07e420:  0000000000000020  00007fe091467640
00007ffdba07e430:  0000000000000000  0000000000000001
00007ffdba07e440:  00007fe091467640  00007fe0b83862bc
runtime: unknown pc 0x7fe0b838759c
stack: frame={sp:0x7ffdba07e350, fp:0x0} stack=[0x7ffdb987f8b0,0x7ffdba07e8f0)
00007ffdba07e250:  0000000000000000  00007ffdba07e298
00007ffdba07e260:  0000557fc5c5085a <runtime.mmap.func1+90>  00007fe0a3919000
00007ffdba07e270:  0000000000001000  0000003200000003
00007ffdba07e280:  00000000ffffffff  00007fe0a3919000
00007ffdba07e290:  00007ffdba07e2d8  00007ffdba07e310
00007ffdba07e2a0:  0000557fc5bf129e <runtime.mmap+158>  00007ffdba07e2e0
00007ffdba07e2b0:  00007ffdba07e300  0000557fc5c074c5 <runtime.sysAlloc+261>
00007ffdba07e2c0:  0000557fc7051e58  0000000000210c10
00007ffdba07e2d0:  0000002200000003  0000557fc66488d0
00007ffdba07e2e0:  0000557fc5c50800 <runtime.mmap.func1+0>  0000557fc5c5dd7e <runtime.callCgoMmap+62>
00007ffdba07e2f0:  00007ffdba07e2f0  0000557fc66488d0
00007ffdba07e300:  00007fe0b8382bc0  000000001c000004
00007ffdba07e310:  00007fe0b833ae30  00007fe0b8394c59
00007ffdba07e320:  0000000000000000  0000000000000014
00007ffdba07e330:  0000000000000160  0000000000000000
00007ffdba07e340:  0000000000000150  00007fe0b838758e
00007ffdba07e350: <0000000000000016  0000000000000100
00007ffdba07e360:  0000000000000000  0000000000000005
00007ffdba07e370:  0000003500000016  00007fe0b8434171
00007ffdba07e380:  00007fe091467640  00007ffdba07e660
00007ffdba07e390:  00007ffdba07e4be  00007ffdba07e4bf
00007ffdba07e3a0:  00007fe091467640  00007fe0b8385485
00007ffdba07e3b0:  00007fe0b84ddc80  0000000000000150
00007ffdba07e3c0:  00000000003d0f00  00007fe091467910
00007ffdba07e3d0:  00007fe091467910  b9fa2ecec86cf700
00007ffdba07e3e0:  00007fe0b80f9840  0000000000000006
00007ffdba07e3f0:  0000557fc679f7de  00007ffdba07e720
00007ffdba07e400:  0000557fc6789d72  00007fe0b833ad86
00007ffdba07e410:  00007fe0b84dee90  00007fe0b830d833
00007ffdba07e420:  0000000000000020  00007fe091467640
00007ffdba07e430:  0000000000000000  0000000000000001
00007ffdba07e440:  00007fe091467640  00007fe0b83862bc

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/lib/golang/src/runtime/asm_amd64.s:339 fp=0xc00004e788 sp=0xc00004e780 pc=0x557fc5c5a200
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:144 +0x8d fp=0xc00004e7e0 sp=0xc00004e788 pc=0x557fc5c2774d
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00004e7e8 sp=0xc00004e7e0 pc=0x557fc5c5c041

rax    0x0
rbx    0x7fe0b80f9840
rcx    0xffffffffffffffff
rdx    0x6
rdi    0x1
rsi    0x1
rbp    0x1
rsp    0x7ffdba07e350
r8     0x7ffdba07e420
r9     0x7fe0b84954e0
r10    0x8
r11    0x246
r12    0x6
r13    0x7ffdba07e720
r14    0x557fc6789d72
r15    0x0
rip    0x7fe0b838759c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

Using --security-opt seccomp=unconfined works but it's not ideal in a CI runner use case.

@ghost
Copy link

ghost commented Apr 8, 2023

Issue is still present

@mtrmac
Copy link
Collaborator

mtrmac commented Apr 11, 2023

@venomone I’m not aware of anything changing in this area recently, and the over-a-year-old recommendation #1501 (comment) would apply to that old situation.

If that’s not what you see, please file a new report with full details of the environment and the involved versions of all relevant components.

@lzang
Copy link

lzang commented Apr 27, 2023

Hit the same error, and upgrading docker to a newer version such as 20.10.21 worked for me.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants