Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker exec fails with exit code 255 #15713

Closed
pooya opened this issue Aug 20, 2015 · 17 comments
Closed

docker exec fails with exit code 255 #15713

pooya opened this issue Aug 20, 2015 · 17 comments
Labels
kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.

Comments

@pooya
Copy link

pooya commented Aug 20, 2015

Sometimes when I issue docker exec, it fails without printing any error messages and just exits with error code 255.
The image I am using is docker-in-docker (dind) and there are some docker containers running inside this container which is probably relevant.

in docker.log I see:

INFO[0878] POST /v1.19/containers/017ac2609ad4/exec     
INFO[0878] POST /v1.19/exec/1362c8cd73fcde5242b8d48b9f0de75a503ab4ed33088d9e2fec570271d6488a/start 
WARN[0878] os: process already finished                 
ERRO[0878] Error running command in existing container 017ac2609ad48006eb319c72763dc66920b9bd527bf7bb1f207fd9e8fcd85db9: [8] System error: exit status 1 
ERRO[0878] Error starting exec command in container 1362c8cd73fcde5242b8d48b9f0de75a503ab4ed33088d9e2fec570271d6488a: Cannot run exec command 1362c8cd73fcde5242b8d48b9f0de75a503ab4ed33088d9e2fec570271d6488a in container 017ac2609ad48006eb319c72763dc66920b9bd527bf7bb1f207fd9e8fcd85db9: [8] System error: exit status 1 
ERRO[0878] Handler for POST /exec/{name:.*}/start returned error: Cannot run exec command 1362c8cd73fcde5242b8d48b9f0de75a503ab4ed33088d9e2fec570271d6488a in container 017ac2609ad48006eb319c72763dc66920b9bd527bf7bb1f207fd9e8fcd85db9: [8] System error: exit status 1 
ERRO[0878] HTTP Error                                    err=Cannot run exec command 1362c8cd73fcde5242b8d48b9f0de75a503ab4ed33088d9e2fec570271d6488a in container 017ac2609ad48006eb319c72763dc66920b9bd527bf7bb1f207fd9e8fcd85db9: [8] System error: exit status 1 statusCode=500
2015/08/20 00:22:31 http: response.WriteHeader on hijacked connection
2015/08/20 00:22:31 http: response.Write on hijacked connection
INFO[0878] POST /v1.19/exec/1362c8cd73fcde5242b8d48b9f0de75a503ab4ed33088d9e2fec570271d6488a/resize?h=17&w=140 
ERRO[0878] Handler for POST /exec/{name:.*}/resize returned error: bad file descriptor 
ERRO[0878] HTTP Error                                    err=bad file descriptor statusCode=500
INFO[0878] GET /v1.19/exec/1362c8cd73fcde5242b8d48b9f0de75a503ab4ed33088d9e2fec570271d6488a/json 

This happens in the parent container so it is bug in docker, not in dind.

uname -a

$ uname -a
Linux ubuntu2 3.11.0-15-generic #25~precise1-Ubuntu SMP Thu Jan 30 17:39:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

docker version

Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker -D info

Containers: 2
Images: 82
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 88
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.11.0-15-generic
Operating System: Ubuntu precise (12.04.4 LTS)
CPUs: 2
Total Memory: 3.86 GiB
Name: ubuntu2
ID: SI3N:Y2HK:GSGO:O4IW:IT55:TGTY:6WCX:ZJ64:BNEO:KXUW:KNT6:AJFZ
WARNING: No swap limit support

The error code is the same as issue #9680, but this is a newer kernel.

@weivall
Copy link

weivall commented Aug 31, 2015

I have exactly the same issue

@cpuguy83
Copy link
Member

This looks like the container is stopped but only after the exec instance is actually attempted to be started?
Can you confirm?

@thaJeztah
Copy link
Member

closing because OP didn't respond, but I'm happy to reopen after

@hairyhenderson
Copy link
Contributor

FWIW, I'm running into this in RancherOS 0.4.0:

[root@ip-10-1-1-242 ~]# docker exec -it consul-server /bin/bash
[root@ip-10-1-1-242 ~]# echo $?
255
[root@ip-10-1-1-242 ~]# docker info
Containers: 2
Images: 21
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.3-rancher
Operating System: RancherOS (containerized)
CPUs: 1
Total Memory: 1.956 GiB
Name: ip-10-1-1-242.ec2.internal
ID: NHHC:FZ63:MUX4:4YSA:CLBF:HCMN:SBRX:PC22:V3JS:MIKX:O6LL:RFLF
[root@ip-10-1-1-242 ~]# docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   5eab58c
 Built:        Wed Oct 14 13:42:27 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   5eab58c
 Built:        Wed Oct 14 13:42:27 UTC 2015
 OS/Arch:      linux/amd64

@hairyhenderson
Copy link
Contributor

Oh - also, the container is not stopped. It appears to be running just fine. One (perhaps relevant) detail is this particular container was initially run before the last reboot, so it's been automatically restarted (because --restart=always).

@hairyhenderson
Copy link
Contributor

So, for the record, I'm a total moron - there is no /bin/bash in the image I was using... :P

@duglin
Copy link
Contributor

duglin commented Oct 27, 2015

LOL

@aidanhs
Copy link
Contributor

aidanhs commented Nov 3, 2015

I've hit the exact same as the original issue:

INFO[1623040] POST /v1.20/containers/db-daily/exec         
INFO[1623040] POST /v1.20/exec/cd96c13663bf9b7c79927777ba0d03730b211bb4baf76c32ada8bcfe01be88ea/start 
WARN[1623040] os: process already finished                 
ERRO[1623040] Error running command in existing container 7f65e0e82365fc81824159ea3eace0fe9a7d49653e5f8f9f8a250913dcd861a3: [8] System error: exit status 1 
ERRO[1623040] Error starting exec command in container cd96c13663bf9b7c79927777ba0d03730b211bb4baf76c32ada8bcfe01be88ea: Cannot run exec command cd96c13663bf9b7c79927777ba0d03730b211bb4baf76c32ada8bcfe01be88ea in container 7f65e0e82365fc8
1824159ea3eace0fe9a7d49653e5f8f9f8a250913dcd861a3: [8] System error: exit status 1 
ERRO[1623040] Handler for POST /exec/{name:.*}/start returned error: Cannot run exec command cd96c13663bf9b7c79927777ba0d03730b211bb4baf76c32ada8bcfe01be88ea in container 7f65e0e82365fc81824159ea3eace0fe9a7d49653e5f8f9f8a250913dcd861a3: [
8] System error: exit status 1 
ERRO[1623040] HTTP Error                                    err=Cannot run exec command cd96c13663bf9b7c79927777ba0d03730b211bb4baf76c32ada8bcfe01be88ea in container 7f65e0e82365fc81824159ea3eace0fe9a7d49653e5f8f9f8a250913dcd861a3: [8] Sy
stem error: exit status 1 statusCode=500
2015/11/03 11:33:02 http: response.WriteHeader on hijacked connection
2015/11/03 11:33:02 http: response.Write on hijacked connection
INFO[1623040] POST /v1.20/exec/cd96c13663bf9b7c79927777ba0d03730b211bb4baf76c32ada8bcfe01be88ea/resize?h=59&w=119 
ERRO[1623040] Handler for POST /exec/{name:.*}/resize returned error: bad file descriptor 
ERRO[1623040] HTTP Error                                    err=bad file descriptor statusCode=500
INFO[1623040] GET /v1.20/exec/cd96c13663bf9b7c79927777ba0d03730b211bb4baf76c32ada8bcfe01be88ea/json 

Info:

root@docker18:/var/log/upstart# docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:37:18 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:37:18 UTC 2015
 OS/Arch:      linux/amd64
root@docker18:/var/log/upstart# docker info
Containers: 9
Images: 171
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 191
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-50-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 3.692 GiB
Name: docker18
ID: BSNE:WBVH:DHF4:IFL3:V7P6:W4GY:MOGC:5BSN:BATM:ZWOK:F54L:WPNU
WARNING: No swap limit support
Labels:
 team=OBTI

However, this issue may be a symptom, as I think the container was deadlocked at the time - docker kill was hanging, docker attach was hanging and I had to restart the daemon.

@aidanhs
Copy link
Contributor

aidanhs commented Dec 1, 2015

Argh, I've just come across this error again on a different machine, searched for the error and ended up back here. Except now I'm on docker 1.9.1.

ahobsons@docker09:~$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Again, kill is hanging.
@thaJeztah can you reopen?

@aidanhs
Copy link
Contributor

aidanhs commented Dec 1, 2015

Dec  1 09:24:11 docker09 kernel: [13730069.421254] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
Dec  1 09:24:11 docker09 kernel: [13730069.421269] IP: [<ffffffff811bb7d4>] filp_close+0x24/0x70
Dec  1 09:24:11 docker09 kernel: [13730069.421276] PGD 145b68067 PUD 10ba3c067 PMD 0 
Dec  1 09:24:11 docker09 kernel: [13730069.421282] Oops: 0000 [#1] SMP 
Dec  1 09:24:11 docker09 kernel: [13730069.421285] Modules linked in: xt_nat(X) xt_tcpudp(X) veth(X) btrfs(X) raid6_pq(X) xor(X) ufs(X) qnx4(X) hfsplus(X) hfs(X) minix(X) ntfs(X) msdos(X) jfs(X) xfs(X) libcrc32c(X) xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer snd intel_rapl x86_pkg_temp_thermal soundcore intel_powerclamp mei_me mei coretemp kvm_intel dcdbas kvm joydev lpc_ich shpchp serio_raw mac_hid lp parport hid_generic usbhid hid dm_crypt nouveau crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper mxm_wmi cryptd wmi i2c_algo_bit ttm e1000e psmouse ahci drm_kms_helper libahci ptp drm pps_core video
Dec  1 09:24:11 docker09 kernel: [13730069.421381] CPU: 3 PID: 19186 Comm: oninit Tainted: G            X 3.13.0-55-generic #92-Ubuntu
Dec  1 09:24:11 docker09 kernel: [13730069.421384] Hardware name: Dell Inc. OptiPlex 990/06D7TR, BIOS A07 09/10/2011
Dec  1 09:24:11 docker09 kernel: [13730069.421388] task: ffff880100fb9800 ti: ffff88016d4d6000 task.ti: ffff88016d4d6000
Dec  1 09:24:11 docker09 kernel: [13730069.421391] RIP: 0010:[<ffffffff811bb7d4>]  [<ffffffff811bb7d4>] filp_close+0x24/0x70
Dec  1 09:24:11 docker09 kernel: [13730069.421396] RSP: 0018:ffff88016d4d7f20  EFLAGS: 00010246
Dec  1 09:24:11 docker09 kernel: [13730069.421398] RAX: 0000000000000000 RBX: ffff880144ee2000 RCX: ffff88010f87e000
Dec  1 09:24:11 docker09 kernel: [13730069.421401] RDX: 0000000000000100 RSI: ffff880140932800 RDI: ffff880144ee2000
Dec  1 09:24:11 docker09 kernel: [13730069.421404] RBP: ffff88016d4d7f38 R08: 00007fe2e5c9c720 R09: 0000000000000038
Dec  1 09:24:11 docker09 kernel: [13730069.421407] R10: 00007fe2e5a85240 R11: 0000000000000246 R12: ffff880140932800
Dec  1 09:24:11 docker09 kernel: [13730069.421410] R13: 0000000000000000 R14: ffff880144ee2000 R15: 0000000000000000
Dec  1 09:24:11 docker09 kernel: [13730069.421414] FS:  00007fe2e5c9c720(0000) GS:ffff88042dc60000(0000) knlGS:0000000000000000
Dec  1 09:24:11 docker09 kernel: [13730069.421417] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  1 09:24:11 docker09 kernel: [13730069.421420] CR2: 0000000000000060 CR3: 0000000174966000 CR4: 00000000000407e0
Dec  1 09:24:11 docker09 kernel: [13730069.421422] Stack:
Dec  1 09:24:11 docker09 kernel: [13730069.421424]  ffff880140932800 0000000000000100 ffff880140932840 ffff88016d4d7f68
Dec  1 09:24:11 docker09 kernel: [13730069.421430]  ffffffff811dbf7b 0000000000000101 0000000000000000 0000000000000000
Dec  1 09:24:11 docker09 kernel: [13730069.421435]  0000000000000000 ffff88016d4d7f78 ffffffff811bb843 000000008db9e280
Dec  1 09:24:11 docker09 kernel: [13730069.421441] Call Trace:
Dec  1 09:24:11 docker09 kernel: [13730069.421445]  [<ffffffff811dbf7b>] __close_fd+0x7b/0xa0
Dec  1 09:24:11 docker09 kernel: [13730069.421448]  [<ffffffff811bb843>] SyS_close+0x23/0x50
Dec  1 09:24:11 docker09 kernel: [13730069.421453]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f
Dec  1 09:24:11 docker09 kernel: [13730069.421455] Code: 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 55 41 54 53 48 8b 47 38 48 89 fb 48 85 c0 74 46 48 8b 47 28 45 31 ed 49 89 f4 <48> 8b 40 60 48 85 c0 74 05 ff d0 41 89 c5 f6 43 45 40 75 16 4c 
Dec  1 09:24:11 docker09 kernel: [13730069.421506] RIP  [<ffffffff811bb7d4>] filp_close+0x24/0x70
Dec  1 09:24:11 docker09 kernel: [13730069.421510]  RSP <ffff88016d4d7f20>
Dec  1 09:24:11 docker09 kernel: [13730069.421512] CR2: 0000000000000060
Dec  1 09:24:11 docker09 kernel: [13730069.426444] ---[ end trace c2c6d961bd597e36 ]---
Dec  1 09:24:11 docker09 kernel: [13730069.426802] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
Dec  1 09:24:11 docker09 kernel: [13730069.426818] IP: [<ffffffff8120cce7>] do_io_submit+0x427/0x790
Dec  1 09:24:11 docker09 kernel: [13730069.426828] PGD 415d73067 PUD 129310067 PMD 0 
Dec  1 09:24:11 docker09 kernel: [13730069.426838] Oops: 0000 [#2] SMP 
Dec  1 09:24:11 docker09 kernel: [13730069.426845] Modules linked in: xt_nat(X) xt_tcpudp(X) veth(X) btrfs(X) raid6_pq(X) xor(X) ufs(X) qnx4(X) hfsplus(X) hfs(X) minix(X) ntfs(X) msdos(X) jfs(X) xfs(X) libcrc32c(X) xt_addrtype xt_conntrac
k ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd
_page_alloc snd_timer snd intel_rapl x86_pkg_temp_thermal soundcore intel_powerclamp mei_me mei coretemp kvm_intel dcdbas kvm joydev lpc_ich shpchp serio_raw mac_hid lp parport hid_generic usbhid hid dm_crypt nouveau crct10dif_pclmul crc3
2_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper mxm_wmi cryptd wmi i2c_algo_bit ttm e1000e psmouse ahci drm_kms_helper libahci ptp drm pps_core video
Dec  1 09:24:11 docker09 kernel: [13730069.427020] CPU: 0 PID: 8200 Comm: oninit Tainted: G      D     X 3.13.0-55-generic #92-Ubuntu
Dec  1 09:24:11 docker09 kernel: [13730069.427027] Hardware name: Dell Inc. OptiPlex 990/06D7TR, BIOS A07 09/10/2011
Dec  1 09:24:11 docker09 kernel: [13730069.427032] task: ffff880413146000 ti: ffff880141fe4000 task.ti: ffff880141fe4000
Dec  1 09:24:11 docker09 kernel: [13730069.427038] RIP: 0010:[<ffffffff8120cce7>]  [<ffffffff8120cce7>] do_io_submit+0x427/0x790
Dec  1 09:24:11 docker09 kernel: [13730069.427049] RSP: 0018:ffff880141fe5e80  EFLAGS: 00010246
Dec  1 09:24:11 docker09 kernel: [13730069.427055] RAX: 0000000000000000 RBX: 0000000000000800 RCX: 000000008ced24f8
Dec  1 09:24:11 docker09 kernel: [13730069.427060] RDX: 0000000000000000 RSI: 0000000052946800 RDI: ffff880141fe5eb8
Dec  1 09:24:11 docker09 kernel: [13730069.427065] RBP: ffff880141fe5f68 R08: 0000000000015f60 R09: ffff88041116be80
Dec  1 09:24:11 docker09 kernel: [13730069.427071] R10: 0000000000000000 R11: 0000000000000297 R12: ffff880411d5e300
Dec  1 09:24:11 docker09 kernel: [13730069.427076] R13: ffff880144ee2000 R14: ffff88041116be80 R15: 0000000000000000
Dec  1 09:24:11 docker09 kernel: [13730069.427081] FS:  00007fe2e5c9c720(0000) GS:ffff88042dc00000(0000) knlGS:0000000000000000
Dec  1 09:24:11 docker09 kernel: [13730069.427088] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  1 09:24:11 docker09 kernel: [13730069.427093] CR2: 0000000000000020 CR3: 0000000415d72000 CR4: 00000000000407f0
Dec  1 09:24:11 docker09 kernel: [13730069.427099] Stack:
Dec  1 09:24:11 docker09 kernel: [13730069.427103]  ffff8801189f4a00 0000000000000008 ffff880141fe5fd8 000000008d52bdd8
Dec  1 09:24:11 docker09 kernel: [13730069.427124]  0000000000000001 00000000001bfe3e ffff880141fe5eb8 ffff8801189f4a10
Dec  1 09:24:11 docker09 kernel: [13730069.427147]  0000000000000000 0000000091827364 ffff880141fe5ed0 ffff880141fe5ed0
Dec  1 09:24:11 docker09 kernel: [13730069.427175] Call Trace:
Dec  1 09:24:11 docker09 kernel: [13730069.427185]  [<ffffffff8120d060>] SyS_io_submit+0x10/0x20
Dec  1 09:24:11 docker09 kernel: [13730069.427197]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f
Dec  1 09:24:11 docker09 kernel: [13730069.427209] Code: fa 01 0f 84 2c 01 00 00 48 89 da 48 8b b5 48 ff ff ff 4c 89 f7 41 ff d3 49 89 c7 e9 42 fe ff ff 0f 1f 40 00 49 8b 55 28 45 31 d2 <4c> 8b 5a 20 ba 01 00 00 00 e9 1e ff ff ff 0f 1f 00
 f6 05 e3 5d 
Dec  1 09:24:11 docker09 kernel: [13730069.427414] RIP  [<ffffffff8120cce7>] do_io_submit+0x427/0x790
Dec  1 09:24:11 docker09 kernel: [13730069.428617]  RSP <ffff880141fe5e80>
Dec  1 09:24:11 docker09 kernel: [13730069.429878] CR2: 0000000000000020
Dec  1 09:24:11 docker09 kernel: [13730069.430939] ---[ end trace c2c6d961bd597e37 ]---
Dec  1 09:24:11 docker09 kernel: [13730069.441681] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
Dec  1 09:24:11 docker09 kernel: [13730069.442307] IP: [<ffffffff811bb7d4>] filp_close+0x24/0x70
Dec  1 09:24:11 docker09 kernel: [13730069.443414] PGD 0 
Dec  1 09:24:11 docker09 kernel: [13730069.444536] Oops: 0000 [#3] SMP 
Dec  1 09:24:11 docker09 kernel: [13730069.445754] Modules linked in: xt_nat(X) xt_tcpudp(X) veth(X) btrfs(X) raid6_pq(X) xor(X) ufs(X) qnx4(X) hfsplus(X) hfs(X) minix(X) ntfs(X) msdos(X) jfs(X) xfs(X) libcrc32c(X) xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer snd intel_rapl x86_pkg_temp_thermal soundcore intel_powerclamp mei_me mei coretemp kvm_intel dcdbas kvm joydev lpc_ich shpchp serio_raw mac_hid lp parport hid_generic usbhid hid dm_crypt nouveau crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper mxm_wmi cryptd wmi i2c_algo_bit ttm e1000e psmouse ahci drm_kms_helper libahci ptp drm pps_core video
Dec  1 09:24:11 docker09 kernel: [13730069.450823] CPU: 0 PID: 8200 Comm: oninit Tainted: G      D     X 3.13.0-55-generic #92-Ubuntu
Dec  1 09:24:11 docker09 kernel: [13730069.452033] Hardware name: Dell Inc. OptiPlex 990/06D7TR, BIOS A07 09/10/2011
Dec  1 09:24:11 docker09 kernel: [13730069.453314] task: ffff880413146000 ti: ffff880141fe4000 task.ti: ffff880141fe4000
Dec  1 09:24:11 docker09 kernel: [13730069.454606] RIP: 0010:[<ffffffff811bb7d4>]  [<ffffffff811bb7d4>] filp_close+0x24/0x70
Dec  1 09:24:11 docker09 kernel: [13730069.455780] RSP: 0018:ffff880141fe5ac8  EFLAGS: 00010246
Dec  1 09:24:11 docker09 kernel: [13730069.456855] RAX: 0000000000000000 RBX: ffff880144ee2000 RCX: 0000000000000100
Dec  1 09:24:11 docker09 kernel: [13730069.458068] RDX: ffff8800361fe000 RSI: ffff880411d5cf00 RDI: ffff880144ee2000
Dec  1 09:24:11 docker09 kernel: [13730069.459295] RBP: ffff880141fe5ae0 R08: 0000000000000000 R09: 00000000565d62e9
Dec  1 09:24:11 docker09 kernel: [13730069.460549] R10: 000000000f5b9d6e R11: ffffea00051fa980 R12: ffff880411d5cf00
Dec  1 09:24:11 docker09 kernel: [13730069.461809] R13: 0000000000000000 R14: ffff880412ca30c0 R15: 0000000000000100
Dec  1 09:24:11 docker09 kernel: [13730069.462923] FS:  0000000000000000(0000) GS:ffff88042dc00000(0000) knlGS:0000000000000000
Dec  1 09:24:11 docker09 kernel: [13730069.464184] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  1 09:24:11 docker09 kernel: [13730069.465477] CR2: 0000000000000060 CR3: 0000000001c0e000 CR4: 00000000000407f0
Dec  1 09:24:11 docker09 kernel: [13730069.466782] Stack:
Dec  1 09:24:11 docker09 kernel: [13730069.467986]  000000000000000f 0000000000000000 ffff880411d5cf00 ffff880141fe5b20
Dec  1 09:24:11 docker09 kernel: [13730069.469228]  ffffffff811dbbf8 000000050c98be00 ffff880413146000 ffff880411d5cf00
Dec  1 09:24:11 docker09 kernel: [13730069.470482]  ffff880413146650 ffff880413146000 ffff880413146000 ffff880141fe5b48
Dec  1 09:24:11 docker09 kernel: [13730069.471757] Call Trace:
Dec  1 09:24:11 docker09 kernel: [13730069.473047]  [<ffffffff811dbbf8>] put_files_struct+0x88/0xe0
Dec  1 09:24:11 docker09 kernel: [13730069.474344]  [<ffffffff811dbcf7>] exit_files+0x47/0x50
Dec  1 09:24:11 docker09 kernel: [13730069.475396]  [<ffffffff81069d96>] do_exit+0x296/0xa50
Dec  1 09:24:11 docker09 kernel: [13730069.476420]  [<ffffffff8172c7e9>] oops_end+0xa9/0x150
Dec  1 09:24:11 docker09 kernel: [13730069.477523]  [<ffffffff8171bc23>] no_context+0x27e/0x28b
Dec  1 09:24:11 docker09 kernel: [13730069.478747]  [<ffffffff8171bca3>] __bad_area_nosemaphore+0x73/0x1ca
Dec  1 09:24:11 docker09 kernel: [13730069.479972]  [<ffffffff8171bfdc>] bad_area+0x43/0x4a
Dec  1 09:24:11 docker09 kernel: [13730069.481201]  [<ffffffff8172f5a9>] __do_page_fault+0x479/0x570
Dec  1 09:24:11 docker09 kernel: [13730069.482302]  [<ffffffff8123c012>] ? ext4_file_write+0xa2/0x3f0
Dec  1 09:24:11 docker09 kernel: [13730069.483468]  [<ffffffff8114f8f3>] ? unlock_page+0x23/0x30
Dec  1 09:24:11 docker09 kernel: [13730069.484579]  [<ffffffff8117809a>] ? do_wp_page+0x39a/0x7c0
Dec  1 09:24:11 docker09 kernel: [13730069.485750]  [<ffffffff8172f6ba>] do_page_fault+0x1a/0x70
Dec  1 09:24:11 docker09 kernel: [13730069.486894]  [<ffffffff8172bae8>] page_fault+0x28/0x30
Dec  1 09:24:11 docker09 kernel: [13730069.487976]  [<ffffffff8120cce7>] ? do_io_submit+0x427/0x790
Dec  1 09:24:11 docker09 kernel: [13730069.488992]  [<ffffffff8120d060>] SyS_io_submit+0x10/0x20
Dec  1 09:24:11 docker09 kernel: [13730069.489914]  [<ffffffff81733d5d>] system_call_fastpath+0x1a/0x1f
Dec  1 09:24:11 docker09 kernel: [13730069.490931] Code: 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 55 41 54 53 48 8b 47 38 48 89 fb 48 85 c0 74 46 48 8b 47 28 45 31 ed 49 89 f4 <48> 8b 40 60 48 85 c0 74 05 ff d0 41 89 c5 f6 43 45 40 75 16 4c 
Dec  1 09:24:11 docker09 kernel: [13730069.492149] RIP  [<ffffffff811bb7d4>] filp_close+0x24/0x70
Dec  1 09:24:11 docker09 kernel: [13730069.493227]  RSP <ffff880141fe5ac8>
Dec  1 09:24:11 docker09 kernel: [13730069.494243] CR2: 0000000000000060
Dec  1 09:24:11 docker09 kernel: [13730069.495259] ---[ end trace c2c6d961bd597e38 ]---
Dec  1 09:24:11 docker09 kernel: [13730069.496241] Fixing recursive fault but reboot is needed!

Yuck.

Edit: this is 3.13.0-55-generic on Ubuntu 14.04.3 LTS.

@thaJeztah
Copy link
Member

Thanks @aidanhs, yuk indeed. I'll reopen this one

@thaJeztah thaJeztah reopened this Dec 1, 2015
@thaJeztah thaJeztah added this to the 1.9.2 milestone Dec 1, 2015
@thaJeztah thaJeztah added the kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. label Dec 1, 2015
@icecrime icecrime modified the milestones: 1.10, 1.9.2 Dec 3, 2015
@wil93
Copy link

wil93 commented Dec 12, 2015

I just faced this issue and, by digging a bit, I found out that the last command in my Dockerfile was

CMD supervisord

and that command was failing internally with:

Error: The directory named as part of the path /var/log/cms/nginx.log does not exist.

so I changed the path to the correct one (/var/local/log/...) and then rebuilt the image, and now it works

If you want to try and reproduce the issue, you can use the Dockerfile from here. The commit that fixes the issue (and that allows me to use docker exec again) is here.

I don't know how useful this is, I hope it helps 😄

@icecrime icecrime removed this from the 1.10 milestone Dec 29, 2015
@lockwobr
Copy link

I am getting this behavior on one of my containers (Below). I am able to get in at some point but is hard to do some times, just have to keep at it, and once you get it never exit if you can help it.

$ docker exec -it 528291fc58ec  /bin/bash ; echo $? ; date
255
Wed Mar 30 18:16:34 UTC 2016
$ docker exec -it 528291fc58ec  /bin/bash ; echo $? ; date
255
Wed Mar 30 18:16:35 UTC 2016
$ docker exec -it 528291fc58ec  /bin/bash ; echo $? ; date
255
Wed Mar 30 18:16:36 UTC 2016
$ docker exec -it 528291fc58ec  /bin/bash ; echo $? ; date
255
Wed Mar 30 18:16:37 UTC 2016
$ docker exec -it 528291fc58ec  /bin/bash ; echo $? ; date
255
Wed Mar 30 18:16:37 UTC 2016



$ docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 06:06:01 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 06:06:01 UTC 2015
 OS/Arch:      linux/amd64

$ docker info
Containers: 7
Images: 237
Storage Driver: devicemapper
 Pool Name: docker-8:17-1836437-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.46 GB
 Data Space Total: 107.4 GB
 Data Space Available: 93.96 GB
 Metadata Space Used: 13.1 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.134 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /mnt/docker/devicemapper/devicemapper/data
 Metadata loop file: /mnt/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 13.7 GiB
Name: vmmaster11
ID: PALK:7ERH:Q2R2:XACR:K3ET:BGA6:VCUY:7FST:YK7P:3RN6:A4VM:N2NP

$ uname -a
Linux hostname 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

@cpuguy83
Copy link
Member

Pretty sure this was fixed in libcontainer some time ago (1.9-1.10 ish).

@wil93 Not sure how this applies to this issue.

@lockwobr
Copy link

lockwobr commented Apr 1, 2016

@cpuguy83 that seems to have fixed things for me, updated to 1.10 and now I can get in predictably. Thanks!

@cpuguy83
Copy link
Member

cpuguy83 commented Apr 1, 2016

Awesome! Thanks for reporting @lockwobr

@cpuguy83 cpuguy83 closed this as completed Apr 1, 2016
@stefanpandele
Copy link

I made some edits on the docker-compose.yml file to add a new service and my containers don't start anymore. I get a 255 error message.

Is there any way to get to a log file? Not a container log file, since the docker ps output is empty.
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
Projects
None yet
Development

No branches or pull requests