New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"shim error: invalid argument" when trying to run a container after upgrading to v1.11 #22037

Closed
x3ro opened this Issue Apr 14, 2016 · 24 comments

Comments

Projects
None yet
@x3ro

x3ro commented Apr 14, 2016

Output of docker version:

Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:17:17 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:17:17 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 1
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 12
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.2.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 17.68 GiB
Name: d1
ID: DRUU:7R2I:CVOG:RAS5:UI73:ZI54:DARD:YUU7:GII2:JBMT:ANCS:G4VR
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Steps to reproduce the issue:

  1. Upgrade v1.10 to v1.11 on Debian Jessie (Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.73-2+deb7u2 x86_64 GNU/Linux).
  2. Execute docker run hello-world

Describe the results you received:

$ docker -D run hello-world
DEBU[0000] Corrupted prefix: []
DEBU[0000] [hijack] End of stdout
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".

daemon.log:

Apr 14 14:55:28 d1 systemd-udevd[4008]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4010]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4018]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4016]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4019]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4020]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 docker[3196]: time="2016-04-14T14:55:28+02:00" level=error msg="containerd: start container" error="shim error: invalid argument" id=9c17448db69fc62c68dc09fb6c52437a2faa8d01ba8c6018ee17dbe24734a303
Apr 14 14:55:28 d1 systemd-udevd[4033]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4034]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4036]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4037]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4040]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:28 d1 systemd-udevd[4042]: failed to execute '/lib/udev/socket:@/org/freedesktop/hal/udev_event' 'socket:@/org/freedesktop/hal/udev_event': No such file or directory
Apr 14 14:55:29 d1 docker[3196]: time="2016-04-14T14:55:29.095680354+02:00" level=error msg="Handler for POST /v1.23/containers/9c17448db69fc62c68dc09fb6c52437a2faa8d01ba8c6018ee17dbe24734a303/start returned error: rpc error: code = 2 desc = \"shim error: invalid argument\""
Apr 14 14:55:37 d1 ntpd_intres[1893]: ntp_intres.request: permission denied
Apr 14 14:55:37 d1 ntpd_intres[1893]: ntp_intres.request: permission denied

Describe the results you expected:
The hello-world text is displayed.

Additional info:
I'm assuming this is not a general Debian Jessie issue, but purging/reinstalling Docker has not helped. Building an image fails with the same message after pulling the layers.

@x3ro x3ro changed the title from "shim error: invalid argument" when trying to run a container after upgrading to v11 to "shim error: invalid argument" when trying to run a container after upgrading to v1.11 Apr 14, 2016

@stw

This comment has been minimized.

Show comment
Hide comment
@stw

stw Apr 14, 2016

+1 same issue on debian jessie upgrade, let me know if you figure anything out. I'll do the same.

stw commented Apr 14, 2016

+1 same issue on debian jessie upgrade, let me know if you figure anything out. I'll do the same.

@rustamagasanov

This comment has been minimized.

Show comment
Hide comment
@rustamagasanov

rustamagasanov Apr 14, 2016

same here. the error started to appear when I try to start official Postgresql container.

rustamagasanov commented Apr 14, 2016

same here. the error started to appear when I try to start official Postgresql container.

@thaJeztah thaJeztah added this to the 1.11.1 milestone Apr 14, 2016

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 14, 2016

Member

Are you all running on a 3.2 kernel? Because the minimum kernel version for Docker to run is 3.10

Member

thaJeztah commented Apr 14, 2016

Are you all running on a 3.2 kernel? Because the minimum kernel version for Docker to run is 3.10

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 14, 2016

Member

(not sure if it's related to this issue)

Member

thaJeztah commented Apr 14, 2016

(not sure if it's related to this issue)

@rustamagasanov

This comment has been minimized.

Show comment
Hide comment
@rustamagasanov

rustamagasanov Apr 14, 2016

@thaJeztah the kernel version is 3.13.0-79-generic

rustamagasanov commented Apr 14, 2016

@thaJeztah the kernel version is 3.13.0-79-generic

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 14, 2016

Member

@rustamagasanov that's on debian jessie, or ubuntu? (3.13.0-79 is the default for Ubuntu 14.04 IIRC)

Member

thaJeztah commented Apr 14, 2016

@rustamagasanov that's on debian jessie, or ubuntu? (3.13.0-79 is the default for Ubuntu 14.04 IIRC)

@thaJeztah thaJeztah added the kind/bug label Apr 14, 2016

@rustamagasanov

This comment has been minimized.

Show comment
Hide comment
@rustamagasanov

rustamagasanov Apr 14, 2016

@thaJeztah it's the Debian Jessie, reproducible on this image: https://hub.docker.com/_/postgres/

rustamagasanov commented Apr 14, 2016

@thaJeztah it's the Debian Jessie, reproducible on this image: https://hub.docker.com/_/postgres/

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 14, 2016

Member

@rustamagasanov trying to reproduce; I did a fresh install on Debian Jessie (but kernel 3.16.0-4-amd64), but no issues. Also tried running the postgres image; not able to reproduce so far 😢

Member

thaJeztah commented Apr 14, 2016

@rustamagasanov trying to reproduce; I did a fresh install on Debian Jessie (but kernel 3.16.0-4-amd64), but no issues. Also tried running the postgres image; not able to reproduce so far 😢

@stw

This comment has been minimized.

Show comment
Hide comment
@stw

stw Apr 14, 2016

I was actually on wheezy on the machine w/ the issue, thought it was jessie, upgrading to jessie to see if that resolves it. Kernel was 3.2.0-4-amd64, so most likely my issue. Thanks @thaJeztah

stw commented Apr 14, 2016

I was actually on wheezy on the machine w/ the issue, thought it was jessie, upgrading to jessie to see if that resolves it. Kernel was 3.2.0-4-amd64, so most likely my issue. Thanks @thaJeztah

@jeek120

This comment has been minimized.

Show comment
Hide comment
@jeek120

jeek120 Apr 14, 2016

me too,
14.04.4 LTS
Linux MyServer 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

run command:docker run hello-world
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".

jeek120 commented Apr 14, 2016

me too,
14.04.4 LTS
Linux MyServer 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

run command:docker run hello-world
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".

@rustamagasanov

This comment has been minimized.

Show comment
Hide comment
@rustamagasanov

rustamagasanov Apr 14, 2016

@thaJeztah sorry, seems like I use 9.5 and you checked the latest version, I will try it later and tell the results

rustamagasanov commented Apr 14, 2016

@thaJeztah sorry, seems like I use 9.5 and you checked the latest version, I will try it later and tell the results

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Apr 14, 2016

Contributor

@jeek120 Docker is not supported on the 3.2 kernel...

Contributor

cpuguy83 commented Apr 14, 2016

@jeek120 Docker is not supported on the 3.2 kernel...

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Apr 14, 2016

Contributor

So far it seems all impacted people were in a 3.2 kernel except @rustamagasanov: please tell us when you know more!

Contributor

icecrime commented Apr 14, 2016

So far it seems all impacted people were in a 3.2 kernel except @rustamagasanov: please tell us when you know more!

@rustamagasanov

This comment has been minimized.

Show comment
Hide comment
@rustamagasanov

rustamagasanov Apr 14, 2016

@thaJeztah you are right, I installed from scratch and can't reproduce it either for some reason.
@icecrime I tested on Ubuntu boxes now.

Both are:

$ uname -a
Linux movinga 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

In one box:

$ docker -v
Docker version 1.10.3, build 20f81dd
$ docker run postgres:9.5
...everything is fine

In the other box:

$ docker -v
Docker version 1.11.0, build 4dc5990
$ docker run postgres:9.5
...pulling
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".

So, starting from 1.11, you dropped support for the old kernel versions?

rustamagasanov commented Apr 14, 2016

@thaJeztah you are right, I installed from scratch and can't reproduce it either for some reason.
@icecrime I tested on Ubuntu boxes now.

Both are:

$ uname -a
Linux movinga 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

In one box:

$ docker -v
Docker version 1.10.3, build 20f81dd
$ docker run postgres:9.5
...everything is fine

In the other box:

$ docker -v
Docker version 1.11.0, build 4dc5990
$ docker run postgres:9.5
...pulling
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".

So, starting from 1.11, you dropped support for the old kernel versions?

@pikespeak

This comment has been minimized.

Show comment
Hide comment
@pikespeak

pikespeak Apr 14, 2016

Linux malta1356 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

docker -D run hello-world
DEBU[0000] Corrupted prefix: []
DEBU[0000] [hijack] End of stdout
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".

pikespeak commented Apr 14, 2016

Linux malta1356 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

docker -D run hello-world
DEBU[0000] Corrupted prefix: []
DEBU[0000] [hijack] End of stdout
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Apr 14, 2016

Contributor

@rustamagasanov Thanks for the new info!

We didn't drop support for 3.2 as this kernel was never supported: it just happens that 1.11 is the first "obviously" incompatible release. @crosbymichael confirmed that this is due to containerd-shim usage of PR_SET_CHILD_SUBREAPER (introduced in 3.4).

Thank you all for reporting, we can only recommend to upgrade to a newer kernel.

Contributor

icecrime commented Apr 14, 2016

@rustamagasanov Thanks for the new info!

We didn't drop support for 3.2 as this kernel was never supported: it just happens that 1.11 is the first "obviously" incompatible release. @crosbymichael confirmed that this is due to containerd-shim usage of PR_SET_CHILD_SUBREAPER (introduced in 3.4).

Thank you all for reporting, we can only recommend to upgrade to a newer kernel.

@icecrime icecrime closed this Apr 14, 2016

@MiquelMarino

This comment has been minimized.

Show comment
Hide comment
@MiquelMarino

MiquelMarino Apr 14, 2016

Same here!
Vagrant + ansible + docker

config.vm.box = "chef/debian-7.8”
uname -a
Linux badges-io-dev 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 x86_64 GNU/Linux
msg: Docker API Error: rpc error: code = 2 desc = "shim error: invalid argument"
Error!

config.vm.box = "debian/jessie64”
uname -a
Linux badges-io-dev 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
It works!

sudo docker version
Client:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 18:17:17 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 18:17:17 2016
OS/Arch: linux/amd64

Thanks!

MiquelMarino commented Apr 14, 2016

Same here!
Vagrant + ansible + docker

config.vm.box = "chef/debian-7.8”
uname -a
Linux badges-io-dev 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 x86_64 GNU/Linux
msg: Docker API Error: rpc error: code = 2 desc = "shim error: invalid argument"
Error!

config.vm.box = "debian/jessie64”
uname -a
Linux badges-io-dev 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
It works!

sudo docker version
Client:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 18:17:17 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 18:17:17 2016
OS/Arch: linux/amd64

Thanks!

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Apr 14, 2016

Contributor

@MiquelMarino Thanks for the report.
As stated above, 3.2 is not a supported kernel (nor has it ever been).
The minimum support kernel is 3.10.

Contributor

cpuguy83 commented Apr 14, 2016

@MiquelMarino Thanks for the report.
As stated above, 3.2 is not a supported kernel (nor has it ever been).
The minimum support kernel is 3.10.

@MiquelMarino

This comment has been minimized.

Show comment
Hide comment
@MiquelMarino

MiquelMarino Apr 14, 2016

ok!
Thanks for the information.

MiquelMarino commented Apr 14, 2016

ok!
Thanks for the information.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Apr 14, 2016

Member

@MiquelMarino great avatar! 👍

Member

thaJeztah commented Apr 14, 2016

@MiquelMarino great avatar! 👍

@gutsal-arsen

This comment has been minimized.

Show comment
Hide comment
@gutsal-arsen

gutsal-arsen Jun 29, 2016

+1

archer@capitan ~ $ docker -D run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
a9d36faac0fe: Pull complete 
Digest: sha256:e52be8ffeeb1f374f440893189cd32f44cb166650e7ab185fa7735b7dc48d619
Status: Downloaded newer image for hello-world:latest
DEBU[0007] Corrupted prefix: []                         
DEBU[0007] [hijack] End of stdout                       
docker: Error response from daemon: docker-containerd-shim not installed on system.

archer@capitan ~ $ uname -a
Linux capitan 4.6.2-gentoo #3 SMP Thu Jun 16 15:31:10 EEST 2016 x86_64 Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz GenuineIntel GNU/Linux

archer@capitan ~ $ docker -v
Docker version 1.12.0-rc2, build 906eacd

gutsal-arsen commented Jun 29, 2016

+1

archer@capitan ~ $ docker -D run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
a9d36faac0fe: Pull complete 
Digest: sha256:e52be8ffeeb1f374f440893189cd32f44cb166650e7ab185fa7735b7dc48d619
Status: Downloaded newer image for hello-world:latest
DEBU[0007] Corrupted prefix: []                         
DEBU[0007] [hijack] End of stdout                       
docker: Error response from daemon: docker-containerd-shim not installed on system.

archer@capitan ~ $ uname -a
Linux capitan 4.6.2-gentoo #3 SMP Thu Jun 16 15:31:10 EEST 2016 x86_64 Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz GenuineIntel GNU/Linux

archer@capitan ~ $ docker -v
Docker version 1.12.0-rc2, build 906eacd

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 29, 2016

Contributor

@gutsal-arsen This is not related at all.
Your error is because the docker-containerd-shim binary is not in PATH.

Contributor

cpuguy83 commented Jun 29, 2016

@gutsal-arsen This is not related at all.
Your error is because the docker-containerd-shim binary is not in PATH.

@serofir

This comment has been minimized.

Show comment
Hide comment
@serofir

serofir Jul 2, 2016

The same problem.

root@vp:# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".
root@vp:
# uname -a
Linux vp 3.2.0-4-amd64 #1 SMP Debian 3.2.81-1 x86_64 GNU/Linux
root@vp:~# docker -v
Docker version 1.11.2, build b9f10c9

serofir commented Jul 2, 2016

The same problem.

root@vp:# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: rpc error: code = 2 desc = "shim error: invalid argument".
root@vp:
# uname -a
Linux vp 3.2.0-4-amd64 #1 SMP Debian 3.2.81-1 x86_64 GNU/Linux
root@vp:~# docker -v
Docker version 1.11.2, build b9f10c9

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 2, 2016

Member

@serofyr looks like you're running on a 3.2 kernel; docker requires 3.10 as a minimum to run; https://docs.docker.com/engine/installation/linux/debian/

Member

thaJeztah commented Jul 2, 2016

@serofyr looks like you're running on a 3.2 kernel; docker requires 3.10 as a minimum to run; https://docs.docker.com/engine/installation/linux/debian/

MiquelMarinoEspinosa pushed a commit to MiquelMarinoEspinosa/badges-io that referenced this issue Sep 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment