Cannot exec into docker container ("containerd: process not found for container") #22909
Comments
|
Was the container running when you tried to exec into it? You can only Note that a docker container is not a full virtual machine. For example, the |
|
I also spotted that your command doesn't seem to have a container-name; Should be |
|
I got a similar problem: |
|
I had the container name in the command, forgot to add it in the description (updated now). I ran sudo docker exec -it CONTAINERNAME /bin/bash. |
|
Is this the case for all containers? Can you just try and see if you get a shell? |
|
@justincormack : I am still getting the same error. |
|
as i mentioned before, it happens to me as well on all containers - and i am using ubuntu 14.04 as well. |
|
Ok, i investigated it a bit - the problem doesn't exist in version 1.10.3-0, it starts at version 1.11.0, which means that it happens in 1.11.0 and 1.11.1 |
|
Going back to version 1.10.3 fixed the problem for me as well. |
|
@galtet @agyanttiwari are you both installing from the docker apt repository (apt.dockerproject.org), or from the Ubuntu-managed repo? Can you both give the output of |
|
I followed those instructions: docker version output: Server: |
|
Error is coming from here supervisor/signal.go#L12-L27 or here /supervisor/update.go#L62-L92, so either the process died before you're trying to If you run a container interactively (e.g. simply Did you happen to have docker 1.10.0-rc1 installed at some point on that machine? Wondering if this could be a apparmor issue (see #21702 (comment), which also contains some instructions to remove the problematic profile) |
|
If i run the container interactively, it works fine. |
|
The interactive exec worked for me as well, I didn't have 1.10.0 version installed on my machine at any time. |
|
The example of @justincormack is missing a |
|
@vdemeester, the real problem here is that exec is not working, in my case I don't want to start the machine interactively. |
|
@agyanttiwari just to exclude that possibility, can you try |
So this still does not work ? |
|
@thaJeztah i still get the same error: my docker opts: |
|
@thaJeztah : I am getting the same error when I run And as for the options, I didn't change any configuration and hence, I should be using the default values. |
thaJeztah
added
area/runtime
kind/bug
thaJeztah
changed the title
|
Alright, I'm marking this as 'bug", but wondering if it's an interaction with something on the host and containerd/RunC. @galtet I'm closing your issue #22881, so that we keep the discussion in a single place (and I think it's the same issue). To try and get some more insight into what can possibly cause this, could you both try;
ping @mlaventure @crosbymichael perhaps you have further ideas? |
|
Nothing really stood out when I ran check-config.sh Generally Necessary:
Optional Features:
|
|
check-config.sh results: Generally Necessary:
Optional Features:
|
|
Is this fixed yet ? This is happening for me too when i try to connect to prometheus container. Steps to reproduce this:
|
|
@Miyurz the prometheus image is based on busybox; https://hub.docker.com/r/prom/prometheus/~/dockerfile/, which doesn't have bash. Try |
|
Cool, it works for me now. Thanks @thaJeztah |
|
We're seeing the same issue running a container on 1.11.1 (on Amazon Linux). The check-config.sh results from this env also look generally good: |
|
In our case "docker exec -it bash" initially worked after the container was started, but after ~8 minutes it stopped working. Below are the /var/log/docker file contents up to the first failure. I tried stopping the container to see if restarting it would re-enable docker exec, but the container then wouldn't start up again. |
|
@jwilmoth would it be possible for you to switch the daemon into debug mode so the logs can provide us more information? Then if you could post the updated logs when it re-occurs, it would be helpful. Also, your logs seems to show that the daemon got stopped 3 times before the issue occurred, is this an expected behavior of your setup? |
|
I work with @jonwilmoth, I was able to run this with docker debug enabled (It wasn't easy, the host seems to be in a state where I can't get it to read in anything and restarting docker doesn't help, it just returns to this state. Even a reboot fails.) It seems potentially to be related to the docker socket /var/run/docker.sock, I tried echo'ing the POST command to the socket directly and it complains of no such file or directory. I can see it there and everything. I'm attaching strace output of the run when I tried docker exec and I have a separate debug output where I ran docker manually on the host with -D at the end to get debug output. Let me know if you need anything else. |
|
I am hitting this issue as well (with docker version 1.11.1). Not able to recover my containers even after reboot. Environment is RHEL 7.2. I am hitting this issue periodically, when hit unable to recover from the problem. I think this has some connection with devicemapper driver possibly but not sure.. @mlaventure: docker-containerd.txt |
|
Any updates? at this point I'm going to try to roll back to 1.9.x to see if I can keep my services accessible post deploy. |
|
@pompeiisneaks Have you tried stracing it (see #22909 (comment)) Also, have you tried using Since I still can't reproduce the issue, it makes debugging it quite complicated. |
|
@abivarghese looking at your logs, it seems that you have quite a few device mapper related error, could it be that you're running out of space on your device? |
|
@mlaventure I'll give the strace a try. Was my strace output of the command itself not enough information, or the debug output? I'm currently trying to move back to 1.9.1, as we don't have the ability to go forward, we're locked into AWS' EB environment options, which, for docker, go only to 1.11.1 |
|
@pompeiisneaks the exec comes from runc itself, its strace may be useful to determine which path exactly is being taken |
|
@mlaventure : We have enough space on the disk, so it do not look to be a disk issue. We consistently hit device mapper issues on our containers on RHEL. Do not know how to avoid the issues? Appreciate any pointers which help to reduce the issues or impacts on RHEL.. |
|
@abivarghese sorry, at the moment, I still don't have an idea of what could be the root cause. If you could provide a strace of runc when doing the exec, it could help me getting a better idea of what's going on. |
|
ping @crosbymichael in case you get any ideas ping @abivarghese any luck with strace? :) |
|
Having the same error here too: Error: Host machine: Docker image setup command: Commands that results in the error:
e.g Command 2:
Image is running, running the commands interactively works. |
|
@aa21 docker exec $DOCKER_MYSQL_NAME which mysqlor (with a shell); docker exec $DOCKER_MYSQL_NAME /bin/sh -c "which mysql"And it should work |
|
@mlaventure, yes I tested in 1.12 (first time ever with docker) and I get same error
$ docker version |
|
@qupro is that an image you built yourself? Make sure that;
These errors are usually directly from the kernel, and can be cryptic |
|
@thaJeztah I am using this docker container https://github.com/geo-data/openstreetmap-tiles-docker, and I was running this |
|
@qupro the parameter to exec after the container id is directly the command to be executed, you can forgo the run. I'll close this issue meanwhile seems it seems the issue was the command used. If you have more questions on how to use docker, feel free to join our slack community: https://community.docker.com/registrations/groups/4316 |
|
I used to manage this issue by simply restarting the docker daemon:
|
|
Also, restart did help :( |
|
@RunsFor looks like you're running an old pre-release version of docker; at least try updating to the current release. If you still have issues, please open a new issue with details, and steps to reproduce. |
|
It looks like the original issue reported here was resolved, and I want to prevent this issue from becoming a kitchen sink of possibly unrelated issues, so I'm going to lock the conversation here. If you still have issues and suspect there's a bug at hand, please open a new issue. Be sure to include as much details as possible, and steps to reproduce to help debugging the issue. |
Output of
docker version:Docker version 1.11.1, build 5604cbe
Output of
docker info:Containers: 8
Running: 6
Paused: 0
Stopped: 2
Images: 14
Server Version: 1.11.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 87
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.16.0-67-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.31 GiB
Name: agyantdev
ID: XSB4:V4GP:MKT2:R3U5:7P3B:ZPPA:AOLJ:6AYU:SCS4:SYXI:B73D:YA6J
Docker Root Dir: /var/lib/docker
Debug mode (client): true
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Steps to reproduce the issue:
also tried bash, bin/bash, sh, bin/sh, /bin/sh, ect
Describe the results you received:
time="2016-05-23T15:26:43.731439437+01:00" level=error msg="Error running exec in container: rpc error: code = 2 desc = "oci runtime error: exec failed: exit status 1"\n"
time="2016-05-23T15:26:43.731481314+01:00" level=error msg="Handler for POST /v1.23/exec/4e370de345ab1b06504c1242375c3b41e4b5ecd843aa486b54d288803e1f402a/start returned error: rpc error: code = 2 desc = "oci runtime error: exec failed: exit status 1""
2016/05/23 15:26:43 http: response.WriteHeader on hijacked connection
2016/05/23 15:26:43 http: response.Write on hijacked connection
time="2016-05-23T15:26:43.731718484+01:00" level=error msg="Handler for POST /v1.23/exec/4e370de345ab1b06504c1242375c3b41e4b5ecd843aa486b54d288803e1f402a/resize returned error: rpc error: code = 2 desc = "containerd: processs not found for container""
Describe the results you expected:
SSH terminal of the machine
Additional information you deem important (e.g. issue happens only occasionally):
Uninstalled and installed docker-engine multiple times
The text was updated successfully, but these errors were encountered: