Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Daemon + containers stuck with "container not found" #24985

Closed
paralin opened this issue Jul 24, 2016 · 12 comments
Closed

Daemon + containers stuck with "container not found" #24985

paralin opened this issue Jul 24, 2016 · 12 comments
Labels
area/runtime status/more-info-needed version/1.12

Comments

@paralin
Copy link

paralin commented Jul 24, 2016

This is in rc4.

Jul 24 18:35:21 c2 dockerd[1130]: time="2016-07-24T18:35:21.197213000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 59f16c30a5e0875b8d5b1785b948c32512adc68b95bdef286d6276888b856805: rpc error: code = 2 desc = containerd: container not found"
Jul 24 18:35:31 c2 dockerd[1130]: time="2016-07-24T18:35:31.198737000Z" level=info msg="Container 59f16c30a5e0 failed to exit within 10 seconds of kill - trying direct SIGKILL"
Jul 24 18:35:31 c2 dockerd[1130]: time="2016-07-24T18:35:31.200938000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 59f16c30a5e0875b8d5b1785b948c32512adc68b95bdef286d6276888b856805: rpc error: code = 2 desc = containerd: container not found"
Jul 24 18:35:41 c2 dockerd[1130]: time="2016-07-24T18:35:41.202139000Z" level=info msg="Container 59f16c30a5e0 failed to exit within 10 seconds of kill - trying direct SIGKILL"
Jul 24 18:35:41 c2 dockerd[1130]: time="2016-07-24T18:35:41.204417000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 59f16c30a5e0875b8d5b1785b948c32512adc68b95bdef286d6276888b856805: rpc error: code = 2 desc = containerd: container not found"
Jul 24 18:35:44 c2 dockerd[1130]: time="2016-07-24T18:35:44.205716000Z" level=info msg="Container 59f16c30a5e0875b8d5b1785b948c32512adc68b95bdef286d6276888b856805 failed to exit within 3 seconds of signal 15 - using the force"
Jul 24 18:35:44 c2 dockerd[1130]: time="2016-07-24T18:35:44.208008000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 59f16c30a5e0875b8d5b1785b948c32512adc68b95bdef286d6276888b856805: rpc error: code = 2 desc = containerd: container not found"
Jul 24 18:35:54 c2 dockerd[1130]: time="2016-07-24T18:35:54.209302000Z" level=info msg="Container 59f16c30a5e0 failed to exit within 10 seconds of kill - trying direct SIGKILL"
Jul 24 18:36:40 c2 dockerd[1130]: time="2016-07-24T18:36:40.818179000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 0a2a1e9896c07b7b18b0b756245eca7c008df5923c66f21b7644bde453d8d927: rpc error: code = 2 desc = containerd: container not found"
Jul 24 18:36:50 c2 dockerd[1130]: time="2016-07-24T18:36:50.818615000Z" level=info msg="Container 0a2a1e9896c0 failed to exit within 10 seconds of kill - trying direct SIGKILL"
Jul 24 18:36:50 c2 dockerd[1130]: time="2016-07-24T18:36:50.820860000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 0a2a1e9896c07b7b18b0b756245eca7c008df5923c66f21b7644bde453d8d927: rpc error: code = 2 desc = containerd: container not found"
Jul 24 18:37:00 c2 dockerd[1130]: time="2016-07-24T18:37:00.822067000Z" level=info msg="Container 0a2a1e9896c0 failed to exit within 10 seconds of kill - trying direct SIGKILL"
Jul 24 18:37:00 c2 dockerd[1130]: time="2016-07-24T18:37:00.824340000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 0a2a1e9896c07b7b18b0b756245eca7c008df5923c66f21b7644bde453d8d927: rpc error: code = 2 desc = containerd: container not found"
Jul 24 18:37:03 c2 dockerd[1130]: time="2016-07-24T18:37:03.825652000Z" level=info msg="Container 0a2a1e9896c07b7b18b0b756245eca7c008df5923c66f21b7644bde453d8d927 failed to exit within 3 seconds of signal 15 - using the force"
Jul 24 18:37:03 c2 dockerd[1130]: time="2016-07-24T18:37:03.827840000Z" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 0a2a1e9896c07b7b18b0b756245eca7c008df5923c66f21b7644bde453d8d927: rpc error: code = 2 desc = containerd: container not found"

Docker info:

Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 7
Server Version: v1.12.0-rc4
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.14.65
Operating System: Buildroot 2016.08-git
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.928 GiB
Name: c2
ID: WUZ2:AUHN:5MHH:T4DK:FRYA:FQPE:YGLT:EAT7:FQ64:D3J4:KSBM:ACVL
Docker Root Dir: /mnt/persist/skiff/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Docker version:

Client:
 Version:      v1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   unknown
 Built:        Sun Jul 24 00:29:11 PDT 2016
 OS/Arch:      linux/arm64

Server:
 Version:      v1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   unknown
 Built:        Sun Jul 24 00:29:11 PDT 2016
 OS/Arch:      linux/arm64

It starts the first layer of the docker build, that layer finishes, and then it just sits there frozen.

containerd version 0.2.0 commit: 0ac3cd1be170d180b2baed755e8f0da547ceb267

runc -v
runc version commit: v1.0.0-rc1
spec: 1.0.0-rc1
@paralin
Copy link
Author

paralin commented Jul 24, 2016

Possibly related to #24974

@paralin
Copy link
Author

paralin commented Jul 24, 2016

Reverting to docker-containerd 1b3a81545ca79456086dc2aa424357be98b962ee and docker-engine v1.12.0-rc3 fixes these issues.

paralin added a commit to skiffos/buildroot that referenced this issue Jul 24, 2016
@paralin
docker-containerd, docker-engine: temporary revert
8b3f377 
Temporary revert until these issues are fixed:

 - moby/moby#24974
 - moby/moby#24985

Signed-off-by: Christian Stewart <christian@paral.in>
paralin added a commit to skiffos/buildroot that referenced this issue Jul 24, 2016
@paralin
docker-containerd, docker-engine: temporary revert
4bf5232 
Temporary revert until these issues are fixed:

 - moby/moby#24974
 - moby/moby#24985

Signed-off-by: Christian Stewart <christian@paral.in>
paralin added a commit to skiffos/buildroot that referenced this issue Jul 24, 2016
@paralin
docker-containerd, docker-engine: temporary revert
f46d5dc 
Temporary revert until these issues are fixed:

 - moby/moby#24974
 - moby/moby#24985

Signed-off-by: Christian Stewart <christian@paral.in>
paralin added a commit to skiffos/buildroot that referenced this issue Jul 24, 2016
@paralin
docker-containerd, docker-engine: temporary revert
4767b4b 
Temporary revert until these issues are fixed:

 - moby/moby#24974
 - moby/moby#24985

Signed-off-by: Christian Stewart <christian@paral.in>
paralin added a commit to skiffos/buildroot that referenced this issue Jul 24, 2016
@paralin
docker-containerd, docker-engine: temporary revert
e9ad055 
Temporary revert until these issues are fixed:

 - moby/moby#24974
 - moby/moby#24985

Signed-off-by: Christian Stewart <christian@paral.in>
paralin added a commit to skiffos/buildroot that referenced this issue Jul 27, 2016
@paralin
docker-containerd, docker-engine: temporary revert
5968cc4 
Temporary revert until these issues are fixed:

 - moby/moby#24974
 - moby/moby#24985

Signed-off-by: Christian Stewart <christian@paral.in>
@thaJeztah
Copy link
Member

@paralin are you still having this on 1.12.0, or can this be closed?

@paralin
Copy link
Author

paralin commented Aug 5, 2016

Will test today.

@paralin
Copy link
Author

paralin commented Aug 12, 2016

This is still happening on v1.12.0.

# docker info
docker info
Containers: 3
 Running: 3
 Paused: 0
 Stopped: 0
Images: 5
Server Version: v1.12.0
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.14.65
Operating System: Buildroot 2016.08-rc1
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.928 GiB
Name: base3
ID: TAWH:F74C:6UOG:P73E:SNWQ:NGOY:UYJB:I4GQ:ICES:2GOD:OWZJ:VNDO
Docker Root Dir: /mnt/persist/skiff/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

# docker version
docker version
Client:
 Version:      v1.12.0
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   unknown
 Built:        Wed Aug 10 15:08:58 PDT 2016
 OS/Arch:      linux/arm64

Server:
 Version:      v1.12.0
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   unknown
 Built:        Wed Aug 10 15:08:58 PDT 2016
 OS/Arch:      linux/arm64

docker-containerd --version
containerd version 0.2.0 commit: 0ac3cd1be170d180b2baed755e8f0da547ceb267

runc --version
runc version commit: v1.0.0-rc1
spec: 1.0.0-rc1

@paralin
Copy link
Author

paralin commented Aug 12, 2016

cc @thaJeztah

@thaJeztah
Copy link
Member

Looks like you're using a different version of RunC than is used in 1.12.0? opencontainers/runc@v1.0.0-rc1...cc29e3d

@paralin
Copy link
Author

paralin commented Aug 12, 2016

Probably, I can bump it and see if it fixes the problem. I don't see any drastic changes in that changelog though.

@thaJeztah
Copy link
Member

ping @paralin is this still an issue, or OK to close?

@NZVIDIA
Copy link

NZVIDIA commented Feb 22, 2017

I'm seeing this issue with docker 1.13.1 on CentOS 7.3.

@thaJeztah
Copy link
Member

@NZVIDIA could you open a new issue? It's possible that it's the same issue, but you're running a different version of docker and different distro, so it may be good to start with a "fresh" issue.

@paralin I'll go ahead and close this issue, but happy to hear if you're still having this problem

@amidvidy amidvidy mentioned this issue May 31, 2017
Closed
@Rajavelu
Copy link

Rajavelu commented Mar 13, 2018 •
edited
Loading

Still met the same error as follows, Please help someone for fix the same. CentOS7.3 & Docker version 1.12.6

WARN[0239] failed to close stdin: rpc error: code = 2 desc = containerd: container not found
ERRO[0293] devmapper: Error unmounting device abdaa2fcf5c5e6a23b520a32b4dae354b09b9b629e9bf1e931c8e2b9b3188143: Device is Busy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/runtime status/more-info-needed version/1.12
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@paralin @thaJeztah @Rajavelu @NZVIDIA