Creating fail with Could not find container for entity id <id> after upgrading to 1.9.0 #17691

Closed
gionn opened this Issue Nov 4, 2015 · 42 comments

Comments

Projects
None yet
@gionn

gionn commented Nov 4, 2015


Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 17:43:42 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 17:43:42 UTC 2015
 OS/Arch:      linux/amd64

Linux gt-xps 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Containers: 1
Images: 37
Server Version: 1.9.0
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-31-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 7.372 GiB
Name: gt-xps
ID: 2562:2TQM:TR3I:CDFJ:T6JP:WXBN:AV4F:LZYT:HIAX:ZFYC:MMWD:Q7AN
Username: gionn
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Before I was using 1.8.3 with aufs storage driver.

$ docker run --rm -it --name cloudesire_broker_1 ubuntu
Error response from daemon: Could not find container for entity id 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396

$ docker inspect cloudesire_broker_1
Error: No such image or container: cloudesire_broker_1
[]

$ docker inspect 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396
Error: No such image or container: 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396
[]

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

First debug happened in compose issue tracker since it appeared related to it: docker/compose#2316

I've a feeling that has something to do with containers that was existing before the upgrade to 1.9.0 from 1.8.3, with the contextual switch from storage driver aufs to overlay.

@GordonTheTurtle

This comment has been minimized.

Show comment
Hide comment
@GordonTheTurtle

GordonTheTurtle Nov 4, 2015

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues


BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues


BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Nov 4, 2015

Contributor

@gionn Hmm, yes I think this is probably it. Maybe remove all containers, and then remove /var/lib/docker/linkgraph.db.
Not sure if this is feasible for your setup.
There's a PR to namepsace these link graphs per graph driver.

Contributor

cpuguy83 commented Nov 4, 2015

@gionn Hmm, yes I think this is probably it. Maybe remove all containers, and then remove /var/lib/docker/linkgraph.db.
Not sure if this is feasible for your setup.
There's a PR to namepsace these link graphs per graph driver.

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Nov 4, 2015

Member

There's some relevant discussion here: https://github.com/docker/docker/pull/17389/files#r43595659

And the PR @cpuguy83 is referring to which proposes a fix: #17657

Member

dnephin commented Nov 4, 2015

There's some relevant discussion here: https://github.com/docker/docker/pull/17389/files#r43595659

And the PR @cpuguy83 is referring to which proposes a fix: #17657

@markmandel

This comment has been minimized.

Show comment
Hide comment
@markmandel

markmandel Nov 4, 2015

I just ran into the same issue, on my local machine.

I tried everything above, and also reinstalled docker - solution for me - start the container with a different --name value. YMMV.

I just ran into the same issue, on my local machine.

I tried everything above, and also reinstalled docker - solution for me - start the container with a different --name value. YMMV.

@abhas

This comment has been minimized.

Show comment
Hide comment
@abhas

abhas Nov 9, 2015

The only way I could fix this issue was by deleting the contents of /var/lib/docker/containers and also deleting the /var/lib/docker/linkgraph.db file. Thanks @cpuguy83.

abhas commented Nov 9, 2015

The only way I could fix this issue was by deleting the contents of /var/lib/docker/containers and also deleting the /var/lib/docker/linkgraph.db file. Thanks @cpuguy83.

@jbq

This comment has been minimized.

Show comment
Hide comment
@jbq

jbq Nov 10, 2015

Same for me while migrating from devicemapper storage to overlay. I had to manually remove the aforementioned low-level docker files.

jbq commented Nov 10, 2015

Same for me while migrating from devicemapper storage to overlay. I had to manually remove the aforementioned low-level docker files.

@ruudud

This comment has been minimized.

Show comment
Hide comment
@ruudud

ruudud Nov 13, 2015

Same for me, migrated with a clean cut from aufs to overlay. Had to remove linkgraph.db to to be able to start a container with the same name as one I had been using before with aufs.

ruudud commented Nov 13, 2015

Same for me, migrated with a clean cut from aufs to overlay. Had to remove linkgraph.db to to be able to start a container with the same name as one I had been using before with aufs.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 18, 2015

I've just update docker to 1.9 and I've got the same issue.
Thank @abhas, I've deleted the directory /var/lib/docker/containers and the file /var/lib/docker/linkgraph.db and it worked.

ghost commented Nov 18, 2015

I've just update docker to 1.9 and I've got the same issue.
Thank @abhas, I've deleted the directory /var/lib/docker/containers and the file /var/lib/docker/linkgraph.db and it worked.

@naag

This comment has been minimized.

Show comment
Hide comment
@naag

naag Nov 19, 2015

Contributor

Same here. It didn't happen for all containers though, in some cases even just one container as affected.

$ docker info
Containers: 7
Images: 1834
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1948
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.0-16-generic
Operating System: Ubuntu 15.10
CPUs: 4
Total Memory: 15.56 GiB
Name: phuong-HP-ProBook-6470b
ID: QWGD:PGHS:24JH:Q4D2:UMU3:HICM:TLNV:CEFS:47GM:MHCE:SA5M:GDKW
WARNING: No swap limit support

$ docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 17:48:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 17:48:04 UTC 2015
 OS/Arch:      linux/amd64

$ uname -a
Linux phuong-HP-ProBook-6470b 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

The actual error message was Error response from daemon: Could not find container for entity id b36386384e81a15c8891708502ee40419a312a41dd1db733eb114293572c8541. Problem was also fixed by deleting /var/lib/docker/containers/* and /var/lib/docker/linkgraph.db.

Contributor

naag commented Nov 19, 2015

Same here. It didn't happen for all containers though, in some cases even just one container as affected.

$ docker info
Containers: 7
Images: 1834
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1948
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.0-16-generic
Operating System: Ubuntu 15.10
CPUs: 4
Total Memory: 15.56 GiB
Name: phuong-HP-ProBook-6470b
ID: QWGD:PGHS:24JH:Q4D2:UMU3:HICM:TLNV:CEFS:47GM:MHCE:SA5M:GDKW
WARNING: No swap limit support

$ docker version
Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 17:48:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   76d6bc9
 Built:        Tue Nov  3 17:48:04 UTC 2015
 OS/Arch:      linux/amd64

$ uname -a
Linux phuong-HP-ProBook-6470b 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

The actual error message was Error response from daemon: Could not find container for entity id b36386384e81a15c8891708502ee40419a312a41dd1db733eb114293572c8541. Problem was also fixed by deleting /var/lib/docker/containers/* and /var/lib/docker/linkgraph.db.

@gionn

This comment has been minimized.

Show comment
Hide comment
@gionn

gionn Nov 19, 2015

hey guys, we already had multiple confirmation of this bug, please avoid repeating that you got this bug and you fixed it with the suggested workaround.

Use subscribe button if you want updates.

Cheers

gionn commented Nov 19, 2015

hey guys, we already had multiple confirmation of this bug, please avoid repeating that you got this bug and you fixed it with the suggested workaround.

Use subscribe button if you want updates.

Cheers

@zymtx5g79k

This comment has been minimized.

Show comment
Hide comment
@zymtx5g79k

zymtx5g79k Nov 24, 2015

Всем привет!
docker version:

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

docker info:

Containers: 26
Images: 30
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 82
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-68-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 7.704 GiB
Name: laptop-U36SG
ID: WXB3:ENML:LQFP:XFZB:WSDZ:J2X6:ENDV:P3AT:ZNH3:OJ4R:GOI4:XXSJ
WARNING: No swap limit support

uname -a:

Linux laptop-U36SG 3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Provide additional environment details (AWS, VirtualBox, physical, etc.):
На локальной машине. Обновил штатными средствами ubuntu и приехал. Пробовал перегрузиться. Удалять все контейнеры и образы. Не помогло ):

List the steps to reproduce the issue:

  1. Собрать контейнер. Dockerfile: https://gist.github.com/zymtx5g79k/a39e1196d93ae9c4067d
  2. Запустить

Describe the results you received:

Error response from daemon: Could not find container for entity id 92ab1a4d2e7ffb6ff29cefca5ea735387cdf6be8fefa7142c282d41ca7a0eb25

Describe the results you expected:
Ожидал увидеть вывод супервизора, работающего в контейнере.

Provide additional info you think is important:

Не все образы постигла такая участь. Вот этот контейнер запускается !без проблем Dockerfile: https://gist.github.com/zymtx5g79k/80270f08124e2d461215

Всем привет!
docker version:

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

docker info:

Containers: 26
Images: 30
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 82
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-68-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 7.704 GiB
Name: laptop-U36SG
ID: WXB3:ENML:LQFP:XFZB:WSDZ:J2X6:ENDV:P3AT:ZNH3:OJ4R:GOI4:XXSJ
WARNING: No swap limit support

uname -a:

Linux laptop-U36SG 3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Provide additional environment details (AWS, VirtualBox, physical, etc.):
На локальной машине. Обновил штатными средствами ubuntu и приехал. Пробовал перегрузиться. Удалять все контейнеры и образы. Не помогло ):

List the steps to reproduce the issue:

  1. Собрать контейнер. Dockerfile: https://gist.github.com/zymtx5g79k/a39e1196d93ae9c4067d
  2. Запустить

Describe the results you received:

Error response from daemon: Could not find container for entity id 92ab1a4d2e7ffb6ff29cefca5ea735387cdf6be8fefa7142c282d41ca7a0eb25

Describe the results you expected:
Ожидал увидеть вывод супервизора, работающего в контейнере.

Provide additional info you think is important:

Не все образы постигла такая участь. Вот этот контейнер запускается !без проблем Dockerfile: https://gist.github.com/zymtx5g79k/80270f08124e2d461215
@zymtx5g79k

This comment has been minimized.

Show comment
Hide comment
@zymtx5g79k

zymtx5g79k Nov 24, 2015

Ty @naag

deleting /var/lib/docker/containers/* and /var/lib/docker/linkgraph.db.

  • удалил все контейнеры и образы
  • /var/lib/docker/containers/ была пустой
  • /var/lib/docker/linkgraph.db удалил руками
  • перезапустил службу docker service docker restart
  • собрал контейнер по новой.

Заработало! (:

Ty @naag

deleting /var/lib/docker/containers/* and /var/lib/docker/linkgraph.db.

  • удалил все контейнеры и образы
  • /var/lib/docker/containers/ была пустой
  • /var/lib/docker/linkgraph.db удалил руками
  • перезапустил службу docker service docker restart
  • собрал контейнер по новой.

Заработало! (:

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 25, 2015

Member

@zymtx5g79k could you please edit your comments and translate it to English, most people here won't be able to read it otherwise.

Member

thaJeztah commented Nov 25, 2015

@zymtx5g79k could you please edit your comments and translate it to English, most people here won't be able to read it otherwise.

@error10

This comment has been minimized.

Show comment
Hide comment
@error10

error10 Nov 26, 2015

docker version:
Client:
Version: 1.9.1-fc23
API version: 1.21
Package version: docker-1.9.1-2.git78bc3ea.fc23.x86_64
Go version: go1.5.1
Git commit: f7c1d52-dirty
Built: Fri Nov 20 21:07:14 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.9.1-fc23
API version: 1.21
Package version: docker-1.9.1-2.git78bc3ea.fc23.x86_64
Go version: go1.5.1
Git commit: f7c1d52-dirty
Built: Fri Nov 20 21:07:14 UTC 2015
OS/Arch: linux/amd64

docker info:
Containers: 9
Images: 123
Server Version: 1.9.1-fc23
Storage Driver: devicemapper
Pool Name: docker-253:1-573125-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/docker-volumes/docker-data
Metadata file: /dev/docker-volumes/docker-meta
Data Space Used: 2.518 GB
Data Space Total: 42.9 GB
Data Space Available: 40.38 GB
Metadata Space Used: 5.562 MB
Metadata Space Total: 46.14 MB
Metadata Space Available: 40.57 MB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.2.6-300.fc23.x86_64
Operating System: Fedora 23 (Twenty Three)
CPUs: 4
Total Memory: 3.86 GiB
Name: docker2..com
ID: JG2X:PHQM:DX2R:IYCR:UREV:3AWO:O5DC:3NKY:7YTW:26V2:NNLY:HABR

uname -a:
Linux docker2..com 4.2.6-300.fc23.x86_64 #1 SMP Tue Nov 10 19:32:21 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Provide additional environment details (AWS, VirtualBox, physical, etc.):
KVM virtual machine

List the steps to reproduce the issue:

  1. On Fedora 23, updated docker-1.8.2 to docker-1.9.1 with dnf update
  2. Restarted Docker

Describe the results you received:
Nov 25 23:57:18 docker2..com docker[780]: time="2015-11-25T23:57:18.055177233Z" level=error msg="HTTP Error" err="Could not find container for entity id 985b3b0c08cd6b139c5029cb486eab28aca9fb8ab73992380285275e9395bc23" statusCode=500

Describe the results you expected:
Containers starting normally

Provide additional info you think is important:
We did not change the storage driver during this update. It remained devicemapper, with the same backing store. The workaround of deleting containers and linkgraph.db restored normal operation.

error10 commented Nov 26, 2015

docker version:
Client:
Version: 1.9.1-fc23
API version: 1.21
Package version: docker-1.9.1-2.git78bc3ea.fc23.x86_64
Go version: go1.5.1
Git commit: f7c1d52-dirty
Built: Fri Nov 20 21:07:14 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.9.1-fc23
API version: 1.21
Package version: docker-1.9.1-2.git78bc3ea.fc23.x86_64
Go version: go1.5.1
Git commit: f7c1d52-dirty
Built: Fri Nov 20 21:07:14 UTC 2015
OS/Arch: linux/amd64

docker info:
Containers: 9
Images: 123
Server Version: 1.9.1-fc23
Storage Driver: devicemapper
Pool Name: docker-253:1-573125-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/docker-volumes/docker-data
Metadata file: /dev/docker-volumes/docker-meta
Data Space Used: 2.518 GB
Data Space Total: 42.9 GB
Data Space Available: 40.38 GB
Metadata Space Used: 5.562 MB
Metadata Space Total: 46.14 MB
Metadata Space Available: 40.57 MB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.2.6-300.fc23.x86_64
Operating System: Fedora 23 (Twenty Three)
CPUs: 4
Total Memory: 3.86 GiB
Name: docker2..com
ID: JG2X:PHQM:DX2R:IYCR:UREV:3AWO:O5DC:3NKY:7YTW:26V2:NNLY:HABR

uname -a:
Linux docker2..com 4.2.6-300.fc23.x86_64 #1 SMP Tue Nov 10 19:32:21 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Provide additional environment details (AWS, VirtualBox, physical, etc.):
KVM virtual machine

List the steps to reproduce the issue:

  1. On Fedora 23, updated docker-1.8.2 to docker-1.9.1 with dnf update
  2. Restarted Docker

Describe the results you received:
Nov 25 23:57:18 docker2..com docker[780]: time="2015-11-25T23:57:18.055177233Z" level=error msg="HTTP Error" err="Could not find container for entity id 985b3b0c08cd6b139c5029cb486eab28aca9fb8ab73992380285275e9395bc23" statusCode=500

Describe the results you expected:
Containers starting normally

Provide additional info you think is important:
We did not change the storage driver during this update. It remained devicemapper, with the same backing store. The workaround of deleting containers and linkgraph.db restored normal operation.

@ikor

This comment has been minimized.

Show comment
Hide comment
@ikor

ikor Dec 2, 2015

For anyone who is still encountering this issue, this is a workaround:

  • Remove rm -f /var/lib/linkpgraph.db (or if you modified graphdb=/opt/docker : rm -rf /opt/docker/linkgraph.db)
  • Remove rm -rf /var/lib/containers/
  • Restart docker daemon : (for Ubuntu) sudo service docker restart

ikor commented Dec 2, 2015

For anyone who is still encountering this issue, this is a workaround:

  • Remove rm -f /var/lib/linkpgraph.db (or if you modified graphdb=/opt/docker : rm -rf /opt/docker/linkgraph.db)
  • Remove rm -rf /var/lib/containers/
  • Restart docker daemon : (for Ubuntu) sudo service docker restart
@cturra

This comment has been minimized.

Show comment
Hide comment
@cturra

cturra Dec 2, 2015

i too have run into this across a small fleet of docker host i run. the instructions noted above by @ikor got us back on track, but to make sure this doesn't bite us again, i have rolled back docker-engine on these nodes to 1.8.3.

cturra commented Dec 2, 2015

i too have run into this across a small fleet of docker host i run. the instructions noted above by @ikor got us back on track, but to make sure this doesn't bite us again, i have rolled back docker-engine on these nodes to 1.8.3.

@areidyOTH

This comment has been minimized.

Show comment
Hide comment
@areidyOTH

areidyOTH Dec 10, 2015

This problem occurs when the tables 'edge' and 'entity' in the SQLite database /var/lib/linkgraph.db contain records for a container name already but /var/lib/containers has no data for the corresponding containerid. I think it's caused by aee5486 which was added to 1.9.0 to fix #17389

Previously, if you tried to start a container that had records in linkgraph.db but there was no data in /var/lib/containers then the records would be deleted from the database. Now this error occurs instead.

I can't work out what causes the mismatch between the records in linkgraph.db and /var/lib/containers - abnormal termination maybe?

The fixes are:

  • downgrade to 1.8.3, or
  • manually delete the records in linkgraph.db every time the error occurs

This problem occurs when the tables 'edge' and 'entity' in the SQLite database /var/lib/linkgraph.db contain records for a container name already but /var/lib/containers has no data for the corresponding containerid. I think it's caused by aee5486 which was added to 1.9.0 to fix #17389

Previously, if you tried to start a container that had records in linkgraph.db but there was no data in /var/lib/containers then the records would be deleted from the database. Now this error occurs instead.

I can't work out what causes the mismatch between the records in linkgraph.db and /var/lib/containers - abnormal termination maybe?

The fixes are:

  • downgrade to 1.8.3, or
  • manually delete the records in linkgraph.db every time the error occurs
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
Member

thaJeztah commented Dec 10, 2015

ping @tonistiigi ^^

@iwidanalage

This comment has been minimized.

Show comment
Hide comment
@iwidanalage

iwidanalage Dec 11, 2015

We hit the same issue with 1.9.0. But deleting all containers was not an option for us. So did a very hacky workaround by going in to modifying the record on the edge table using sqllite cli.

 sqlite3 linkgraph.db 

update edge set name='some-random-container-name' where name='failing-container-name';

Hope this helps.

We hit the same issue with 1.9.0. But deleting all containers was not an option for us. So did a very hacky workaround by going in to modifying the record on the edge table using sqllite cli.

 sqlite3 linkgraph.db 

update edge set name='some-random-container-name' where name='failing-container-name';

Hope this helps.

@nazar-pc

This comment has been minimized.

Show comment
Hide comment
@nazar-pc

nazar-pc Dec 14, 2015

Same here on Docker 1.9.1, BTRFS, Ubuntu 16.04, Kernel 4.4-rc4
Happened after machine stuck (I have a lot of experimental software here, not sure why it happened, this is for the first time).
After reboot one of containers doesn't start using Docker Compose, while other even from the same image starting fine.
I'm removing all containers & volumes and it fails when I want to start the same container with Docker Compose again.
I do have directory with such ID under /var/lib/docker/btrfs/subvolumes and /var/lib/docker/containers, but one under /var/lib/docker/containers is completely empty. Also, container is present in /var/lib/docker/linkgraph.db in both tables, but not visible in docker ps -a.
After this:

sudo rmdir /var/lib/docker/containers/5d85dd4a476f7c05a667676bcd53ad7cd8e1653db0f4c1288257759eb39ac142
sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/5d85dd4a476f7c05a667676bcd53ad7cd8e1653db0f4c1288257759eb39ac142
sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/5d85dd4a476f7c05a667676bcd53ad7cd8e1653db0f4c1288257759eb39ac142-init

Error still happens, so there is a need for some mechanism to handle empty directory under /var/lib/docker/containers/ nicely and normalize /var/lib/docker/linkgraph.db

Same here on Docker 1.9.1, BTRFS, Ubuntu 16.04, Kernel 4.4-rc4
Happened after machine stuck (I have a lot of experimental software here, not sure why it happened, this is for the first time).
After reboot one of containers doesn't start using Docker Compose, while other even from the same image starting fine.
I'm removing all containers & volumes and it fails when I want to start the same container with Docker Compose again.
I do have directory with such ID under /var/lib/docker/btrfs/subvolumes and /var/lib/docker/containers, but one under /var/lib/docker/containers is completely empty. Also, container is present in /var/lib/docker/linkgraph.db in both tables, but not visible in docker ps -a.
After this:

sudo rmdir /var/lib/docker/containers/5d85dd4a476f7c05a667676bcd53ad7cd8e1653db0f4c1288257759eb39ac142
sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/5d85dd4a476f7c05a667676bcd53ad7cd8e1653db0f4c1288257759eb39ac142
sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/5d85dd4a476f7c05a667676bcd53ad7cd8e1653db0f4c1288257759eb39ac142-init

Error still happens, so there is a need for some mechanism to handle empty directory under /var/lib/docker/containers/ nicely and normalize /var/lib/docker/linkgraph.db

@TJC

This comment has been minimized.

Show comment
Hide comment
@TJC

TJC Dec 14, 2015

We've been seeing this occur on some production machines on docker version 1.9.1-0~trusty. But luckily, not all machines.

Pretty disappointing to have services going down due to this bug.. We skipped 1.9.0 and went straight to 1.9.1, but I don't think that's relevant.

TJC commented Dec 14, 2015

We've been seeing this occur on some production machines on docker version 1.9.1-0~trusty. But luckily, not all machines.

Pretty disappointing to have services going down due to this bug.. We skipped 1.9.0 and went straight to 1.9.1, but I don't think that's relevant.

@niccokunzmann

This comment has been minimized.

Show comment
Hide comment
@niccokunzmann

niccokunzmann Dec 16, 2015

Having this in 1.9.1

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:20:08 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:20:08 UTC 2015
 OS/Arch:      linux/amd64


Error response from daemon: Could not find container for entity id 33f6f30a91eb31a86e9c715a1fba4cc28c8031ab62826d93c799e3849c341d26

When removing the --name parameter for docker run, it works.

Having this in 1.9.1

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:20:08 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:20:08 UTC 2015
 OS/Arch:      linux/amd64


Error response from daemon: Could not find container for entity id 33f6f30a91eb31a86e9c715a1fba4cc28c8031ab62826d93c799e3849c341d26

When removing the --name parameter for docker run, it works.

niccokunzmann added a commit to niccokunzmann/dockerfiles that referenced this issue Dec 17, 2015

@dennybaa

This comment has been minimized.

Show comment
Hide comment
@dennybaa

dennybaa Dec 23, 2015

+1 recently run into the same ugly issue. I've deleted all containers, images partly and anyway(

+1 recently run into the same ugly issue. I've deleted all containers, images partly and anyway(

@LK4D4 LK4D4 closed this in #16032 Jan 11, 2016

@thaJeztah thaJeztah added this to the 1.10.0 milestone Jan 12, 2016

@koliyo

This comment has been minimized.

Show comment
Hide comment
@koliyo

koliyo Jan 28, 2016

Same problem. Was able to solve it by removing /var/lib/docker/containers and /var/lib/docker/linkgraph.db and then restarting the docker-machine vm

koliyo commented Jan 28, 2016

Same problem. Was able to solve it by removing /var/lib/docker/containers and /var/lib/docker/linkgraph.db and then restarting the docker-machine vm

@mikeatlas

This comment has been minimized.

Show comment
Hide comment
@mikeatlas

mikeatlas Feb 2, 2016

Encountered same issue here while switching kernel versions from 3.13.76 (avoiding the buggy AUFS kernel issue #18180) to 3.19 in order to use overlayfs (development box only)... able to work around the problem by removing /var/lib/docker/containers and /var/lib/docker/linkgraph.db and then restarting the docker daemon (in my case, with the storage driver explicitly set to overlay):

$ sudo rm -rf /var/lib/docker/containers
$ sudo rm /var/lib/docker/linkgraph.db
$ sudo service docker restart
$ docker info
Containers: 6
Images: 141
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-47-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 2
Total Memory: 7.774 GiB

After rebuilding my docker-composed images and projects, all working again (although you tend to forget how many layers need to be re-downloaded and rebuilt in situations like this... a lot of finger tapping)...

Encountered same issue here while switching kernel versions from 3.13.76 (avoiding the buggy AUFS kernel issue #18180) to 3.19 in order to use overlayfs (development box only)... able to work around the problem by removing /var/lib/docker/containers and /var/lib/docker/linkgraph.db and then restarting the docker daemon (in my case, with the storage driver explicitly set to overlay):

$ sudo rm -rf /var/lib/docker/containers
$ sudo rm /var/lib/docker/linkgraph.db
$ sudo service docker restart
$ docker info
Containers: 6
Images: 141
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-47-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 2
Total Memory: 7.774 GiB

After rebuilding my docker-composed images and projects, all working again (although you tend to forget how many layers need to be re-downloaded and rebuilt in situations like this... a lot of finger tapping)...

@raimille1

This comment has been minimized.

Show comment
Hide comment
@raimille1

raimille1 Feb 2, 2016

For people using docker in more of a production-like environment and don't want to rm the world, I just did this and it worked:

sudo sqlite3 /var/lib/docker/linkgraph.db
delete from edge where entity_id = '<conflicting-entity-id-sha-1>';
delete from entity where id = '<conflicting-entity-id-sha-1>';

Where is the entity id causing you trouble like: 33f6f30a91eb31a86e9c715a1fba4cc28c8031ab62826d93c799e3849c341d26

For people using docker in more of a production-like environment and don't want to rm the world, I just did this and it worked:

sudo sqlite3 /var/lib/docker/linkgraph.db
delete from edge where entity_id = '<conflicting-entity-id-sha-1>';
delete from entity where id = '<conflicting-entity-id-sha-1>';

Where is the entity id causing you trouble like: 33f6f30a91eb31a86e9c715a1fba4cc28c8031ab62826d93c799e3849c341d26

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Feb 2, 2016

Contributor

All, this is fixed in 1.10 as we no longer use this sqlite db.

Contributor

cpuguy83 commented Feb 2, 2016

All, this is fixed in 1.10 as we no longer use this sqlite db.

@Cohedrin

This comment has been minimized.

Show comment
Hide comment
@Cohedrin

Cohedrin Feb 7, 2016

Still having this happen, albeit more rarely, on 1.10. Is there any information I could give that could help debug this? @cpuguy83

Basic information:
docker info

Containers: 92
 Running: 91
 Paused: 0
 Stopped: 1
Images: 172
Server Version: 1.10.0
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-2.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 78.64 GiB
Name: arizona5.kickback.gs.securedservers.com
ID: 4UIZ:P5HD:NPGR:OTRS:SEIX:KYXN:SMXA:3ZTT:OSYK:2GA2:JLZG:MO3F
Username: trianglegs
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

error:

'Conflict. The name "/server-1454829848" is already in use by container 3992d274d9f7673490606a245cc4145481c54bcf21d37258009806b453405441. You have to remove (or rename) that container to be able to reuse that name.

EDIT This seems to be able to be (somewhat) reliably reproduced by starting many containers at a time. In this case I'm creating and starting 92 containers. Roughly every 4 full starts of ~100 containers will cause at least 1 container to run into a conflict issue, but this could just be coincidence.

Cohedrin commented Feb 7, 2016

Still having this happen, albeit more rarely, on 1.10. Is there any information I could give that could help debug this? @cpuguy83

Basic information:
docker info

Containers: 92
 Running: 91
 Paused: 0
 Stopped: 1
Images: 172
Server Version: 1.10.0
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-2.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 78.64 GiB
Name: arizona5.kickback.gs.securedservers.com
ID: 4UIZ:P5HD:NPGR:OTRS:SEIX:KYXN:SMXA:3ZTT:OSYK:2GA2:JLZG:MO3F
Username: trianglegs
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

error:

'Conflict. The name "/server-1454829848" is already in use by container 3992d274d9f7673490606a245cc4145481c54bcf21d37258009806b453405441. You have to remove (or rename) that container to be able to reuse that name.

EDIT This seems to be able to be (somewhat) reliably reproduced by starting many containers at a time. In this case I'm creating and starting 92 containers. Roughly every 4 full starts of ~100 containers will cause at least 1 container to run into a conflict issue, but this could just be coincidence.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Feb 7, 2016

Contributor

@Cohedrin this is not the same issue.
Please open a new issue with details about how to reproduce.

Contributor

cpuguy83 commented Feb 7, 2016

@Cohedrin this is not the same issue.
Please open a new issue with details about how to reproduce.

@Cohedrin

This comment has been minimized.

Show comment
Hide comment

Cohedrin commented Feb 7, 2016

will do

@estevaoam

This comment has been minimized.

Show comment
Hide comment
@estevaoam

estevaoam Feb 10, 2016

It's impressive how every time docker gets updated, it solves an issue and creates ten more.
It's a +1.0 tool that feels like beta. Sad.

I got this bug while trying to switch over to overlayfs, because aufs was taking so much space that my 500GB server got at a point that I couldn't do a simple ls.

It's impressive how every time docker gets updated, it solves an issue and creates ten more.
It's a +1.0 tool that feels like beta. Sad.

I got this bug while trying to switch over to overlayfs, because aufs was taking so much space that my 500GB server got at a point that I couldn't do a simple ls.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Feb 10, 2016

Contributor

@estevaoam As you can see this issue is resolved.

Contributor

cpuguy83 commented Feb 10, 2016

@estevaoam As you can see this issue is resolved.

@mikeatlas

This comment has been minimized.

Show comment
Hide comment
@mikeatlas

mikeatlas Feb 10, 2016

@estevaoam at least in terms of semantic versioning, the major/minor/build version number does not imply that any product is bug-free... particularly free, OSS ones. It is indeed rapidly evolving at an insane pace; if you need stability in your functionality, lock down a working version for your environment, or opt into paid support solutions.

As a side note, if you've got image bloat filling your disk, make sure you're running images with --rm and not accumulating a bunch of unused instances of images.

@estevaoam at least in terms of semantic versioning, the major/minor/build version number does not imply that any product is bug-free... particularly free, OSS ones. It is indeed rapidly evolving at an insane pace; if you need stability in your functionality, lock down a working version for your environment, or opt into paid support solutions.

As a side note, if you've got image bloat filling your disk, make sure you're running images with --rm and not accumulating a bunch of unused instances of images.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 11, 2016

Member

Thanks @mikeatlas ❤️

Member

thaJeztah commented Feb 11, 2016

Thanks @mikeatlas ❤️

@mikeatlas

This comment has been minimized.

Show comment
Hide comment
@mikeatlas

mikeatlas Feb 11, 2016

Welcome @thaJeztah.... but @estevaoam's perspective is definitely not invalid either; many people have to decide whether or not some technology is mature enough for wide-scale adoption into organizations; Docker's pace (evolution, I suppose? ... 1 giant step forward, 3 new major bugs backward) is indeed enough of a frightening prospect to pitch as a viable solution if his/her task is to evaluate the readiness of using it as a reliable day-to-day tool (or not, perhaps in his case).

Welcome @thaJeztah.... but @estevaoam's perspective is definitely not invalid either; many people have to decide whether or not some technology is mature enough for wide-scale adoption into organizations; Docker's pace (evolution, I suppose? ... 1 giant step forward, 3 new major bugs backward) is indeed enough of a frightening prospect to pitch as a viable solution if his/her task is to evaluate the readiness of using it as a reliable day-to-day tool (or not, perhaps in his case).

@mikeatlas

This comment has been minimized.

Show comment
Hide comment
@mikeatlas

mikeatlas Feb 11, 2016

Anyway that's kinda OT from the OP, but it does underscore perhaps a need for more comprehensive migration testing even for minor releases. I do believe many install Docker via apt on Ubuntu and just bumped the Docker apt package up to 1.10, and as such many people automatically got upgraded and thus should expect a minor version (according to semver rules anways) to not introduce any breaking changes in functionality (and ideally no new bugs either).

Anyway that's kinda OT from the OP, but it does underscore perhaps a need for more comprehensive migration testing even for minor releases. I do believe many install Docker via apt on Ubuntu and just bumped the Docker apt package up to 1.10, and as such many people automatically got upgraded and thus should expect a minor version (according to semver rules anways) to not introduce any breaking changes in functionality (and ideally no new bugs either).

@estevaoam

This comment has been minimized.

Show comment
Hide comment
@estevaoam

estevaoam Feb 11, 2016

@mikeatlas thank you for your polite answer. Although my message could be a little bit harsh, it was because the tiredness of issues Docker-related in my environment.

Don't get me wrong, Docker made our application deployment process and management a gazillion times easier, but it seriously lack a more mature version where I could rely to run in production.

I actually use Docker in production for multiple projects, and it works well in a very basic configuration. My suggestion would be in maintaining a LTS version, focusing in more stability than new features. As Docker is a project with several years of development, we should have a production-ready and stable version, which I didn't found since the beginning.

About my size issue, it was some issue with device mapper, aufs and volumes not being deleted correctly. Now that we switched over to overlayfs I ran in this current issue, that @cpuguy83 said is solved in 1.10.

But I'm afraid to update, as I saw many issues opened on this repository addressing CPU-related issues. I don't want to upgrade and get other issues which I'll have to waste time trying to figure out the solution.

Thanks again for taking the time to read my comment and trying to understand my point.

@mikeatlas thank you for your polite answer. Although my message could be a little bit harsh, it was because the tiredness of issues Docker-related in my environment.

Don't get me wrong, Docker made our application deployment process and management a gazillion times easier, but it seriously lack a more mature version where I could rely to run in production.

I actually use Docker in production for multiple projects, and it works well in a very basic configuration. My suggestion would be in maintaining a LTS version, focusing in more stability than new features. As Docker is a project with several years of development, we should have a production-ready and stable version, which I didn't found since the beginning.

About my size issue, it was some issue with device mapper, aufs and volumes not being deleted correctly. Now that we switched over to overlayfs I ran in this current issue, that @cpuguy83 said is solved in 1.10.

But I'm afraid to update, as I saw many issues opened on this repository addressing CPU-related issues. I don't want to upgrade and get other issues which I'll have to waste time trying to figure out the solution.

Thanks again for taking the time to read my comment and trying to understand my point.

@mikeatlas

This comment has been minimized.

Show comment
Hide comment
@mikeatlas

mikeatlas Feb 13, 2016

@estevaoam @thaJeztah one of my favorite quotes ever, from a joke Twitter (HorseJS):
"open source is letting people down that you’ve never met"

@estevaoam @thaJeztah one of my favorite quotes ever, from a joke Twitter (HorseJS):
"open source is letting people down that you’ve never met"

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 13, 2016

Member

I actually use Docker in production for multiple projects, and it works well in a very basic configuration. My suggestion would be in maintaining a LTS version, focusing in more stability than new features. As Docker is a project with several years of development, we should have a production-ready and stable version, which I didn't found since the beginning.

Docker, Inc. offers a commercially supported version of docker ("Docker CS"), https://www.docker.com/products/overview, that you may be interested in.

Member

thaJeztah commented Feb 13, 2016

I actually use Docker in production for multiple projects, and it works well in a very basic configuration. My suggestion would be in maintaining a LTS version, focusing in more stability than new features. As Docker is a project with several years of development, we should have a production-ready and stable version, which I didn't found since the beginning.

Docker, Inc. offers a commercially supported version of docker ("Docker CS"), https://www.docker.com/products/overview, that you may be interested in.

southerngs added a commit to southerngs/docker that referenced this issue Mar 2, 2016

Attempt to solve error "Could not find container for entity id"
See GitHub issue: moby#17691
for details.

Solution is in: moby#16032

This commit cherry-picks 0f9f995
and applies it to the cr-defunct branch.

Build names and links at runtime

Don't rely on sqlite db for name registration and linking.
Instead register names and links when the daemon starts to an in-memory
store.
@Yajo

This comment has been minimized.

Show comment
Hide comment
@Yajo

Yajo Mar 23, 2016

No solution worked for me, and 1.10 is not packaged yet for Fedora. I had to:

# rm -Rf /var/lib/docker/*
# lvm lvremove system/docker-pool
# systemctl restart docker-storage-setup.service docker.service

And wait for all to be downloaded and built again. 😢

Yajo commented Mar 23, 2016

No solution worked for me, and 1.10 is not packaged yet for Fedora. I had to:

# rm -Rf /var/lib/docker/*
# lvm lvremove system/docker-pool
# systemctl restart docker-storage-setup.service docker.service

And wait for all to be downloaded and built again. 😢

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 23, 2016

Member

@Yajo the official docker packages for Fedora have 1.10.3; https://docs.docker.com/engine/installation/linux/fedora/

Member

thaJeztah commented Mar 23, 2016

@Yajo the official docker packages for Fedora have 1.10.3; https://docs.docker.com/engine/installation/linux/fedora/

@gdubicki

This comment has been minimized.

Show comment
Hide comment
@gdubicki

gdubicki Mar 31, 2016

For me a simple restart of docker service helped.

UPDATE: ..or sometimes I also had to do rm -Rf /var/lib/docker/* before (thanks, @Yajo !)

gdubicki commented Mar 31, 2016

For me a simple restart of docker service helped.

UPDATE: ..or sometimes I also had to do rm -Rf /var/lib/docker/* before (thanks, @Yajo !)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment