WTF with docker engine 1.9.0 and docker compose 1.5.0

[gionn]

This was my previously working docker-compose.yml:

broker:
  image: cloudesire/activemq
  ports:
    - "61616:61616"
    - "8161:8161"
pg:
  image: postgres:9.3
  ports:
    - "5432:5432"
  volumes:
    - "/run/shm/pg:/var/lib/postgresql/data"
mongo:
  image: mongo:3.0
  command: mongod --smallfiles
  ports:
    - "27017:27017"
  volumes:
    - "/run/shm/mongo:/data/db"

Today I've upgraded to docker engine 1.9.0 and docker compose 1.5.0 and I am getting a very strange error on compose up:

Creating cloudesire_broker_1
ERROR: Could not find container for entity id 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396

docker.log contains:

INFO[2253] GET /v1.19/images/cloudesire%2Factivemq/json 
INFO[2253] POST /v1.19/containers/create?name=cloudesire_broker_1 
ERRO[2253] Handler for POST /v1.19/containers/create returned error: Could not find container for entity id 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396 
ERRO[2253] HTTP Error                                    err=Could not find container for entity id 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396 statusCode=500

No container get created, however I can manually create the container without composer, e.g.:

docker run -d -p "5432:5432" -v "/run/shm/pg:/var/lib/postgresql/data" postgres:9.3

docker info

Containers: 1
Images: 37
Server Version: 1.9.0
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-31-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 7.372 GiB
Name: gt-xps
ID: 2562:2TQM:TR3I:CDFJ:T6JP:WXBN:AV4F:LZYT:HIAX:ZFYC:MMWD:Q7AN
Username: gionn
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

[dnephin]

The difference between your docker command and what compose runs is the lack of a container name. If you add --name cloudesire_broker_1 I suspect you'll get the same error from docker.

The error looks to be coming from GetByName (https://github.com/docker/docker/blob/944ea3134d/daemon/daemon.go#L511-L526), which I know was changed in 1.9.0 to remove a race condition.

Could the output of docker inspect cloudesire_broker_1 to this issue so we can open a proper issue on the docker issue tracker?

After that, I think you can work around the problem by running docker-compose rm.

[gionn]

Hello @dnephin,

you are partially right, got the same error running:

docker run --rm -it --name cloudesire_broker_1 ubuntu
Error response from daemon: Could not find container for entity id 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396

but after that error, any container has been created, so inspect can't works.

[dnephin]

So if you run inspect you get the error about "No such image or container: cloudesire_broker_1" ?

What happens if you docker inspect 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396 ?

[gionn]

Correct.

$ docker-compose up
Pulling broker (cloudesire/activemq:latest)...
latest: Pulling from cloudesire/activemq
c63fb41c2213: Pull complete
99fcaefe76ef: Pull complete
5a4526e952f0: Pull complete
1d073211c498: Pull complete
3569f5e0820a: Pull complete
6b513a1c775c: Pull complete
a03da997d240: Pull complete
01c4d69a6b7f: Pull complete
462e6ce9edc7: Pull complete
7c1a092ac217: Pull complete
bcfbb2708800: Pull complete
ae19bfb1617c: Pull complete
74c6c3b3cfbc: Pull complete
2d7e09a3927d: Pull complete
4540c382f130: Pull complete
bf42fd4e1058: Pull complete
Digest: sha256:9da0d20485a551a71573af073c18280d687b272f3b010f89ecddb4c0a08c8ccb
Status: Downloaded newer image for cloudesire/activemq:latest
Creating cloudesire_broker_1
ERROR: Could not find container for entity id 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396

$ docker inspect cloudesire_broker_1
Error: No such image or container: cloudesire_broker_1
[]

$ docker inspect 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396
Error: No such image or container: 72845af4d9964d42eab55fd84c1a2cc25d7b32b8c587ed47b766d9a7a8fe5396
[]

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

[gionn]

I've work-arounded by changing aliases of every defined container in my docker-compose.yml.

I've a feeling that has something to do with containers that was existing before the upgrade to 1.9.0 from 1.8.3, with the contextual switch from storage driver aufs to overlay.

[dnephin]

That sounds likely. Would you mind opening an issue at https://github.com/docker/docker/issues/new with all of this info (docker version, the inspect output, the docker run command, and the note about upgrading and changing from aufs to overlay) ?

And please cc me. I think the fix will need to happen in docker/docker.

[gionn]

👍

[sergio-bershadsky]

👍