[BUG] Compose v2 is needlessly recreating containers when NO changes have been made

[captainfalcon23]

Description

Separating out an issue described on #9600

[BUG] Compose v2 is needlessly recreating containers when NO changes have been made. Towards the end of the troubleshooting, you will see I make a change to an image tag and run "docker compose up -d" multiple times, and it recreates all the containers every time.

I have spent some more time troubleshooting this.

Using this docker-compose.yml. Note that no services rely on image3:

services:
 image1:
  image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image1:3.24-pdns-815"
  restart: always
  env_file:
    - .env_image1
  network_mode: host
 image2:
  image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image2:4.5.10-pdns-817"
  restart: always
  env_file:
    - .env_image2
  network_mode: host
  depends_on:
    - image1
    - image4
 image3:
  image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image3:1.7.0-pdns-818"
  restart: always
  env_file:
    - .env_image3
  network_mode: host
  depends_on:
    - image2
    - image4
 image4:
  image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image4:1.1.0-15-pdns-816"
  restart: always
  env_file:
    - .env_image4
  network_mode: host

Performed the following:

[root@dockerhost  ]# docker compose up -d
[root@dockerhost  ]# docker compose config --hash "*"
image3 ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6
image1 26f81a297463e0ffc24ba98a820c26a3543ae6c4bdaab3341b245442d4a0dbc6
image4 6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8
image2 ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57

All the hashes are correct:

[root@dockerhost ~]# docker inspect  -image3-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6

[root@dockerhost ~]# docker inspect  -image1-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
26f81a297463e0ffc24ba98a820c26a3543ae6c4bdaab3341b245442d4a0dbc6

[root@dockerhost ~]# docker inspect  -image4-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8

[root@dockerhost ~]# docker inspect  -image2-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57

Then, I changed the image tag of image1:

[root@dockerhost ~]# docker compose up -d

[root@dockerhost  ]# docker compose config --hash "*"
image2 ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57
image3 ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6
image1 5615a16902d9d1af30efcf431c17a683205d8b4d5c6d39a17677cf90bce5dce5 < -------------------- This changed as expected
image4 6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8
[root@dockerhost  ]#


[root@dockerhost  ]# docker inspect  image3-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6

[root@dockerhost  ]# docker inspect  image1-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
5615a16902d9d1af30efcf431c17a683205d8b4d5c6d39a17677cf90bce5dce5

[root@dockerhost  ]# docker inspect  image4-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8 < -------------------- This changed as expected

[root@dockerhost  ]# docker inspect  image2-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57

Only image1-1 hash changed (which is expected), however, all other containers are recreated (except image3, again noting none of the other services depend on it).

I have tested this on compose 2.16.0, 2.15.1 and even 2.0.1 and all have the same behviour.

I retested using the latest version of compose v1 i.e 1.29.2 and the issue is NOT present, so it clearly it something to do with how v2 handles dependant images:

[root@dockerhost  ]# docker-compose up -d
Creating image3 ... done
Creating image1      ... done
Creating image2  ... done
Creating image4        ... done


# change image tag of  image1-1


[root@dockerhost  ]# docker-compose up -d
image3 is up-to-date
Recreating image1 ... done <------------------------------------ only this container got changed (as expected)
image2 is up-to-date
image4 is up-to-date

Then, I went back to 2.16.0 to do some more testing:

Using compose 2.16.0:

[root@dockerhost  ]# docker compose up -d
[+] Running 4/0
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠿ Container image4-1       Running                                                                                           0.0s
 ⠿ Container image2-1   Running                                                                                           0.0s
 ⠿ Container image1-1         Running                                                                                           0.0s
 

[root@dockerhost  ]# docker compose up -d
[+] Running 4/0
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠿ Container image4-1       Running                                                                                           0.0s
 ⠿ Container image2-1   Running                                                                                           0.0s
 ⠿ Container image1-1         Running                                                                                           0.0s
 
 
[root@dockerhost  ]# docker compose up -d
[+] Running 4/0
 ⠿ Container image4-1       Running                                                                                           0.0s
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠿ Container image2-1   Running                                                                                           0.0s
 ⠿ Container image1-1         Running             
 

Made no changes, and looks good, no action taken. And no changes after running the command 3 times in a row.

Again, modified the image tag of 1 of the images:

[root@dockerhost  ]# docker compose up -d
[+] Running 4/4
 ⠿ Container image4-1       Started                                                                                          21.0s < -------------- got recreated
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠿ Container image2-1   Started                                                                                          20.9s < -------------- got recreated
 ⠿ Container image1-1         Started                                                                                          10.8s < -------------- got recreated
 
 ********<I did NOT make any changes here, I simply ran compose up again, and again it recreated the containers! And yes, they took EXACTLY the same amount of time to create>***********

 [root@dockerhost  ]# docker compose up -d
[+] Running 4/4
 ⠿ Container image4-1       Started                                                                                          21.0s < -------------- got recreated
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠿ Container image2-1   Started                                                                                          20.9s < -------------- got recreated
 ⠿ Container image1-1         Started                                                                                          10.8s < -------------- got recreated

Tried it a few more times, and it still recreates the containers:

[root@dockerhost  ]# docker compose up -d
[+] Running 4/4
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠿ Container image4-1       Started                                                                                          21.2s
 ⠿ Container image2-1   Started                                                                                          21.1s
 ⠿ Container image1-1         Started                                                                                          10.7s
[root@dockerhost  ]# docker compose up -d
[+] Running 2/3
 ⠿ Container image4-1       Recreated                                                                                         0.2s
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠧ Container image2-1   Recreate                                                                                          2.7s

[root@dockerhost  ]# docker compose config --hash "*"
XXX 3aa471602b2be184bbc2f78ba579c08e841e92d488f6dc6c499717c512bf0636
XXX 20b943ef0b80c65fcbba74cfdcc8c9ca8758cd7533a4f9383ca062ba7812a969
XXX 00765ce04ab7fd8166d8de909a69e366edd51f7bbf67390e7e69d998b5674b6b
XXX f364475068952084b4ff86358831ffd95e020ed84cd5e99b5d43871c61f5ba66
[root@dockerhost  ]# docker compose up -d
[+] Running 4/4
 ⠿ Container image4-1       Started                                                                                          21.3s
 ⠿ Container image3-1  Running                                                                                           0.0s
 ⠿ Container image2-1   Started                                                                                          21.2s
 ⠿ Container image1-1         Started                                                                                          10.7s
XXX 00765ce04ab7fd8166d8de909a69e366edd51f7bbf67390e7e69d998b5674b6b
XXX f364475068952084b4ff86358831ffd95e020ed84cd5e99b5d43871c61f5ba66
XXX 3aa471602b2be184bbc2f78ba579c08e841e92d488f6dc6c499717c512bf0636
XXX 20b943ef0b80c65fcbba74cfdcc8c9ca8758cd7533a4f9383ca062ba7812a969
[root@dockerhost  ]#

Steps To Reproduce

As per description.

Compose Version

Docker Compose version v2.16.0

Docker Environment

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.16.0)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.80.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.06GiB
 Name: XXX
 ID: GR4X:Q35F:5E4H:TGFY:O34K:JEYN:WYBW:3QY5:4G7N:TXP6:4GLE:QOG2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

Refer to #9600 for alot of history.

[ndeloof]

This regression was introduced in 2.16 and fixed by #10275
Can you please confirm v2.16.1 fixed it ?

[trim21]

This regression was introduced in 2.16 and fixed by #10275
Can you please confirm v2.16.1 fixed it ?

Sorry, there isn't a v2.16.1 tag or release...

[ndeloof]

My bad, next release is v2.17.0

[trim21]

My bad, next release is v2.17.0

can i expect a release date?

[ndeloof]

release candidate is available, I can't tell about final release date
Closing this issue as "fix is on his way"

[captainfalcon23]

As per my comment and testing here -> #10308 (comment) this looks resolved.