Containers created with `docker compose` cannot be updated #1019

UncleSamSwiss · 2021-07-08T13:09:45Z

Describe the bug
Using docker compose under Windows 10 with Docker in WSL2 leads to wrong image name being used, thus watchtower can't check for updates properly.

To Reproduce

Create a "regular" docker-compose.yml with some containers and add watchtower
Execute docker compose up (no hyphen!)
❌ Watchtower can't find the images (looking for the wrong name, see log for details)
Exit the above command and execute docker compose down
Execute docker-compose up (with hyphen!)
✔️ Everything works as expected
Exit the above command and execute docker-compose down

Expected behavior
Both docker-compose up and docker compose up should work the same way.

Logs
Below are extracts of the log file of both runs.

Using docker compose:

watchtower-ghcr_1       | time="2021-07-08T12:24:49Z" level=debug msg="Making sure everything is sane before starting"
watchtower-ghcr_1       | time="2021-07-08T12:24:49Z" level=debug msg="Retrieving running containers"
watchtower-ghcr_1       | time="2021-07-08T12:24:49Z" level=debug msg="There are no additional watchtower containers"
watchtower-ghcr_1       | time="2021-07-08T12:24:49Z" level=debug msg="Watchtower HTTP API skipped."
watchtower-ghcr_1       | time="2021-07-08T12:24:49Z" level=info msg="Watchtower 1.3.0\nUsing no notifications\nOnly checking containers in scope \"ghcr\"\nScheduling first run: 2021-07-08 12:25:19 +0000 UTC\nNote that the first check will be performed in 29 seconds"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Checking containers for updated images"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Retrieving running containers"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Trying to load authentication credentials." container=/weblate-docker-compose_weblate_1 image="sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Loaded auth credentials for user UncleSamSwiss on registry sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Got image name: sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Credentials loaded"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Checking if pull is needed" container=/weblate-docker-compose_weblate_1 image="sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Setting scope for auth token" image=sha256 scope="repository:library/sha256:pull"
watchtower-ghcr_1       | time="2021-07-08T12:25:19Z" level=debug msg="Credentials found."
watchtower-ghcr_1       | time="2021-07-08T12:25:22Z" level=debug msg="Parsing image ref" host=index.docker.io image=sha256 normalized="docker.io/library/sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8" tag=e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8
watchtower-ghcr_1       | time="2021-07-08T12:25:22Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/library/sha256/manifests/e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:22Z" level=warning msg="Could not do a head request for \"sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8\", falling back to regular pull." container=/weblate-docker-compose_weblate_1 image="sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:22Z" level=warning msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://auth.docker.io/token\\\",service=\\\"registry.docker.io\\\",scope=\\\"repository:library/sha256:pull\\\"\"" container=/weblate-docker-compose_weblate_1 image="sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:22Z" level=debug msg="Pulling image" container=/weblate-docker-compose_weblate_1 image="sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
watchtower-ghcr_1       | time="2021-07-08T12:25:27Z" level=debug msg="Error pulling image sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8, Error response from daemon: Head https://registry-1.docker.io/v2/library/sha256/manifests/e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8: unauthorized: incorrect username or password"
watchtower-ghcr_1       | time="2021-07-08T12:25:27Z" level=info msg="Unable to update container \"/weblate-docker-compose_weblate_1\": Error response from daemon: Head https://registry-1.docker.io/v2/library/sha256/manifests/e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8: unauthorized: incorrect username or password. Proceeding to next."
watchtower-ghcr_1       | time="2021-07-08T12:25:27Z" level=debug msg="Session done: 1 scanned, 0 updated, 1 failed"
watchtower-ghcr_1       | time="2021-07-08T12:25:27Z" level=debug msg="Scheduled next run: 2021-07-08 12:25:49 +0000 UTC"

Using docker-compose:

watchtower-ghcr_1       | time="2021-07-08T12:44:55Z" level=debug msg=ghcr
watchtower-ghcr_1       | time="2021-07-08T12:44:55Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
watchtower-ghcr_1       | time="2021-07-08T12:44:56Z" level=debug msg="Making sure everything is sane before starting"
watchtower-ghcr_1       | time="2021-07-08T12:44:56Z" level=debug msg="Retrieving running containers"
watchtower-ghcr_1       | time="2021-07-08T12:44:56Z" level=debug msg="There are no additional watchtower containers"
watchtower-ghcr_1       | time="2021-07-08T12:44:56Z" level=debug msg="Watchtower HTTP API skipped."
watchtower-ghcr_1       | time="2021-07-08T12:44:56Z" level=info msg="Watchtower 1.3.0\nUsing no notifications\nOnly checking containers in scope \"ghcr\"\nScheduling first run: 2021-07-08 12:45:26 +0000 UTC\nNote that the first check will be performed in 29 seconds"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Checking containers for updated images"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Retrieving running containers"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Trying to load authentication credentials." container=/weblate-docker-compose_weblate_1 image="ghcr.io/iobrokertranslator/weblate:master"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Loaded auth credentials for user UncleSamSwiss on registry ghcr.io/iobrokertranslator/weblate:master"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Got image name: ghcr.io/iobrokertranslator/weblate:master"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Credentials loaded"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Checking if pull is needed" container=/weblate-docker-compose_weblate_1 image="ghcr.io/iobrokertranslator/weblate:master"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Building challenge URL" URL="https://ghcr.io/v2/"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:user/image:pull\"" status="401 Unauthorized"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Checking challenge header content" realm="https://ghcr.io/token" service=ghcr.io
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Setting scope for auth token" image=ghcr.io/iobrokertranslator/weblate scope="repository:ghcr.io/iobrokertranslator/weblate:pull"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Credentials found."
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Parsing image ref" host=ghcr.io image=iobrokertranslator/weblate normalized="ghcr.io/iobrokertranslator/weblate:master" 
tag=master
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://ghcr.io/v2/iobrokertranslator/weblate/manifests/master"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Found a remote digest to compare with" remote="sha256:a3082489de4a3bd2690cdb2efd27f83ec76e716db9832eb0fc202664dcb3b2a5"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg=Comparing local="sha256:a3082489de4a3bd2690cdb2efd27f83ec76e716db9832eb0fc202664dcb3b2a5" remote="sha256:a3082489de4a3bd2690cdb2efd27f83ec76e716db9832eb0fc202664dcb3b2a5"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Found a match"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="No pull needed. Skipping image."
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="No new images found for /weblate-docker-compose_weblate_1"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Session done: 1 scanned, 0 updated, 0 failed"
watchtower-ghcr_1       | time="2021-07-08T12:45:26Z" level=debug msg="Scheduled next run: 2021-07-08 12:45:56 +0000 UTC"

Main differences:

  level=debug msg="Checking containers for updated images"
  level=debug msg="Retrieving running containers"
- level=debug msg="Trying to load authentication credentials." container=/weblate-docker-compose_weblate_1 image="sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
+ level=debug msg="Trying to load authentication credentials." container=/weblate-docker-compose_weblate_1 image="ghcr.io/iobrokertranslator/weblate:master"
- level=debug msg="Loaded auth credentials for user UncleSamSwiss on registry sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
+ level=debug msg="Loaded auth credentials for user UncleSamSwiss on registry ghcr.io/iobrokertranslator/weblate:master"
- level=debug msg="Got image name: sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
+ level=debug msg="Got image name: ghcr.io/iobrokertranslator/weblate:master"
  level=debug msg="Credentials loaded"
- level=debug msg="Checking if pull is needed" container=/weblate-docker-compose_weblate_1 image="sha256:e2dff46bca7b31539a043a3a42c714a21873d99ab92b9a865370479fd83122d8"
+ level=debug msg="Checking if pull is needed" container=/weblate-docker-compose_weblate_1 image="ghcr.io/iobrokertranslator/weblate:master"
- level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
+ level=debug msg="Building challenge URL" URL="https://ghcr.io/v2/"

Environment

Windows 10 Pro
x64
Docker version 20.10.7, build f0df350 (WSL2 backend)
docker-compose version 1.29.2, build 5becea4c

Additional context
My docker-compose.yml

version: "3"
services:
  weblate:
    image: ghcr.io/iobrokertranslator/weblate:master
    volumes:
      - weblate-data:/app/data
    env_file:
      - ./environment
    restart: unless-stopped
    depends_on:
      - database
      - cache
    labels:
      com.centurylinklabs.watchtower.scope: "ghcr"
  database:
    image: postgres:12-alpine
    env_file:
      - ./environment
    volumes:
      - postgres-data:/var/lib/postgresql/data
    restart: unless-stopped
  cache:
    image: redis:6-alpine
    restart: unless-stopped
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - redis-data:/data
  watchtower-ghcr:
    image: containrrr/watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --debug --cleanup --interval 30 --stop-timeout 30s --scope ghcr
volumes:
  weblate-data: {}
  postgres-data: {}
  redis-data: {}

Of course you need a docker-compose.override.yml, but I can't provide it here for obvious reasons (credentials).

The text was updated successfully, but these errors were encountered:

github-actions · 2021-07-08T13:10:30Z

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

piksel · 2021-07-08T15:56:38Z

So for some reason, docker container inspect does not contain the image name, just a sha256 hash, and only when using docker compose? That does seem strange, but watchtower can't do much about it.

My guess is that it has something to do with the Docker Desktop binary aliases 🤷‍♀️

piksel · 2021-07-08T16:17:29Z

If this actually gives different results depending on which of the docker(-| )composes used, it might be used as a report for a bug report to docker:

docker container inspect weblate-docker-compose_weblate_1 -f '       Image: {{ .Image }}{{"\n"}}Config.Image: {{.Config.Image}}'

(result should look similar to this:)

       Image: sha256:dd78a816fb764ac6f44bd30d599e3605ebce50bc987e627573ab495092561a69
Config.Image: containrrr/watchtower

If not, then the difference is somewhere deeper in the API, which may be much harder to report...

Update:
Tried it on my own windows desktop machine, and yeah, the image name is not contained in docker container inspect output at all, which means that watchtower cannot work with containers created by docker compose. 😩

Inspect output

$ docker container inspect kafka -f '       Image: {{ .Image }}{{"\n"}}Config.Image: {{.Config.Image}}'
       Image: sha256:d694642b6bef4693082bd6192d923f37af9e27ef15beef9de7d93a51d1a5d74d
Config.Image: sha256:d694642b6bef4693082bd6192d923f37af9e27ef15beef9de7d93a51d1a5d74d

UncleSamSwiss · 2021-07-08T18:29:33Z

With an additional intermediate step, you could still figure it out:
docker image ls --no-trunc shows the correct image with its ID in the given sha256:xxxx format.

I assume, you could use this to get the real image repo from the SHA ID.

piksel · 2021-07-09T11:20:40Z

Well, in cases where the image is only tagged once, but that is rarely the case with latest tags...

$ docker image ls | grep dd78a816fb76
containrrr/watchtower                               1.3.0                       dd78a816fb76   2 months ago   16.4MB
containrrr/watchtower                               latest                      dd78a816fb76   2 months ago   16.4MB

So we might end up switching to another tag...

P-Verbrugge · 2021-11-05T16:12:07Z

Having the same issue. I have installed all my docker images using stacks through Portainer. Some images can't be checked, while other can.

Running on RaspberryPi OS and with OpenMediaVault

piksel · 2021-11-05T16:55:53Z

See #1050 for the main issue about this.

SergeAx · 2023-05-06T17:04:54Z

It seems like it was fixed in https://github.com/docker/compose-cli/releases/tag/v1.0.18 by docker/compose-cli#2038

@UncleSamSwiss , can you check it out?

UncleSamSwiss · 2023-05-07T05:37:29Z

Thanks for the heads-up. Unfortunately I'm no longer using docker compose with watchtower as I moved to k8s. So I don't have an easy way to test this.

simskij · 2023-11-13T16:57:44Z

Closing this as I have a very hard time convincing myself that this is something Watchtower should even try to work around.

UncleSamSwiss added Priority: Medium Status: Available Type: Bug Unintended behavior labels Jul 8, 2021

amarcu5 mentioned this issue Aug 2, 2021

Compose V2 container Image attribute is inconsistent with V1 and prevents Watchtower docker/compose-cli#1983

Closed

piksel changed the title ~~Bug using docker compose on Win/WSL2~~ Containers created with docker compose cannot be updated Aug 8, 2021

piksel mentioned this issue Aug 8, 2021

Docker containers with hash-only image cannot be updated #1050

Closed

simskij closed this as completed Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Containers created with `docker compose` cannot be updated #1019

Containers created with `docker compose` cannot be updated #1019

UncleSamSwiss commented Jul 8, 2021 •

edited

github-actions bot commented Jul 8, 2021

piksel commented Jul 8, 2021 •

edited

piksel commented Jul 8, 2021 •

edited

UncleSamSwiss commented Jul 8, 2021

piksel commented Jul 9, 2021

P-Verbrugge commented Nov 5, 2021 •

edited

piksel commented Nov 5, 2021

SergeAx commented May 6, 2023

UncleSamSwiss commented May 7, 2023

simskij commented Nov 13, 2023

Containers created with docker compose cannot be updated #1019

Containers created with docker compose cannot be updated #1019

Comments

UncleSamSwiss commented Jul 8, 2021 • edited

github-actions bot commented Jul 8, 2021

piksel commented Jul 8, 2021 • edited

piksel commented Jul 8, 2021 • edited

UncleSamSwiss commented Jul 8, 2021

piksel commented Jul 9, 2021

P-Verbrugge commented Nov 5, 2021 • edited

piksel commented Nov 5, 2021

SergeAx commented May 6, 2023

UncleSamSwiss commented May 7, 2023

simskij commented Nov 13, 2023

Containers created with `docker compose` cannot be updated #1019

Containers created with `docker compose` cannot be updated #1019

UncleSamSwiss commented Jul 8, 2021 •

edited

piksel commented Jul 8, 2021 •

edited

piksel commented Jul 8, 2021 •

edited

P-Verbrugge commented Nov 5, 2021 •

edited