one container no longer updated

[jorisvandenberg]

Hi community,

Recently we noticed one of our containers is no longer automatically updated by watchtower. We can manually upgrade using docker-compose tough. All other containers get updated without any problems.

I modified watchtower's docker-compose.yml to enable tracelogs and i see the following:

watchtower    | time="2022-11-25T14:09:59Z" level=debug msg="Trying to load authentication credentials." container=/changedetection image="ghcr.io/dgtlmoon/changedetection.io:latest"
watchtower    | time="2022-11-25T14:09:59Z" level=debug msg="No credentials for ghcr.io found" config_file=/config.json
watchtower    | time="2022-11-25T14:09:59Z" level=debug msg="Got image name: ghcr.io/dgtlmoon/changedetection.io:latest"
watchtower    | time="2022-11-25T14:09:59Z" level=debug msg="Checking if pull is needed" container=/changedetection image="ghcr.io/dgtlmoon/changedetection.io:latest"
watchtower    | time="2022-11-25T14:09:59Z" level=debug msg="Building challenge URL" URL="https://ghcr.io/v2/"
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:user/image:pull\"" **status="401 Unauthorized"**
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="Checking challenge header content" realm="https://ghcr.io/token" service=ghcr.io
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="Setting scope for auth token" image=ghcr.io/dgtlmoon/changedetection.io scope="repository:ghcr.io/dgtlmoon/changedetection.io:pull"
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="No credentials found."
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="Parsing image ref" host=ghcr.io image=dgtlmoon/changedetection.io normalized="ghcr.io/dgtlmoon/changedetection.io:latest" tag=latest
watchtower    | time="2022-11-25T14:10:00Z" level=trace msg="Setting request token" token="Bearer <DELETED>"
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://ghcr.io/v2/dgtlmoon/changedetection.io/manifests/latest"
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="Found a remote digest to compare with" remote="sha256:d25de0cd2119edd5e3f615a03bc40798ed2457e442ccb8b17c9ace78ac06cf44"
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg=Comparing local="sha256:d25de0cd2119edd5e3f615a03bc40798ed2457e442ccb8b17c9ace78ac06cf44" remote="sha256:d25de0cd2119edd5e3f615a03bc40798ed2457e442ccb8b17c9ace78ac06cf44"
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="Found a match"
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="No pull needed. Skipping image."
watchtower    | time="2022-11-25T14:10:00Z" level=debug msg="No new images found for /changedetection"

I already tried to provide login credentials for ghcr.io by using an auth token, cat ~/.docker/config.json shows:

{
    "auths": {
        "ghcr.io": {
            "auth": "<REMOVED>"
         }
    }
}

[piksel]

When asking the repository for the latest version, it responds with a hash that matches the current image, so no update is performed. Are you sure that the image is really updated?

[jorisvandenberg]

Hi, Thanks for the fast reply.

You are correct, we did indeed update our container manually, so the output i have shown was from a container that didn't have an update available. I actually forgot to mention the output was exactly the same as when there was, in fact, a container that needed updating.

I have manually downgraded changedetection to version 0.39.21 (while currently the latest version is 0.39.22.1) and then retried the procedure.
Here's the output:

watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Trying to load authentication credentials." container=/changedetection image="ghcr.io/dgtlmoon/changedetection.io:0.39.21"
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="No credentials for ghcr.io found" config_file=/config.json
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Got image name: ghcr.io/dgtlmoon/changedetection.io:0.39.21"
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Checking if pull is needed" container=/changedetection image="ghcr.io/dgtlmoon/changedetection.io:0.39.21"
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Building challenge URL" URL="https://ghcr.io/v2/"
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Got response to challenge request" header="Bearer realm="https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:user/image:pull\"" status="401 Unauthorized"
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Checking challenge header content" realm="https://ghcr.io/token" service=ghcr.io
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Setting scope for auth token" image=ghcr.io/dgtlmoon/changedetection.io scope="repository:ghcr.io/dgtlmoon/changedetection.io:pull"
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="No credentials found."
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Parsing image ref" host=ghcr.io image=dgtlmoon/changedetection.io normalized="ghcr.io/dgtlmoon/changedetection.io:0.39.21" tag=0.39.21
watchtower | time="2022-11-28T09:47:32Z" level=trace msg="Setting request token" token="Bearer "
watchtower | time="2022-11-28T09:47:32Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://ghcr.io/v2/dgtlmoon/changedetection.io/manifests/0.39.21"
watchtower | time="2022-11-28T09:47:33Z" level=debug msg="Found a remote digest to compare with" remote="sha256:a40044d293f3dab72446695577db4c609bf5ab0ad62f9bc95bd04bb6b5601a51"
watchtower | time="2022-11-28T09:47:33Z" level=debug msg=Comparing local="sha256:a40044d293f3dab72446695577db4c609bf5ab0ad62f9bc95bd04bb6b5601a51" remote="sha256:a40044d293f3dab72446695577db4c609bf5ab0ad62f9bc95bd04bb6b5601a51"
watchtower | time="2022-11-28T09:47:33Z" level=debug msg="Found a match"
watchtower | time="2022-11-28T09:47:33Z" level=debug msg="No pull needed. Skipping image."
watchtower | time="2022-11-28T09:47:33Z" level=debug msg="No new images found for /changedetection"

So, even when there's an update available, i see the local and remote checksums matching.

Kind regards,
Joris

[piksel]

What do you mean by "downgrade"? Watchtower can only check what the hash of the container image is and compare it to what is the latest one in repository. It has no notion of what versions of the software running inside the container is beyond that.

[jorisvandenberg]

Hi,

Thanks for the fast reply. I'm not really a docker specialist, nor a watchtower specialist, so i hope i'm not asking dumb questions.

I was under the (wrong) impression watchtower also verified which image was currently in use, and also restarted the container if a newer image was available locally.

To test further, i have now cleaned up my local images, removed the latest image of changedetection, so only the image of version 0.39.21 was available:

docker image list --all
REPOSITORY TAG IMAGE ID CREATED SIZE
portainer/portainer-ce latest 5f11582196a4 7 days ago 287MB
containrrr/watchtower latest d6b5c7ec581c 3 weeks ago 14.5MB
ghcr.io/dgtlmoon/changedetection.io 0.39.21 7b1cf5e652e8 3 weeks ago 604MB
selenium/standalone-chrome-debug 3.141.59 aaf43463c572 14 months ago 1.11GB

I have verified the image used by my running container docker inspect changedetection | grep Image

    "Image": "sha256:7b1cf5e652e83a9a3d0468f32af6a172ab1eaf4916bbf5ecfabbf0d3f12830a2",
        "Image": "ghcr.io/dgtlmoon/changedetection.io:0.39.21",

Then i stopped watchtower, added "--trace --interval 20" to the parameters and started the watchtower container in the foreground. My output from changedetection was:

watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Trying to load authentication credentials." container=/changedetection image="ghcr.io/dgtlmoon/changedetection.io:0.39.21"
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="No credentials for ghcr.io found" config_file=/config.json
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Got image name: ghcr.io/dgtlmoon/changedetection.io:0.39.21"
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Checking if pull is needed" container=/changedetection image="ghcr.io/dgtlmoon/changedetection.io:0.39.21"
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Building challenge URL" URL="https://ghcr.io/v2/"
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Got response to challenge request" header="Bearer realm="https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:user/image:pull\"" status="401 Unauthorized"
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Checking challenge header content" realm="https://ghcr.io/token" service=ghcr.io
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Setting scope for auth token" image=ghcr.io/dgtlmoon/changedetection.io scope="repository:ghcr.io/dgtlmoon/changedetection.io:pull"
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="No credentials found."
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Parsing image ref" host=ghcr.io image=dgtlmoon/changedetection.io normalized="ghcr.io/dgtlmoon/changedetection.io:0.39.21" tag=0.39.21
watchtower | time="2022-11-28T10:25:15Z" level=trace msg="Setting request token" token="Bearer djE6Z2hjci5pby9kZ3RsbW9vbi9jaGFuZ2VkZXRlY3Rpb24uaW86MTY2OTYzMTExNTg1NTA4OTg2NA=="
watchtower | time="2022-11-28T10:25:15Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://ghcr.io/v2/dgtlmoon/changedetection.io/manifests/0.39.21"
watchtower | time="2022-11-28T10:25:16Z" level=debug msg="Found a remote digest to compare with" remote="sha256:a40044d293f3dab72446695577db4c609bf5ab0ad62f9bc95bd04bb6b5601a51"
watchtower | time="2022-11-28T10:25:16Z" level=debug msg=Comparing local="sha256:a40044d293f3dab72446695577db4c609bf5ab0ad62f9bc95bd04bb6b5601a51" remote="sha256:a40044d293f3dab72446695577db4c609bf5ab0ad62f9bc95bd04bb6b5601a51"
watchtower | time="2022-11-28T10:25:16Z" level=debug msg="Found a match"
watchtower | time="2022-11-28T10:25:16Z" level=debug msg="No pull needed. Skipping image."
watchtower | time="2022-11-28T10:25:16Z" level=debug msg="No new images found for /changedetection"

So, eventough the local image is older than the latest image (which i removed from my local system), it still says the sha256 checksum locally is equal to the checksum remotely.

Am i missing something else?

Kind regards,
Joris

[piksel]

Watchtower checks for the the latest version of the image using the same tag. So if you are explicitly using the version tag 0.39.21 it will never see an update when a new version is released (since that will be released under the new versions tag instead).
Instead, what you probably want to do is to use the image ghcr.io/dgtlmoon/changedetection.io:latest which should be updated whenever a new version is released.

The image you are using (with the explicit version in the tag) matches the hash in the log above, and was last updated 25 days ago:
https://github.com/dgtlmoon/changedetection.io/pkgs/container/changedetection.io/48523251?tag=0.39.21

[jorisvandenberg]

Thanks for getting back to me... I now have a better understanding of how watchtower works. For now, i'll mark this one as resolved (or, better, "works as designed") 👍