Error: No such image #1409

ptr727 · 2022-10-05T15:52:15Z

ptr727
Oct 5, 2022

Having some weird issues past couple days, email notification says Error: No such image, but there is no mention of this error or the image name in the logs?

Any idea what the problem may be?
Any idea why the error reported in email is not reported in the logs?

Email:

Found new netdata/netdata:latest image (695e46f8c5ac)
Found new falconchristmas/fpp:latest image (2c0e2b370fcf)
Could not do a head request for "[lscr.io/linuxserver/unifi-controller:latest](http://lscr.io/linuxserver/unifi-controller:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/unifi-controller:pull\""
Could not do a head request for "[lscr.io/linuxserver/qdirstat:latest](http://lscr.io/linuxserver/qdirstat:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/qdirstat:pull\""
Unable to update container "/plex": Error: No such image: linuxserver/plex:latest. Proceeding to next.
Could not do a head request for "[lscr.io/linuxserver/sonarr:latest](http://lscr.io/linuxserver/sonarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/sonarr:pull\""
Found new [lscr.io/linuxserver/sonarr:latest](http://lscr.io/linuxserver/sonarr:latest) image (161f3a681b5f)
Could not do a head request for "[lscr.io/linuxserver/radarr:latest](http://lscr.io/linuxserver/radarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/radarr:pull\""
Could not do a head request for "[lscr.io/linuxserver/radarr:latest](http://lscr.io/linuxserver/radarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/radarr:pull\""
Unable to update container "/heimdall": Error: No such image: linuxserver/heimdall:latest. Proceeding to next.
Unable to update container "/heimdall-public": Error: No such image: linuxserver/heimdall:latest. Proceeding to next.
Unable to update container "/sabnzbd": Error: No such image: linuxserver/sabnzbd:latest. Proceeding to next.
Could not do a head request for "[lscr.io/linuxserver/mariadb:latest](http://lscr.io/linuxserver/mariadb:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/mariadb:pull\""
Stopping /sonarr (c816f9e59da1) with SIGTERM
Stopping /fpp (a405489ffb2e) with SIGTERM
Stopping /netdata (426ddc7a6424) with SIGTERM
Creating /netdata
Creating /fpp
Creating /sonarr
Removing image 307a21df9c59
Removing image a617effcd4c2
Removing image ae0ff7acef07

Ansible configuration:

- name: "Install Watchtower"
  community.docker.docker_container:
    name: watchtower
    image: containrrr/watchtower:latest
    pull: yes
    hostname: "watchtower-{{ ansible_hostname }}"
    domainname: "{{ ansible_domain }}"
    restart_policy: unless-stopped
    # Run as root
    # user: "{{ user_id }}:{{ group_id }}"
    env:
      TZ: "{{ local_timezone }}"
      WATCHTOWER_NOTIFICATIONS: "email"
      WATCHTOWER_NOTIFICATION_EMAIL_FROM: "{{ smtp_from }}"
      WATCHTOWER_NOTIFICATION_EMAIL_TO: "{{ smtp_to }}"
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER: "{{ smtp_server }}"
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT: "{{ smtp_port }}"
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_INCLUDE_STOPPED: "true"
      WATCHTOWER_INCLUDE_RESTARTING: "true"
      WATCHTOWER_TIMEOUT: "90"
      WATCHTOWER_SCHEDULE: "0 0 2 * * *"
      WATCHTOWER_WARN_ON_HEAD_FAILURE: "always"
      WATCHTOWER_DEBUG: "true"
      # Only update containers that opt-in
      WATCHTOWER_LABEL_ENABLE: "true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - name: "{{ docker_local_network }}"
    # published_ports:
    # No ports published
    labels:
      traefik.enable: "false"
      com.centurylinklabs.watchtower.enable: "true"
    recreate: "{{ docker_container_recreate }}"

Log attached.
_watchtower_logs.1.log

piksel · 2022-10-05T17:02:12Z

piksel
Oct 5, 2022
Maintainer

It should be higher up in the logs. Since they contain debug messages as well, there is a bunch of log messages that is not included in the e-mail.

1 reply

ptr727 Oct 5, 2022
Author

That was the full log set that was available from portainer, is there a way to log to file, and a way to force an update (e.g. attach to container in shell)?

ptr727 · 2022-10-06T01:20:21Z

ptr727
Oct 6, 2022
Author

I figured out how to get more logs, attached.

Logs:

time="2022-10-05T02:00:40-07:00" level=debug msg="Trying to load authentication credentials." container=/heimdall image="linuxserver/heimdall:latest"
time="2022-10-05T02:00:40-07:00" level=debug msg="No credentials for linuxserver found" config_file=/config.json
time="2022-10-05T02:00:40-07:00" level=debug msg="Got image name: linuxserver/heimdall:latest"
time="2022-10-05T02:00:40-07:00" level=debug msg="Checking if pull is needed" container=/heimdall image="linuxserver/heimdall:latest"
time="2022-10-05T02:00:40-07:00" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-10-05T02:00:40-07:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-10-05T02:00:40-07:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-10-05T02:00:40-07:00" level=debug msg="Setting scope for auth token" image=linuxserver/heimdall scope="repository:linuxserver/heimdall:pull"
time="2022-10-05T02:00:40-07:00" level=debug msg="No credentials found."
time="2022-10-05T02:00:41-07:00" level=debug msg="Parsing image ref" host=index.docker.io image=linuxserver/heimdall normalized="docker.io/linuxserver/heimdall:latest" tag=latest
time="2022-10-05T02:00:41-07:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/linuxserver/heimdall/manifests/latest"
time="2022-10-05T02:00:41-07:00" level=debug msg="Found a remote digest to compare with" remote="sha256:8f6f464f7b8ec912a9aef176b7dfc4cefe234aaf0277a131398e8816eaca1456"
time="2022-10-05T02:00:41-07:00" level=debug msg=Comparing local="sha256:8f6f464f7b8ec912a9aef176b7dfc4cefe234aaf0277a131398e8816eaca1456" remote="sha256:8f6f464f7b8ec912a9aef176b7dfc4cefe234aaf0277a131398e8816eaca1456"
time="2022-10-05T02:00:41-07:00" level=debug msg="Found a match"
time="2022-10-05T02:00:41-07:00" level=debug msg="No pull needed. Skipping image."
time="2022-10-05T02:00:41-07:00" level=info msg="Unable to update container \"/heimdall\": Error: No such image: linuxserver/heimdall:latest. Proceeding to next."
time="2022-10-05T02:00:41-07:00" level=debug msg="Trying to load authentication credentials." container=/heimdall-public image="linuxserver/heimdall:latest"
time="2022-10-05T02:00:41-07:00" level=debug msg="No credentials for linuxserver found" config_file=/config.json
time="2022-10-05T02:00:41-07:00" level=debug msg="Got image name: linuxserver/heimdall:latest"
time="2022-10-05T02:00:41-07:00" level=debug msg="Checking if pull is needed" container=/heimdall-public image="linuxserver/heimdall:latest"
time="2022-10-05T02:00:41-07:00" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-10-05T02:00:41-07:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-10-05T02:00:41-07:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-10-05T02:00:41-07:00" level=debug msg="Setting scope for auth token" image=linuxserver/heimdall scope="repository:linuxserver/heimdall:pull"
time="2022-10-05T02:00:41-07:00" level=debug msg="No credentials found."
time="2022-10-05T02:00:41-07:00" level=debug msg="Parsing image ref" host=index.docker.io image=linuxserver/heimdall normalized="docker.io/linuxserver/heimdall:latest" tag=latest
time="2022-10-05T02:00:41-07:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/linuxserver/heimdall/manifests/latest"
time="2022-10-05T02:00:42-07:00" level=debug msg="Found a remote digest to compare with" remote="sha256:8f6f464f7b8ec912a9aef176b7dfc4cefe234aaf0277a131398e8816eaca1456"
time="2022-10-05T02:00:42-07:00" level=debug msg=Comparing local="sha256:8f6f464f7b8ec912a9aef176b7dfc4cefe234aaf0277a131398e8816eaca1456" remote="sha256:8f6f464f7b8ec912a9aef176b7dfc4cefe234aaf0277a131398e8816eaca1456"
time="2022-10-05T02:00:42-07:00" level=debug msg="Found a match"
time="2022-10-05T02:00:42-07:00" level=debug msg="No pull needed. Skipping image."
time="2022-10-05T02:00:42-07:00" level=info msg="Unable to update container \"/heimdall-public\": Error: No such image: linuxserver/heimdall:latest. Proceeding to next."

_watchtower_logs.2.log

0 replies

piksel · 2022-10-13T14:01:47Z

piksel
Oct 13, 2022
Maintainer

For some reason you have a container running with an image that docker does not have any information about. Since the image has the same hash as the latest version on docker hub, watchtower doesn't pull the image again (since it already knows that no update is needed).
This leads to a pretty confusing state where we both have an error (docker somehow not having the image that the container uses) and a verified latest version.

This would solve itself as soon as a new image is released, but to fix it right now I think this should work:

docker pull linuxserver/plex:latest

But then again, I have no idea how you ended up in this situation...

0 replies

ptr727 · 2022-10-13T14:39:50Z

ptr727
Oct 13, 2022
Author

Plex is one example, it seems to happen with several of the LSIO images.
The Plex image is reported as up to date, but is not in last nights error report.

Plex is reported as up to date:

pieter@server-1:~$ docker pull lscr.io/linuxserver/plex:latest
latest: Pulling from linuxserver/plex
Digest: sha256:d4f8c0f5c4796b4e00b0668f2c048dfc57a39b7626ccb74fed47f57bb086b22a
Status: Image is up to date for lscr.io/linuxserver/plex:latest
lscr.io/linuxserver/plex:latest

Last night's email error report says:

Unable to update container "/heimdall": Error: No such image: linuxserver/heimdall:latest. Proceeding to next.
Unable to update container "/heimdall-public": Error: No such image: linuxserver/heimdall:latest. Proceeding to next.

But Heimdal is also reported as up to date:

pieter@server-1:~$ docker pull lscr.io/linuxserver/heimdall:latest
latest: Pulling from linuxserver/heimdall
Digest: sha256:8f6f464f7b8ec912a9aef176b7dfc4cefe234aaf0277a131398e8816eaca1456
Status: Image is up to date for lscr.io/linuxserver/heimdall:latest
lscr.io/linuxserver/heimdall:latest

Seems that there is some other problem?
More peculiar is the fact the the logs are not showing the error, the log just reports No such image with no earlier errors. I am speculating, but as I noted in the issue I created that lscr.io seems to be backed by ghcr.io, and I have seen occasional 500 errors from ghcr.io. Maybe a 500 error that is not captured or logged, else some otehr failure that is not getting logged.

Attaching current trace logs.
_watchtower_logs.4.log

10 replies

piksel Oct 13, 2022
Maintainer

as the registry will be the same for all tags

Nope.

❯ docker inspect linuxserver/heimdall:latest | jq .[].RepoTags
[
  "linuxserver/heimdall:latest",
  "lscr.io/linuxserver/heimdall:latest"
]

watchtower should not assume images always come from docker.io when images are pulled from specific registries?

When started from docker or docker-compose, the container Config.Image contains the full image source, including the repository. Ansible seem to remove the repository from the Config.Image which not only makes it not possible to check for updates, but also to recreate the container from watchtower.

❯ docker run -d --name heimdall lscr.io/linuxserver/heimdall:latest
1da96a91b4bd4e0e1d583c9198d7c8938e7ffec5871b583fdddbe23dd35c9569

❯ docker inspect heimdall | jq .[].Config.Image
"lscr.io/linuxserver/heimdall:latest"

ptr727 Oct 13, 2022
Author

Hmm, ok.
To my knowledge Ansible uses the docker API to create images, I'll experiment between ansible and docker create and report back.

piksel Oct 13, 2022
Maintainer

Yeah, it uses the docker daemon API to create the container AFAICT.

I don't really understand how it happens, or if there is some other puzzle piece we're missing here.

Perhaps the create container API as allows for resolving images even when the repository is omitted...

ptr727 Oct 13, 2022
Author

I create a dummy container using ansible, and the image was correct:

pieter@server-1:~$ docker container inspect test | jq .[].Config.Image
"lscr.io/linuxserver/heimdall:latest"

This was different to what I see for my current heimdall.
I then changed the Ansible recreate flag to yes, and recreated the heimdall container, and now I see:

pieter@server-1:~$ docker container inspect heimdall | jq .[].Config.Image
"lscr.io/linuxserver/heimdall:latest"
pieter@server-1:~$ docker container inspect heimdall | jq .[].Image
"sha256:ee9db55d01f96029523b187a1592c35144f3d26c2f6017a3d9856c7c99e12af5"
pieter@server-1:~$ docker image inspect "sha256:ee9db55d01f96029523b187a1592c35144f3d26c2f6017a3d9856c7c99e12af5" | jq .[].RepoTags
[
  "lscr.io/linuxserver/heimdall:latest"
]

I think what happened is that I changed the LSIO images to pull from lscr.io when they changed their default hosting location, but I never recreated the containers, so there was a discrepancy between the Image name and the RepoTags name.

I'll keep an eye on tonight's update, but I expect it to succeed.

It would be good if the logging that produced the missing image without context as to why could be improved, and / or the logic test for consistency with RepoTags.

As FYI, my esphome looks like this, where there is only one RepoTags entry, that does match the Image:

pieter@server-1:~$ docker container inspect esphome | jq .[].Config.Image
"esphome/esphome:latest"
pieter@server-1:~$ docker container inspect esphome | jq .[].Image
"sha256:aa5b4e5153ffe11344430029accfa66088c80e012cc3200824c68a4162ff9714"
pieter@server-1:~$ docker image inspect "sha256:aa5b4e5153ffe11344430029accfa66088c80e012cc3200824c68a4162ff9714" | jq .[].RepoTags
[
  "esphome/esphome:latest"
]

piksel Oct 14, 2022
Maintainer

s FYI, my esphome looks like this ...

Yeah, the RepoTags is a list of all the names the docker daemon has tagged the image with (locally):

❯ docker image tag esphome/esphome:latest example.com/something/else:v2

❯ docker inspect esphome/esphome:latest | jq .[].RepoTags
[
  "esphome/esphome:2022.9",
  "esphome/esphome:latest",
  "esphome/esphome:stable",
  "example.com/something/else:v2"
]

Whenever you pull an image with a name, it will tag that image with the same name. If it already has the image hash, it will just be added to the list.

It would be good if the logging that produced the missing image without context as to why could be improved, and / or the logic test for consistency with RepoTags.

Yeah, we never encountered this before and didn't think it could happen. There is no way to untag images in docker (to my knowledge) except for removing the image (which is not possible while it's in use by a container).
Perhaps we could wrap the error with a context at least: "The image tag %q that the container was started with could not be found in docker: ".

The problem is that we are asking docker for "what is the image ID of the image tagged with foo:latest" to compare it with the current containers image ID, so there is no alternative fallback here really...

or the logic test for consistency with RepoTags.

This could be added to the initial sanity checks actually. We can't use RepoTags to determine the correct tag, but the Config.Image has to match one of the RepoTags.

ptr727 · 2022-10-14T14:08:44Z

ptr727
Oct 14, 2022
Author

Thanks, any verification or logging could help the next person, especially since it is so easy to happen when using API's (like Ansible) to create containers.

As expected, last night had no missing image errors:

Could not do a head request for "[lscr.io/linuxserver/unifi-controller:latest](http://lscr.io/linuxserver/unifi-controller:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/unifi-controller:pull\""
Could not do a head request for "[lscr.io/linuxserver/sonarr:latest](http://lscr.io/linuxserver/sonarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/sonarr:pull\""
Could not do a head request for "[lscr.io/linuxserver/sabnzbd:latest](http://lscr.io/linuxserver/sabnzbd:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/sabnzbd:pull\""
Could not do a head request for "[lscr.io/linuxserver/radarr:latest](http://lscr.io/linuxserver/radarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/radarr:pull\""
Could not do a head request for "[lscr.io/linuxserver/radarr:latest](http://lscr.io/linuxserver/radarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/radarr:pull\""
Could not do a head request for "[lscr.io/linuxserver/qdirstat:latest](http://lscr.io/linuxserver/qdirstat:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/qdirstat:pull\""
Could not do a head request for "[lscr.io/linuxserver/plex:latest](http://lscr.io/linuxserver/plex:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/plex:pull\""
Found new netdata/netdata:latest image (8937980937be)
Could not do a head request for "[lscr.io/linuxserver/mariadb:latest](http://lscr.io/linuxserver/mariadb:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/mariadb:pull\""
Could not do a head request for "[lscr.io/linuxserver/heimdall:latest](http://lscr.io/linuxserver/heimdall:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/heimdall:pull\""
Could not do a head request for "[lscr.io/linuxserver/heimdall:latest](http://lscr.io/linuxserver/heimdall:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/heimdall:pull\""
Found new grafana/grafana-oss:latest image (17412ade42a8)
Stopping /grafana (64e954775e06) with SIGTERM
Stopping /netdata (0966dcebe6f5) with SIGTERM
Creating /netdata
Creating /grafana
Removing image a97ee0e8035f
Removing image c5fcc2150a6d

0 replies

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: No such image #1409

{{title}}

Replies: 7 comments 11 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

This comment has been minimized.

This comment has been minimized.

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Error: No such image #1409

ptr727 Oct 5, 2022

Replies: 7 comments · 11 replies

piksel Oct 5, 2022 Maintainer

ptr727 Oct 5, 2022 Author

ptr727 Oct 6, 2022 Author

This comment has been minimized.

This comment has been minimized.

piksel Oct 13, 2022 Maintainer

ptr727 Oct 13, 2022 Author

piksel Oct 13, 2022 Maintainer

ptr727 Oct 13, 2022 Author

piksel Oct 13, 2022 Maintainer

ptr727 Oct 13, 2022 Author

piksel Oct 14, 2022 Maintainer

ptr727 Oct 14, 2022 Author

ptr727
Oct 5, 2022

Replies: 7 comments 11 replies

piksel
Oct 5, 2022
Maintainer

ptr727 Oct 5, 2022
Author

ptr727
Oct 6, 2022
Author

piksel
Oct 13, 2022
Maintainer

ptr727
Oct 13, 2022
Author

piksel Oct 13, 2022
Maintainer

ptr727 Oct 13, 2022
Author

piksel Oct 13, 2022
Maintainer

ptr727 Oct 13, 2022
Author

piksel Oct 14, 2022
Maintainer

ptr727
Oct 14, 2022
Author