Error: No such image

[ptr727]

Describe the bug

I started with a discussion, but based on trace logs I now think this is most likely a bug.

The past week I noticed that email state reports would report Unable to update container "/plex": Error: No such image: linuxserver/plex:latest. Proceeding to next.
The errors seem to happen for linuxserver containers, but I am not 100% sure it did not happen to other sources.

After enabling tracing, I still did not see any errors, the logic seems to report no update needed, then no image found?

time="2022-10-07T02:00:35-07:00" level=debug msg="No pull needed. Skipping image."
time="2022-10-07T02:00:35-07:00" level=info msg="Unable to update container \"/plex\": Error: No such image: linuxserver/plex:latest. Proceeding to next."

Here is the last email message:

Found new esphome/esphome:latest image (aa5b4e5153ff)
Found new netdata/netdata:latest image (e4fcfe490c1a)
Could not do a head request for "[lscr.io/linuxserver/sonarr:latest](http://lscr.io/linuxserver/sonarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/sonarr:pull\""
Found new traefik:latest image (e6de8578b238)
Found new traefik:latest image (e6de8578b238)
Could not do a head request for "[lscr.io/linuxserver/unifi-controller:latest](http://lscr.io/linuxserver/unifi-controller:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/unifi-controller:pull\""
Could not do a head request for "[lscr.io/linuxserver/qdirstat:latest](http://lscr.io/linuxserver/qdirstat:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/qdirstat:pull\""
Unable to update container "/plex": Error: No such image: linuxserver/plex:latest. Proceeding to next.
Could not do a head request for "[lscr.io/linuxserver/radarr:latest](http://lscr.io/linuxserver/radarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/radarr:pull\""
Could not do a head request for "[lscr.io/linuxserver/radarr:latest](http://lscr.io/linuxserver/radarr:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/radarr:pull\""
Unable to update container "/heimdall": Error: No such image: linuxserver/heimdall:latest. Proceeding to next.
Unable to update container "/heimdall-public": Error: No such image: linuxserver/heimdall:latest. Proceeding to next.
Unable to update container "/sabnzbd": Error: No such image: linuxserver/sabnzbd:latest. Proceeding to next.
Could not do a head request for "[lscr.io/linuxserver/mariadb:latest](http://lscr.io/linuxserver/mariadb:latest)", falling back to regular pull.
Reason: registry responded to head request with "401 Unauthorized", auth: "Bearer realm=\"[https://ghcr.io/token\](https://ghcr.io/token%5C)",service=\"[ghcr.io](http://ghcr.io/)\",scope=\"repository:linuxserver/mariadb:pull\""
Stopping /traefik (d31013f193eb) with SIGTERM
Stopping /traefik-public (d4ff74af97ec) with SIGTERM
Stopping /netdata (add23e187a60) with SIGTERM
Stopping /esphome (88b33d0a8984) with SIGTERM
Creating /esphome
Creating /netdata
Creating /traefik-public
Creating /traefik
Removing image baf1234921d3
Removing image f019d2123cbd
Removing image 0cfe412e54d8

And a snippet from the logs:

time="2022-10-07T02:00:34-07:00" level=debug msg="Trying to load authentication credentials." container=/plex image="linuxserver/plex:latest"
time="2022-10-07T02:00:34-07:00" level=debug msg="No credentials for linuxserver found" config_file=/config.json
time="2022-10-07T02:00:34-07:00" level=debug msg="Got image name: linuxserver/plex:latest"
time="2022-10-07T02:00:34-07:00" level=debug msg="Checking if pull is needed" container=/plex image="linuxserver/plex:latest"
time="2022-10-07T02:00:34-07:00" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-10-07T02:00:34-07:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-10-07T02:00:34-07:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-10-07T02:00:34-07:00" level=debug msg="Setting scope for auth token" image=linuxserver/plex scope="repository:linuxserver/plex:pull"
time="2022-10-07T02:00:34-07:00" level=debug msg="No credentials found."
time="2022-10-07T02:00:35-07:00" level=debug msg="Parsing image ref" host=index.docker.io image=linuxserver/plex normalized="docker.io/linuxserver/plex:latest" tag=latest
time="2022-10-07T02:00:35-07:00" level=trace msg="Setting request token" token="Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsIng1YyI6WyJNSUlDK1RDQ0FwK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakJHTVVRd1FnWURWUVFERXp0U1RVbEdPbEZNUmpRNlEwZFFNenBSTWtWYU9sRklSRUk2VkVkRlZUcFZTRlZNT2taTVZqUTZSMGRXV2pwQk5WUkhPbFJMTkZNNlVVeElTVEFlRncweU1qQXhNVEF5TWpJeE5EbGFGdzB5TXpBeE1qVXlNakl4TkRsYU1FWXhSREJDQmdOVkJBTVRPMUJaUTFJNlNWQmFRanBJUWxGWE9qZE1SVms2UWtGV1FqcEhTRGRhT2xWSVZ6TTZVa3RMVWpwRE4wNHpPbGxOTkZJNlNWaE5TRHBLVkZCQ01JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBbE5JOGFtMlBERnZzNndOeVl2d0dkZWZXVXJXQWdMZ3N2MzA2MnZycG5VNkN0WUpEZDZ2K2NEVG1FN1FlTllEaFMyaU1wU3djZkRCL2RFclEwdnhIZE4ycDIvODZmZy9TeWlIMnhmMGFVTjlDV1dud0JPaTIvS3hLditpbFNlQ01HYXRwRlg3SmYxcWI4N0Q5NUxOVDBvOU9OTmYxT3RidjY5ck9tL1RIVFh3clUvV3dTZlUyWktUbEw4SVRXRkRXN09ZK3hXdUJ0WUpteVhqcVpsaWRBbUNTdTdHY0Y0MVB5em9KTFFTMnJCdXJwOXc0cWgxMFk1bUNIcWdsaEI1Rk9aOUs0T2pUaVhUUHJFUk5WcnArUFVIR3JBYVRPRTBwQzgyUHBuWVZhNzNDUkdsMEdDdC9RckJwVjRpdmswdzF0eEtkV1NiSDNnRmtqZ2g1N0tOcDhRSURBUUFCbzRHeU1JR3ZNQTRHQTFVZER3RUIvd1FFQXdJSGdEQVBCZ05WSFNVRUNEQUdCZ1JWSFNVQU1FUUdBMVVkRGdROUJEdFFXVU5TT2tsUVdrSTZTRUpSVnpvM1RFVlpPa0pCVmtJNlIwZzNXanBWU0Zjek9sSkxTMUk2UXpkT016cFpUVFJTT2tsWVRVZzZTbFJRUWpCR0JnTlZIU01FUHpBOWdEdFNUVWxHT2xGTVJqUTZRMGRRTXpwUk1rVmFPbEZJUkVJNlZFZEZWVHBWU0ZWTU9rWk1WalE2UjBkV1dqcEJOVlJIT2xSTE5GTTZVVXhJU1RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlFQTdIY1VyVm1namo1cE01MXhZVHd2eGE1VnRqd2hub0dRZjFxTU52UGVHeVlDSUFwYm4vWFkvS1F5WWFWRnRjMWtsb0lmZzd4L3hlbkZhbkp4L0F2cURGdFgiXX0.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImxpbnV4c2VydmVyL3BsZXgiLCJhY3Rpb25zIjpbInB1bGwiXSwicGFyYW1ldGVycyI6eyJwdWxsX2xpbWl0IjoiMTAwIiwicHVsbF9saW1pdF9pbnRlcnZhbCI6IjIxNjAwIn19XSwiYXVkIjoicmVnaXN0cnkuZG9ja2VyLmlvIiwiZXhwIjoxNjY1MTMzNTM0LCJpYXQiOjE2NjUxMzMyMzQsImlzcyI6ImF1dGguZG9ja2VyLmlvIiwianRpIjoiZGNrcl9qdGlfVnp5dnhIejRZMjJpM1ZFMjNSc2h4VTZvTmM4PSIsIm5iZiI6MTY2NTEzMjkzNCwic3ViIjoiIn0.am7QJpmcgL7XeY4Bo8V-_0h70sw4I5zMi_Vlbdl3ryoDEgVob2Q9ut-2uc6QhuJssvJJipGo6AZ6LEGr7eWMbKvG5d9HAg2rzcySjBmQ45EDhGk5zP4mJJLYNTx0EmR8-SEHO2Yto5ZmPa2rToNVYFc0cGp3q0zVDSssfMIURM7Etq5SIqbQz6s0f0cQqcGfO55szOdsyPA5vBg_IuyjY7ApEMvkXK1gOZT_pAjteiwkNwa764ews_hDBntZXFlZ4AuZ7vy3MSAoN5ymuzuqgB9eEYaadq8eXS8JO0-9_5-Up6VS5GQkFziAvy8a16VNlRGNgzzwyzA3LL0fdtbEZQ"
time="2022-10-07T02:00:35-07:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/linuxserver/plex/manifests/latest"
time="2022-10-07T02:00:35-07:00" level=debug msg="Found a remote digest to compare with" remote="sha256:d4f8c0f5c4796b4e00b0668f2c048dfc57a39b7626ccb74fed47f57bb086b22a"
time="2022-10-07T02:00:35-07:00" level=debug msg=Comparing local="sha256:d4f8c0f5c4796b4e00b0668f2c048dfc57a39b7626ccb74fed47f57bb086b22a" remote="sha256:d4f8c0f5c4796b4e00b0668f2c048dfc57a39b7626ccb74fed47f57bb086b22a"
time="2022-10-07T02:00:35-07:00" level=debug msg="Found a match"
time="2022-10-07T02:00:35-07:00" level=debug msg="No pull needed. Skipping image."
time="2022-10-07T02:00:35-07:00" level=info msg="Unable to update container \"/plex\": Error: No such image: linuxserver/plex:latest. Proceeding to next."

Full logs with trace enabled attached below.

Steps to reproduce

Configure watchtower using Ansible:

- name: "Install Watchtower"
  community.docker.docker_container:
    name: watchtower
    image: containrrr/watchtower:latest
    pull: yes
    hostname: "watchtower-{{ ansible_hostname }}"
    domainname: "{{ ansible_domain }}"
    restart_policy: unless-stopped
    # Run as root
    # user: "{{ user_id }}:{{ group_id }}"
    env:
      TZ: "{{ local_timezone }}"
      WATCHTOWER_NOTIFICATIONS: "email"
      WATCHTOWER_NOTIFICATION_EMAIL_FROM: "{{ smtp_from }}"
      WATCHTOWER_NOTIFICATION_EMAIL_TO: "{{ smtp_to }}"
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER: "{{ smtp_server }}"
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT: "{{ smtp_port }}"
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_INCLUDE_STOPPED: "true"
      WATCHTOWER_INCLUDE_RESTARTING: "true"
      WATCHTOWER_TIMEOUT: "90"
      WATCHTOWER_SCHEDULE: "0 0 2 * * *"
      WATCHTOWER_WARN_ON_HEAD_FAILURE: "always"
      WATCHTOWER_DEBUG: "true"
      WATCHTOWER_TRACE: "true"
      # Only update containers that opt-in
      WATCHTOWER_LABEL_ENABLE: "true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - name: "{{ docker_local_network }}"
    # published_ports:
    # No ports published
    labels:
      traefik.enable: "false"
      com.centurylinklabs.watchtower.enable: "true"
    recreate: "{{ docker_container_recreate }}"

Configure e.g. Plex using Ansible:

- name: "Install Plex"
  community.docker.docker_container:
    name: plex
    image: lscr.io/linuxserver/plex:latest
    pull: yes
    hostname: plex-direct
    domainname: "{{ ansible_domain }}"
    restart_policy: unless-stopped
    env:
      TZ: "{{ local_timezone }}"
      PUID: "{{ user_id }}"
      PGID: "{{ group_id }}"
      VERSION: "latest" # docker, public, latest
    volumes:
      - "{{ appdata_dir }}/plex/config:/config"
      - "/data/media:/media"
    networks:
      - name: "{{ docker_public_network }}"
        ipv4_address: 192.168.1.47 # plex-direct.home.insanegenius.net
      - name: "{{ docker_local_network }}"
    # published_ports:
    # External routing via traefik and static IP
    # - 32400:32400
    labels:
      traefik.enable: "true"
      traefik.http.routers.plex.rule: "Host(`plex.{{ ansible_domain }}`) || Host(`plex.{{ external_services_domain }}`)"
      traefik.http.routers.plex.entrypoints: "websecure"
      # Manually specify port and schema
      traefik.http.services.plex.loadbalancer.server.scheme: "http"
      traefik.http.services.plex.loadbalancer.server.port: "32400"
      com.centurylinklabs.watchtower.enable: "true"
    recreate: "{{ docker_container_recreate }}"

Expected behavior

No error when image does not require updating, or an error in the log indicating why an image is missing.

Screenshots

No response

Environment

Proxmox 7.2-11 on x64 Kernel Version Linux 5.15.53-1-pve #1 SMP PVE 5.15.53-1 (Fri, 26 Aug 2022 16:53:52 +0200)
Docker version 20.10.18, build b40c2f6
Watchtower 1.4.0

Your logs

See attachment.

Additional context

_watchtower_logs.3.log

[ptr727]

May be related, but I noticed that GitHub Container Registry would occasionally and temporarily return 500 errors, and lscr.io/linuxserver/[some project] appears to be a hosted on GHCR.

Maybe there is a code path in watchtower where 500 errors are not logged resulting in a logical error with no associated debug output?

[ptr727]

ansible-collections/community.docker#485