Container image name not updated when image registry changes

[ptr727]

SUMMARY

I discovered the issue when I change several LSIO containers to pull from linuxserver/heimdall:latest to lscr.io/linuxserver/heimdall:latest.
The problem manifested as Watchtower that reported Unable to update container "/plex": Error: No such image: linuxserver/plex:latest. Proceeding to next.
After some discussions with Watchtower I discovered that the problem may have been due to the container not being recreated when the image changed.

Before I changed recreate to yes:

pieter@server-1:~$ docker container inspect heimdall | jq .[].Config.Image
"linuxserver/heimdall:latest"
pieter@server-1:~$ docker container inspect heimdall | jq .[].Image
"sha256:ee9db55d01f96029523b187a1592c35144f3d26c2f6017a3d9856c7c99e12af5"
pieter@server-1:~$ docker image inspect "sha256:ee9db55d01f96029523b187a1592c35144f3d26c2f6017a3d9856c7c99e12af5" | jq .[].RepoTags
[
  "lscr.io/linuxserver/heimdall:latest"
]

After I changed recreate to yes:

pieter@server-1:~$ docker container inspect heimdall | jq .[].Config.Image
"lscr.io/linuxserver/heimdall:latest"
pieter@server-1:~$ docker container inspect heimdall | jq .[].Image
"sha256:ee9db55d01f96029523b187a1592c35144f3d26c2f6017a3d9856c7c99e12af5"
pieter@server-1:~$ docker image inspect "sha256:ee9db55d01f96029523b187a1592c35144f3d26c2f6017a3d9856c7c99e12af5" | jq .[].RepoTags
[
  "lscr.io/linuxserver/heimdall:latest"
]

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.docker.docker_container

ANSIBLE VERSION

pieter@server-1:~$ ansible --version
ansible [core 2.13.5]
  config file = None
  configured module search path = ['/home/pieter/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  ansible collection location = /home/pieter/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
  jinja version = 3.0.1
  libyaml = True

COLLECTION VERSION

pieter@server-1:~$ ansible-galaxy collection list community.docker

# /home/pieter/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
community.docker 3.1.0

# /usr/local/lib/python3.9/dist-packages/ansible_collections
Collection       Version
---------------- -------
community.docker 2.7.1

CONFIGURATION

pieter@server-1:~$ ansible-config dump --only-changed

OS / ENVIRONMENT

Proxmox 7.2

STEPS TO REPRODUCE

- name: "Install Heimdall"
  community.docker.docker_container:
    name: heimdall
    image: lscr.io/linuxserver/heimdall:latest
    pull: yes
    hostname: "heimdall-{{ ansible_hostname }}"
    domainname: "{{ ansible_domain }}"
    restart_policy: unless-stopped
    env:
      TZ: "{{ local_timezone }}"
      PUID: "{{ user_id }}"
      PGID: "{{ group_id }}"
    volumes:
      - "{{ appdata_dir }}/heimdall/config:/config"
    networks:
      - name: "{{ docker_local_network }}"
    # published_ports:
    # External routing via traefik
    # - 80:80
    # - 443:443
    labels:
      traefik.enable: "true"
      traefik.http.routers.heimdall.rule: "Host(`heimdall.{{ ansible_domain }}`)"
      traefik.http.routers.heimdall.entrypoints: "websecure"
      traefik.http.routers.heimdall.priority: "1"
      com.centurylinklabs.watchtower.enable: "true"
    recreate: "{{ docker_container_recreate }}"

EXPECTED RESULTS

I would like the recreate: no logic to be aware of the current and new image name, and either fail the operation when the image changes, or update the image, including updating to the new registry if different.

ACTUAL RESULTS

When changing the image registry, the image name does not get updated to reflect the new registry.

NA

[felixfontein]

I think the problem is that right now, the module checks whether the image ID differs, and doesn't care about the image name. (Since in the end the relevant thing is the image ID, not its name.) Am I assuming correctly that in your case, the same image is available under two different names, and you want to make sure that the container isn't simply using the correct image, but that the docker ps output also mentions the same name as the image you passed on to the module?

I would like the recreate: no logic to be aware of the current and new image name, and either fail the operation when the image changes, or update the image

The image doesn't need to update, since both names refer to the same image in your case. What you want is the image name to change in the container config, not the image that's used.

[ptr727]

Yes, what I think should happen is that when the image attribute in yaml changes that the image name of the currently existing container should be changed to match the new name, such that they stay in sync, and can be used by watchtower to pull new images.

To summarize below, the the details are in the Watchtower discussion.

The problem is that Watchtower logic needs to know how to recreate the container when it updates the image, and watchtower uses the container image attribute, when Ansible changes the image backing the container but not configured container image name, the image does not match, and watchtower fails.

The problem happened to me because I changed the image in yaml from linuxserver/heimdall:latest to lscr.io/linuxserver/heimdall:latest, the same image name and tag but a different registry.
When Watchtower ran it was using the image attribute of container, and that was still the version with no explicit registry, that defaults to docker.io, watchtower is now confused because the container still says it is using linuxserver/heimdall:latest, (defaulting to docker.io) while the actual image pulled by ansible was lscr.io/linuxserver/heimdall:latest.

In many cases the same image name at different public registries may be the same, it need not be, foo/bar at docker.io could be completely different to foo/bar at ghcr.io. In this case LSIO publishes to lscr.io and docker.io, so that made the troubleshooting more difficult, because a similar hashed image exist, but at the incorrect registry.

[felixfontein]

Since the current behavior has been the default for some time now, I'm wondering whether this should be more a bugfix (i.e. change the behavior) or a new feature (allow to change the behavior by setting an option). It might well be that someone already depends on the current behavior. I'm currently tending to treat this more like a new feature, i.e. have an option which allows to control the behavior. We can still deprecate the current default (i.e. doing the old behavior) so that folks have some time to notice that the behavior will change, and already adjust their playbooks/roles to the new behavior.

[ptr727]

Yes I understand the difficulty of it is not a bug it is a feature.

I have some thoughts:

• Add a new value for recreate, instead of yes, no, add auto, similar to how volume recreation works (that is a bit of a mystery itself). With yes and no same as before, and a new auto behavior that will try to make the container match the config, e.g. recreate or modify if possible.
• Add an error or warning message when the resulting container is not the desired container.
• Fail the operation if the result will not match the config.

[felixfontein]

• Add a new value for recreate, instead of yes, no, add auto, similar to how volume recreation works (that is a bit of a mystery itself). With yes and no same as before, and a new auto behavior that will try to make the container match the config, e.g. recreate or modify if possible.

I don't think this makes sense, since recreate=false already means what you want with recreate=auto. (The recreate option should have better been named force_recreate.)

[felixfontein]

I created a PR for a new option image_name_mismatch in #488. With that option, you can control this behavior.

I think the default value (right now: ignore) for this new option should eventually change to recreate, but I would wait a certain amount of time before deprecating the current default.

[ptr727]

Thank you, agree, recreate should (eventually) be default.

[ptr727]

In case it helps somebody:
TL;DR: Mixing the same image between registries without using an explicit registry seems to confuse docker, when using images that are pushed to multiple registries, and changing between registries, first remove all local images, then pull specifying the full registry path, e.g. docker.io/foo/bar and ghcr.io/foo/bar

I changed a couple container configs to pull from docker.io instead of from ghcr.io and lscr.io because I find GHCR is too unreliable with frequent broken pipe errors (lscr.io is a CNAME for ghcr.io).

E.g.
ghcr.io/home-assistant/home-assistant:latest -> homeassistant/home-assistant:latest
lscr.io/linuxserver/plex:latest -> linuxserver/plex:latest

I added image_name_mismatch: recreate and ran my ansible update.

Next morning Watchtower was unhappy and complained that images could not be found.
I inspected with Portainer and noticed that the images were cached and were no longer associated with a registry.
Pulling a new image also failed.

I deleted all containers and images.
I updated my config to explicitly state docker.io/foo or lscr.io/foo or ghcr.io/foo.
I recreated the containers, and I could now see that Portainer and docker was happy, and listed the explicitly set registry as part of the image name.
Next morning Watchtower was also happy.

[felixfontein]

I started the deprecation process for the current default in #703.