How to revert a "bad" update?

[pmorch]

How does one (ensure that one can) revert to the previous version if an update is "bad"? Does a log of updated containers get persisted anywhere and can I also log what sha256 sum was running before, so I could revert if I wanted to?

The use case

Lets for the sake of argument say that some-container is running fine. Watchtower discovers it has an update, and watchtower updates it. Two days, a reboot and restart of watchtower later, I discover that the new version of some-container is misbehaving. How do I see that watchtower updated some-container two days ago? And see which version was running before?

Am I missing something about logging?

The docker log will disappear when the watchtower container is restarted, right? Does information about which containers got restarted get persisted anywhere?

And how do I know what version was running before the update? When I looked at the standard logging output from watchtower:

$ sudo docker run --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower \
    --run-once wireguard
time="2025-04-06T22:31:51Z" level=info msg="Watchtower 1.7.1"
time="2025-04-06T22:31:51Z" level=info msg="Using no notifications"
time="2025-04-06T22:31:51Z" level=info msg="Only checking containers which name matches \"wireguard\""
time="2025-04-06T22:31:51Z" level=info msg="Running a one time update."
time="2025-04-06T22:31:54Z" level=info msg="Found new lscr.io/linuxserver/wireguard:latest image (d561f9f37c30)"
time="2025-04-06T22:31:54Z" level=info msg="Stopping /wireguard (8fa6949f3cca) with SIGTERM"
time="2025-04-06T22:31:58Z" level=info msg="Creating /wireguard"
time="2025-04-06T22:31:58Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=1 notify=no
time="2025-04-06T22:31:58Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

It did not contain the version that was running before, so I would not be able to revert wiregard (in this case) to the previous version, because the previous version isn't present in the log output.

Brute force solution

I was thinking of a cron job doing something like:

sudo docker image ls --digests --format json | \
    jq '{Repository, Tag, Digest, ID, Size}' | \
    jq -s 'sort_by(.Repository,.Tag,.Digest)'

and then persisting the output in git, which would sort of be enough information to see when hello-world:latest or whatever got updated, but I'm surprised the sha256 sum of the new and old versions aren't present in the watchtower output.

What do you do?

How do others revert "bad" updates? The version information to revert later isn't stored anywhere that I could find.

[nicholas-fedor]

How does one (ensure that one can) revert to the previous version if an update is "bad"?

Remember that containers are ephemeral objects. Recreate the container with the previous image, which should be on the host, unless other tools or automation that you have setup have deleted the image layers. If that's the case, then you'll need to pull the previous image from the container registry.

Does a log of updated containers get persisted anywhere and can I also log what sha256 sum was running before, so I could revert if I wanted to?

Docker's documentation is a good resource for this.
tl;dr - You have to set it up yourself.

In the context of Watchtower, it does have logging capabilities, which are also intertwined with its notifications features: https://containrrr.dev/watchtower/notifications/. If the Watchtower container is still present, then its logs should still be accessible. The best thing is to have notifications setup to help with monitoring and managing such situations. As an example, I use Gotify, which keeps things local, but allows me to have access to important information regarding updates, even if the Watchtower container itself is removed. Remember that you can also set Watchtower to update or not update certain containers. In your case, pinning the container's image will be best, until you can verify the next proceeding version has been fixed.

The docker log will disappear when the watchtower container is restarted, right?

Logs will be gone when the container is removed. Otherwise, they're still there.

Am I missing something about logging?

Yes and no. Watchtower has a ton of configuration options, but it also has issues, especially as the Containrrr version has been left unmaintained. [Shameless plug] Check out my fork, where I've been attempting to help address these sorts of issues.