[Bug]: Docker tag :latest points to nightly builds

[Nutomic]

Requirements

• Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• Did you check to see if this issue already exists?
• Is this only a single bug? Do not put multiple bugs in one issue.
• Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

Various instance admins recently ran into problems after they accidentally upgraded to the :latest Docker tag. This shouldnt be a problem as latest is supposed to point to the most recent release version. But in fact it points to nightly builds from the development branch. This is not at all intended.

Our Docker images are built and pushed by the woodpecker docker plugin. From my understanding of the documentation, our nightly build task should not push to the :latest tag. There is also an issue indicating that the :latest tag is being pushed to incorrectly. So this seems like a bug. It would be very helpful if someone who is familiar with Go could look through the source code to find out how this can be resolved, so that :latest tag only gets pushed for releases.

The same problem is also happening for lemmy-ui.

[Nothing4You]

just looking at https://hub.docker.com/r/dessalines/lemmy/tags i don't think this is from the nightly task, as currently the latest tag is from 2 days ago while dev is only 20 hours old.
also, dev has amd64 and arm64 builds, while latest only has an amd64 build.
they also have different digests, which wouldn't be the case if they were from the same build run.

[Nothing4You]

latest is built from a different Dockerfile than dev, as they have different image layers.

[Nothing4You]

latest is the same image that was pushed as 0.18.5-rc.2, so whatever was used to push that is likely also where the latest tag comes from.

[Nothing4You]

it appears to actually be published by the publish_release_docker job, which runs on any tag: https://github.com/LemmyNet/lemmy/blob/main/.woodpecker.yml#L236-L247

this would also fall in line with the woodpecker docker plugin issue you linked, as it does indeed use auto_tag: true.

i am however confused by the documentation, as the documentation states the following for auto_tag:

If set to true, it will use the default_tag ("latest") on tag event or default branch. If it's a tag event it will also assume sem versioning and add tags accordingly (x, x.x and x.x.x). If it's not a tag event, and no default branch, automated tags are skipped.

i read this as adding the latest tag would be expected in this case.

a workaround would likely be to set default_tag to a different value, but this would also break desired latest tags for non-rc tags.

[Nothing4You]

oh well, i misread the issue you linked.

https://codeberg.org/woodpecker-plugins/docker-buildx/issues/49 is the issue asking to add an option to not add the latest tag when using auto_tag.

this behavior is indeed by design.

the other issue, https://codeberg.org/woodpecker-plugins/docker-buildx/issues/53, is precisely about the workaround i was suggesting above not working, as the latest tag would still be tagged when you specify a different default_tag and it also can't be worked around by specifying default_suffix.

[Nutomic]

Hmm then we should push to :latest only for stable releases, not release candidates. But that seems complicated to configure.

Or we get rid of auto_tag: true so it doesnt push to :latest at all. And also wont create 0.18 tag for 0.18.5 release. What do you think @LemmyNet?

[Nothing4You]

it might be an option to have conditional logic in the CI job to change behavior depending on the tag.

for example, you could enable auto_tag: true if the git tag matches ^[0-9]+\.[0-9]+\.[0-9]+$ only and otherwise use the git tag as tag.
this should allow stable releases to still benefit from automatic tags while any tags that don't match the expected format would only be tagged with the same tag used in git.

when using this approach it should be kept in mind that you'd never want to tag a release like 0.18.5 after a newer minor version, such as 0.19.0 was tagged, as otherwise the latest tag would be downgraded from 0.19.0 to 0.18.5 at that point.

[Trombach]

@Nothing4You I've looked into the woodpecker docs and it seems that the ref filter only supports doublestar matching and not full regular expressions. The filter would probably have to be built using evaluate and CI_COMMIT_TAG. However, I'm not even sure that evaluate supports regex either

Another option would be to use Github actions for deploying the latest build to docker. You can use an on: release trigger such that the workflow only deploys when a release is published instead of on every tagging event. I understand it might not be desirable to have your CI use 2 different platforms, but I just wanted to mention this possibility.

There are still some other considerations here, i.e. it should be ensured that when a minor version gets applied to the previous major release after a new major release is out (for example 18.6 gets released after 19.1) that it doesn't get pushed to the latest docker tag.

[Nothing4You]

evaluate does support regex.
https://woodpecker-ci.org/docs/usage/pipeline-syntax#evaluate refers to https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md, which contains the following example: foo matches "^[A-Z].*"

something like this would probably work:

  publish_release_docker_latest:
    # [...]
    settings:
      auto_tag: true
      # [...]
    when:
      evaluate: 'CI_PIPELINE_EVENT == "tag" && CI_COMMIT_TAG matches "^[0-9]+\.[0-9]+\.[0-9]+$"'

  publish_release_docker_unstable:
    # [...]
    settings:
      auto_tag: false
      tag: ${CI_COMMIT_TAG}
      # [...]
    when:
      evaluate: 'CI_PIPELINE_EVENT == "tag" && not (CI_COMMIT_TAG matches "^[0-9]+\.[0-9]+\.[0-9]+$")'

or one could try to fit it in the same job by setting auto_tag and conditionally setting or not setting tag based on some expression.

if logic to avoid tagging latest for a lower version is to be added, this might make the process quite a bit more complex, as then you'd probably need to run a dedicated script to fetch versions from docker hub and compare version strings.
i can't currently think of a good solution for this other than building a script that queries the docker registry api, tries to determine the latest release and then comparing versions between git tag and discovered existing docker tag.

[Nothing4You]

thoughts on the approach above? should i draft a PR for that?
do you think it's reasonable to expect maintainers to not tag older minor versions for the time being?