Automatically build and publish to Github Container Registry #213

proffalken · 2021-08-13T07:53:22Z

This PR compliments the sentiment behind #53 however it does the following:

Build for multiple architectures using the Github Actions free workflow builders
Release to GITHUB CONTAINER REGISTRY instead of hub.docker.com to avoid the container download restrictions

It also creates tagged images for the following scenarios:

Each new PR that is created, tagged with the PR number and SHA-1
Each merge to master, tagged with the SHA-1, auto-incremented version number, and latest for the most recent version

This is a "copy&paste" from https://github.com/MakeMonmouth/mmbot/blob/main/.github/workflows/container_build.yml and will require packages to be enabled for this repo, however it already uses the new dynamic GITHUB_TOKEN setting to authenticate.

bsord · 2021-08-15T17:53:40Z

I think this is a great improvement and is the 'standard' i've seen across many repos these days. Builds can still be done locally for testing if needed.

.github/workflows/container_build.yml

MichelBaie · 2021-09-01T12:44:27Z

would be helpful for a project who has daily pushes

gaby · 2021-09-04T22:22:04Z

This PR would affect teams using uptime-kuma behind corporate proxies, since ghcr.io is not usually added to those.

proffalken · 2021-09-06T11:30:43Z

This PR would affect teams using uptime-kuma behind corporate proxies, since ghcr.io is not usually added to those.

True, although given hub.docker.com's policies on charging for downloads etc, I suspect that many organisations will start to migrate over to GHCR or similar in the near future.

I'd also argue that as a rule getting new sites added to a whitelist, especially when that site is owned by a "trusted entity" such as github, is probably less of a challenge than if we were hosting on "Dave's Docker Service", so shouldn't prevent us from implementing this approach.

gaby · 2021-09-12T04:59:19Z

This PR would affect teams using uptime-kuma behind corporate proxies, since ghcr.io is not usually added to those.

True, although given hub.docker.com's policies on charging for downloads etc, I suspect that many organisations will start to migrate over to GHCR or similar in the near future.

I'd also argue that as a rule getting new sites added to a whitelist, especially when that site is owned by a "trusted entity" such as github, is probably less of a challenge than if we were hosting on "Dave's Docker Service", so shouldn't prevent us from implementing this approach.

I'd say the workflow could publish the final image to both ghcr.io and docker.io 👍🏻

I like this PR cause it makes the release process also open source, right now it isn't.

Saibamen · 2021-09-14T10:02:45Z

.github/workflows/container_build.yml

+          context: ./
+          file: ./dockerfile
+          push: true
+          platforms: linux/amd64, linux/arm/v7, linux/arm/v6, linux/arm64


Can we use multiline array here (or list like for branches)?

Saibamen · 2021-09-18T20:49:27Z

Related issue: #440

gaby · 2021-09-18T21:32:52Z

@proffalken Can you add support for pushing to both ghcr.io and hub.docker.com ?

proffalken · 2021-09-18T21:48:12Z

@gaby I'll see what I can do in the next week, work and life are reasonably busy right now!

louislam · 2021-09-20T03:07:38Z

My preference is still the npm run build-docker command on my local machine.

Not saying that this PR is not good, but I just don't want to maintain more things in the future. For example, a few week ago, I added Debian + Alpine based images support, I believe that this yaml file have to be updated too.

Also, the build time is very good on my local machine while Github Action is not always good.

Saibamen · 2021-09-20T13:13:38Z

But maybe we can just setup CI for only build (not publish) in GitHub?

I bet you don't build Docker image after each commit.

If we have CI, we could start fixing build issues right after pushing wrong commit/PR

gaby · 2021-09-20T13:16:51Z

But maybe we can just setup CI for only build (not publish) in GitHub?

I bet you don't build Docker image after each commit.

If we have CI, we could start fixing build issues right after pushing wrong commit/PR

^ this! Also if we build a :dev image on each merge, etc. Its super easy to create a release since that would only take retagging the image.

proffalken · 2021-09-20T16:16:36Z

^ this! Also if we build a :dev image on each merge, etc. Its super easy to create a release since that would only take retagging the image.

FWIW, the code in this PR automatically generates a number of images with each run, including the following:

Git Tag when building from master/main (so uptime-kuma:1.2.3 when a commit is tagged v1.2.3 etc.)
latest when building from master/main (uptime-kuma:latest)
PR ID (uptime-kuma:PR-213 for example)
Checksum (uptime-kuma:<SHA SUM>)

I think there's a couple of other tags it builds to as well, so all of this is already in there.

If you take a look at the MVentory setup that I took this from you can see the expected output.

gaby · 2021-10-11T01:03:13Z

.github/workflows/container_build.yml

+        with:
+          context: ./
+          file: ./dockerfile
+          push: true


This should be:

${{ github.event_name != 'pull_request' }}

Else every pull-request is going to try to Push an image.

Yup, that's deliberate, it means we can test the code for each PR in a dedicated docker image rather than waiting for the code to reach main/master before we know if the container works properly.

The problem is that building and push an image per PR will create a lot of unused tags in the registry. The latest changes made by @louislam are now testing/linting the code using Github Actions which should cover your use-case.

OK, so let's say I'm working on a PR, I push that code up, it lints/tests fine via Github actions, and we merge to master.

We then find that there's an issue with the container setup rather than the code (libc version change or something equally obscure), but we only find that out after it's been released to the wider world and causes a slew of github issues to be created.

If we create a container on each PR (and note that this is each PR, not each commit, it is rebuilt with the same tags each time) then we can test that the container itself works as well.

I'm really not convinced that "too many tags in the registry" is a strong enough argument when the alternative is a falling deployment for users of the application?

Fair enough, we do this against a private registry instead of a public one. I do get your point.

.github/workflows/container_build.yml

proffalken · 2021-10-12T21:33:03Z

@louislam whilst we continue to discuss this, is there any chance you can add a "latest" tag when you publish new images?

My deployments are all automated and it gets frustrating when I forget that I've hard-coded the version for Uptime-Kuma when everything else is set to :latest.

It's a minor irritation, but it would be nice to have ;)

louislam · 2021-10-13T02:13:23Z

@louislam whilst we continue to discuss this, is there any chance you can add a "latest" tag when you publish new images?

My deployments are all automated and it gets frustrating when I forget that I've hard-coded the version for Uptime-Kuma when everything else is set to :latest.

It's a minor irritation, but it would be nice to have ;)

It should always point to the latest version of uptime kuma. It is not suggested because of the breaking changes of version 2.x in the future.

https://hub.docker.com/layers/louislam/uptime-kuma/latest/images/sha256-d4947d0d9ed82b22b6364b55f52932c79ee4dbf22a330fb6b92bd09eda234cdc?context=explore

proffalken · 2021-10-13T08:55:18Z

It should always point to the latest version of uptime kuma. It is not suggested because of the breaking changes of version 2.x in the future.

I completely missed this! :D

Thanks, and yeah, understand the issues with v2, this is for my test network so I'd expect stuff there to break every now and again with new releases of the various things running on it.

.github/workflows/container_build.yml

gaby · 2021-10-13T12:50:21Z

@proffalken With the recent changes of moving everything under docker/ all the checks on this PR are going to fail.

proffalken · 2021-10-13T12:51:24Z

@proffalken With the recent changes of moving everything under docker/ all the checks on this PR are going to fail.

Yeah, just spotted that, should be good to go now :)

TonyRL · 2021-10-16T19:35:33Z

.github/workflows/container_build.yml

+          push: true
+          platforms: linux/amd64, linux/arm/v7, linux/arm/v6, linux/arm64
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}


Maybe add a docker layer caching like this?

cache-from: type=gha, scope=${{ github.workflow }} cache-to: type=gha, mode=max, scope=${{ github.workflow }}

This should save some time on pulling.
Reference: Cache eviction policy

The cache policy is experimental though

gaby · 2021-10-18T12:55:37Z

@proffalken The build is having issues again, related to the base images not found.

This can be fixed by adding a step for building the base images before building the final image. The actions you are using support Caching so it will just use the cache from the previous stage.

Ex.

      - name: Build Base and push
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          context: ./
          file: ./docker/debian-base.dockerfile

      - name: Build Final and push
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          context: ./
          file: ./docker/dockerfile

The same would need to be duplicated for Alpine.

Caching between jobs is covered here: https://github.com/docker/build-push-action/blob/master/docs/advanced/test-before-push.md

deanomus · 2021-10-23T14:15:08Z

.github/workflows/container_build.yml

+      - 'master'
+  pull_request:
+    branches:
+      - master


For every commit on a branch for which there is a pull request to the master, a new image is built.
Shouldn't be like this, only when pushing commits to the master (merge commits)

IMO we should do it, because build on Windows is different from building on Linux container

Building the container only on specific branches (ie master/main/dev) means that there is very exact control over when a new image gets built, and anyone attempting to put up a devious image would have to have get it to pass code review. Much harder for someone to accidentally get a devious image.

Images are deliberately built for each PR so that they can be tested before releasing.

Failure to do so could result in underlying issues with the container (glibc / nodejs upgrade at the is level etc) breaking the deployment and unless we test at PR level then we would only find this out after the "production" container has been released.

Building and testing on branches is an established pattern within the ci/cd community because it brings value without adding unnecessary complication.

To specifically address your point about "bad" images, all images built from a PR would tagged with the PR number and result in the format uptime-kuma:PR-3, this means that someone would have to deliberately update their configuration to pull these images, and would never see them otherwise, even if their setup was configured for uptime-kuma:latest.

As the code would not be merged until after a review, the chances of a "bad" container making it through to release is just as likely as "bad" code making it through, at which point it stops being an issue with how we package and publish and becomes an issue with how we review the code.

I'm used to a PR -> dev/nightly -> master/main process. All steps are still done before prod, in dev/nightly. Everything is still fully tested, including the container builds. Some things however, like steps accessing keys or pushing test images to public, wait for code review & approval/merging to dev before running.

I don't like the possibility of ANY container potentially containing un-reviewed code being public. Its unlikely, but we shouldn't assume someone won't just sort by newest tag and use that. Imagine someone un-experienced, who doesn't know git[hub/lab] reversing PR to mean Public Release instead of Pull Request. The more I think on it, the more I am against blindly pushing PRs to the registry.

an established pattern within the ci/cd community

Can you link some other projects that are using this process please? I will still be paranoid, but I will withdraw my comments if this is indeed a commonly accepted pattern.

Sorry if I'm overly security paranoid, but it pays to be so nowadays unfortunately.

.github/workflows/container_build.yml

Co-authored-by: Adam Stachowicz <saibamenppl@gmail.com>

Automatically build and publish to Github Container Registry

be9790c

proffalken mentioned this pull request Aug 13, 2021

Automate Docker Releases with GitHub Actions #53

Closed

louislam added feature-request Request for new features to be added priority:low Low Priority labels Aug 17, 2021

austin-engle reviewed Aug 25, 2021

View reviewed changes

.github/workflows/container_build.yml Outdated Show resolved Hide resolved

Update the container name to the correct value

f14e9df

Saibamen reviewed Sep 14, 2021

View reviewed changes

Saibamen mentioned this pull request Sep 18, 2021

git.kuma.pet not working #440

Closed

gaby reviewed Oct 11, 2021

View reviewed changes

.github/workflows/container_build.yml Show resolved Hide resolved

gaby reviewed Oct 13, 2021

View reviewed changes

.github/workflows/container_build.yml Outdated Show resolved Hide resolved

proffalken added 3 commits October 13, 2021 13:45

Merge branch 'louislam:master' into feature/auto_build_and_release

2778fa5

Update changes to use master rather than main

cd5ff2c

Use correct case for Dockerfile

0ed5453

Dockerfile has been moved, updating path

69a28c5

TonyRL reviewed Oct 16, 2021

View reviewed changes

gaby mentioned this pull request Oct 18, 2021

Automate the release process #740

Closed

deanomus reviewed Oct 23, 2021

View reviewed changes

This comment was marked as resolved.

Sign in to view

Computroniks reviewed May 11, 2022

View reviewed changes

.github/workflows/container_build.yml Outdated Show resolved Hide resolved

Saibamen reviewed May 12, 2022

View reviewed changes

.github/workflows/container_build.yml Outdated Show resolved Hide resolved

fix formatting issue

f6f1fe9

Co-authored-by: Adam Stachowicz <saibamenppl@gmail.com>

CommanderStorm mentioned this pull request May 28, 2023

Improve resolution notification with some additional information #1970

Closed

1 task

CommanderStorm mentioned this pull request Jun 18, 2023

Release docker images on ghcr registry #2464

Closed

1 task

CommanderStorm added the area:core issues describing changes to the core of uptime kuma label Dec 8, 2023

CommanderStorm added the needs:review this PR needs a review by maintainers or other community members label May 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically build and publish to Github Container Registry #213

Automatically build and publish to Github Container Registry #213

proffalken commented Aug 13, 2021

bsord commented Aug 15, 2021

MichelBaie commented Sep 1, 2021

gaby commented Sep 4, 2021

proffalken commented Sep 6, 2021

gaby commented Sep 12, 2021

Saibamen Sep 14, 2021

Saibamen commented Sep 18, 2021

gaby commented Sep 18, 2021

proffalken commented Sep 18, 2021

louislam commented Sep 20, 2021 •

edited

Saibamen commented Sep 20, 2021

gaby commented Sep 20, 2021 •

edited

proffalken commented Sep 20, 2021

gaby Oct 11, 2021

proffalken Oct 11, 2021

gaby Oct 13, 2021

proffalken Oct 13, 2021

gaby Oct 13, 2021

proffalken commented Oct 12, 2021

louislam commented Oct 13, 2021

proffalken commented Oct 13, 2021

gaby commented Oct 13, 2021

proffalken commented Oct 13, 2021

TonyRL Oct 16, 2021 •

edited

gaby Oct 17, 2021 •

edited

gaby commented Oct 18, 2021 •

edited

deanomus Oct 23, 2021

Saibamen Oct 23, 2021

deefdragon Oct 24, 2021

proffalken Oct 25, 2021

deefdragon Oct 25, 2021

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Automatically build and publish to Github Container Registry #213

Are you sure you want to change the base?

Automatically build and publish to Github Container Registry #213

Conversation

proffalken commented Aug 13, 2021

bsord commented Aug 15, 2021

MichelBaie commented Sep 1, 2021

gaby commented Sep 4, 2021

proffalken commented Sep 6, 2021

gaby commented Sep 12, 2021

Choose a reason for hiding this comment

Saibamen commented Sep 18, 2021

gaby commented Sep 18, 2021

proffalken commented Sep 18, 2021

louislam commented Sep 20, 2021 • edited

Saibamen commented Sep 20, 2021

gaby commented Sep 20, 2021 • edited

proffalken commented Sep 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proffalken commented Oct 12, 2021

louislam commented Oct 13, 2021

proffalken commented Oct 13, 2021

gaby commented Oct 13, 2021

proffalken commented Oct 13, 2021

TonyRL Oct 16, 2021 • edited

Choose a reason for hiding this comment

gaby Oct 17, 2021 • edited

Choose a reason for hiding this comment

gaby commented Oct 18, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

louislam commented Sep 20, 2021 •

edited

gaby commented Sep 20, 2021 •

edited

TonyRL Oct 16, 2021 •

edited

gaby Oct 17, 2021 •

edited

gaby commented Oct 18, 2021 •

edited