Multi arch images on Docker Hub

[maggie44]

TLDR: There are now community built images to use in the meantime, which are updated for each new release of CloudflareD and available for different architectures (arm (Raspberry Pi); amd, etc.):

docker pull ghcr.io/maggie0002/cloudflared

Describe the feature you'd like
The GoLang code in Docker builds well for lots of architectures. At the moment though, I am having to do them all manually as only amd64 is pushed to Docker hub. Could we get multiarch images pushed? Seems like all the functionality is there to do so already, just the commands for the build and push process that need amending.

This request used to be edge cases for Raspberry Pi users. Now with Mac M1 and others around, it would be nice for developers to have native images too. The current amd64 does run on Mac, but with warnings:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

[maggie44]

Here is an example build process, deploying to GitHub Container Registry, but the theory is the same. I haven't got hardware to test the arm/v7 but the arm64 image works fine on my Mac M1, and the amd64 on a DigitalOcean droplet:

https://github.com/maggie0002/cloudflared-docker-multiarch
docker pull ghcr.io/maggie0002/cloudflared:latest

A few other things for consideration. Building in to Docker scratch containers and building with the flags -s -w reduces the image size from 52.5MB to 23.3MB and reduces vulnerabilities by removing unused content from the containers (see the commits for details).

It isn't as a pull request, as I imagine there are different deploy workflows.

[mjudeikis]

Would be good to do this too.
Example dockerfile:

FROM debian

ARG TARGETARCH
ENV TARGETARCH $TARGETARCH
RUN apt-get update; apt-get install curl tar gzip ca-certificates -y; apt-get clean
RUN curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-$TARGETARCH.deb
RUN dpkg -i cloudflared.deb    

Build command:

# linux/arm/v6,,linux/arm/v7 
docker buildx build --progress=plain --platform linux/amd64,linux/arm64 -t quay.io/synpse/cloudflared:$(TAG_NAME) --push -f Dockerfile .

Just an issue here than current deb artifacts for arm/v6 and arm/v7 (32 bit) is not compatible and can't be built. Where armv8 (64 bit) works

[mjudeikis]

@Maggie0002 tested your image on our test device fleet (we have all v6,v7,v8 arm out there) in Synpse.net . Worked like a charm.

I think building from source as you did is better than my .deb hack.

[maggie44]

You tried it on armv6 too? Did you build your own through a fork of the repo or pull from the current one? I didn't think I had added an armv6 image, would be curious how that one is working.

Adding some linked issues:
#420
#237
#588
#359

[mjudeikis]

Ah, sorry, My device rack with v6 was off :D (not enough coffee in the morning). yes, it is still failing :D
I gonna try this with my fork.

[maggie44]

I think armv6 is pretty obscure for cloudflared, and not sure I would be supporting it if I was CloudFlare. But for the sake of curiosity, I will add an armv6 image.

[mjudeikis]

I think armv6 is pretty obscure for cloudflared, and not sure I would be supporting it if I was CloudFlare. But for the sake of curiosity, I will add an armv6 image.

I agree. It just old devices laying around. But with chip shortage globally I see some of the people being reluctant to upgrade hardware as they just can't get hand on on enough new ones...

[maggie44]

Should be there in about 25-30 minutes.

[mjudeikis]

Just for those who interested. Works like a charm.
Just some know issue with https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size on older kernels

p.s. took changed from your fork and added to our managed fork for builds https://github.com/synpse-hq/cloudflared
I think we gonna try to maintain this build until CF makes support for it for our users :/
https://github.com/synpse-hq/cloudflared/pkgs/container/cloudflared

[ranrinc]

@Maggie0002 just wondering if your container will work on RPI4 running Debian 64?

[maggie44]

@Maggie0002 just wondering if your container will work on RPI4 running Debian 64?

Yes, Raspberry PI 64 is linux/arm64 and it builds a container for this. When you pull the container from that device, it will fetch the linux/arm64 automatically.

I started putting together a GitHub action to keep the containers on that repo up to date with the CloudFlare versions automatically, so it can be used as an alternative. Next time there is a CloudFlared release I will be able to give it a test. Ideally though of course CloudFlareD maintainers will be able to get the build processes in their own builds, as pulling containers for production servers from a third-party is ill-advised.

[ranrinc]

@Maggie0002 thanks for this it work like a charms. Any possibility in retrieving it with version instead latest? something like this docker pull ghcr.io/maggie0002/cloudflared:2022.5.1-ma

NVM.. found it its 0.0.9

[maggie44]

@Maggie0002 thanks for this it work like a charms. Any possibility in retrieving it with version instead latest? something like this docker pull ghcr.io/maggie0002/cloudflared:2022.5.1-ma

NVM.. found it its 0.0.9

Yes, the new workflow when it is running will create a new tag for each release within 24 hours of the Cloudflared release, and will have the same tag as Cloudflared (i.e. docker pull ghcr.io/maggie0002/cloudflared:2022.5.2). It hasn't done its first run yet though, but should do on the next release, and from then on will continue with the same tagging convention. Right now they are v0.0.x as you mentioned.

[maggie44]

Seems the auto-building is mostly working: https://github.com/maggie0002/cloudflared-docker-multiarch/pkgs/container/cloudflared

Only issue seems to be that the current 2022.5.2 build will tell you that it is 2022.5.1 when you pass version (it is actually 2022.5.2). It is because I hadn't realised that the version string is pulled from the repository tags, and not from within the code. It should be fixed for the next release: 2022.5.3 and the auto-built future releases.

[maggie44]

@Maggie0002 thanks for this it work like a charms. Any possibility in retrieving it with version instead latest? something like this docker pull ghcr.io/maggie0002/cloudflared:2022.5.1-ma

2022.5.3 built properly, so can now fetch the multi-arch cloudflared multi-arch images with docker pull ghcr.io/maggie0002/cloudflared:2022.5.3 or docker pull ghcr.io/maggie0002/cloudflared:latest. Each new release will be automatically built at midnight and follow the same format. Here is the list of built images:

https://github.com/maggie0002/cloudflared-docker-multiarch/pkgs/container/cloudflared

[SayakMukhopadhyay]

Would a PR help in getting this merged quicker @sudarshan-reddy ?
EDIT: Just noticed that this PR #618 probably fixes this.

[rhullah]

@Maggie0002 Thank you for providing the multi-arch images. It is very much needed. Previously I was able to pull 2022.5.3 but now when I attempt to pull I get the following error:

$ docker pull ghcr.io/maggie0002/cloudflared:latest
Trying to pull ghcr.io/maggie0002/cloudflared:latest...
Error: parsing image configuration: unable to retrieve auth token: invalid username/password: unauthorized: unauthenticated: User cannot be authenticated with the token provided.

Seems like it's wanting authentication? As far as I know, I am not providing a token.

[maggie44]

docker pull ghcr.io/maggie0002/cloudflared:latest

I just tried docker pull ghcr.io/maggie0002/cloudflared:latest from a brand new VPS and it pulled without any issues. It may be an issue with your setup.

[rhullah]

Hmm...it is now pulling correctly again. Maybe it was a GitHub issue. 🤷‍♂️ Sorry for the interruption.

[rhullah]

Hmm, like I said it did pull in v2022.6.3, but in the execution logs it shows: INF Version 2022.6.2-ma-3-g822982e4

[maggie44]

Hmm, like I said it did pull in v2022.6.3, but in the execution logs it shows: INF Version 2022.6.2-ma-3-g822982e4

Likely because there was a build error on that build that threw the tags off a little. This Cloudflared repository doesn't include the version numbers in the code base, the build draws it from the GitHub tags added to the repo. It's not ideal, because it relies on the GitHub repo which is not inherently attached to the code base and subsequently building your own version outside of the repo, or forking throws off the build versions (or doesn't build at all without errors).

On the build for that version, there was a change made to a file in a GitHub workflow (https://github.com/cloudflare/cloudflared/commits/master/.github/workflows). Because the fork was trying to merge changes to a GitHub workflow, it needed authorisation as by default it won't allow tampering with things that interact with the GitHub runners or secret keys etc. Which meant for that build I had to trigger it manually, and likely why it didn't pull the CloudflareD tags across properly.

Long story short, GitHub tags = pain. 2022.6.2-ma-3-g822982e4 = 2022.6.3. Next release should reflect the proper tag again.

[rhullah]

Thanks for the clarification and for your work on this. It is very much appreciated!

[maggie44]

Thanks for the clarification and for your work on this. It is very much appreciated!

Quick update, looks like the new build is working as expected:

cloudflared version 2022.7.0-6-ge218fec2 (built 2022-07-06-0054 UTC)

6 is th number of commits on top of this original repo, and ge218fec2 the last commit merged to the build. These are auto generated by GitHub, and I left them on the end of the usual version to indicate that it isn't the official CloudFlared image, but has had modifications (as outlined in the posts above and in the commits of the repo, i.e. smaller image size and new workflows to build for multi arch).

[rhullah]

I appreciate the small image size you've achieved working FROM scratch. Though, if wanting to utilize docker health checks, scratch makes it hard to do that as there's no extra tools in the container. It seems as the recommended way to health check cloudflared is a web request to /ready. It would be nice to have at least curl or wget?

[SayakMukhopadhyay]

I appreciate the small image size you've achieved working FROM scratch. Though, if wanting to utilize docker health checks, scratch makes it hard to do that as there's no extra tools in the container. It seems as the recommended way to health check cloudflared is a web request to /ready. It would be nice to have at least curl or wget?

I actually have an image myself with debian as the base image (since I need all the tools for debugging the container). You can try and use that https://github.com/SayakMukhopadhyay/cloudflared-docker-multiarch/. Its based on @Maggie0002 's image.

[maggie44]

I appreciate the small image size you've achieved working FROM scratch. Though, if wanting to utilize docker health checks, scratch makes it hard to do that as there's no extra tools in the container. It seems as the recommended way to health check cloudflared is a web request to /ready. It would be nice to have at least curl or wget?

You could fork it and change the image it builds in. It should run with pretty much anything, including Alpine. You could then install wget or curl in the Dockerfile. If you really wanted to optimise for size, you could try tiny-curl in to the scratch image: https://github.com/maggie0002/docker-tiny-curl

Although I'm not sure what need there would be for the healthcheck without knowing specific use cases. I think Cloudflared is fairly self contained and auto-retries connections until it finds one. If it exposes ports of paths served by the CloudflareD executable then then should be accessible from other containers too, which would allow whatever is managing your health checks to hit the path directly.

Also depending on use case, it is worth considering health checking against the actual site you want to see served, for example checking the http return code of http://your-website as that is the ultimate test of it functioning without issue.

@SayakMukhopadhyay it would be helpful to elaborate on your use cases too. I think the scratch image makes most sense for this repo so not looking to change it for now, but if I was Cloudflare I would be using this thread as a test balloon for build processes and your input on use cases can feed in to whatever they may have in the pipeline.

[SayakMukhopadhyay]

it would be helpful to elaborate on your use cases too

@Maggie0002 I am building a kubernetes operator for tunnels and for now I am hardcoding the container image, although I plan to make it configurable. I created my own image as I needed all the tools including root access to debug some tunnel issues I was facing. I don't plan to release with that image though and you are correct, a lighter image would be nice. Also, in my experience, I didn't need to use health probes so curl is not all that necessary unless you REALLY WANT to use health checks. So, in my opinion 2 things are necessary

1. ca-certificates
2. curl

Other than that everything is just good to have.

but if I was Cloudflare I would be using this thread as a test balloon for build processes and your input on use cases can feed in to whatever they may have in the pipeline.

This I am quite surprised about. This has been there for a while and a PR also exists which tries to fix this issue, albeit in a different way. I have actually opened a forum thread in the hope that the devs might have missed the notifications in here. As for the test balloon for their build process, I will reiterate what I mentioned in PR

...This PR releases the container as a Github actions workflow but I haven't seen anything regarding how you folks release the containers right now. The only thing I have seen is

cloudflared/Makefile

Lines 131 to 133 in 135c8e6

	container:
	docker build --build-arg=TARGET_ARCH=$(TARGET_ARCH) --build-arg=TARGET_OS=$(TARGET_OS) -t cloudflare/cloudflared-$(TARGET_OS)-$(TARGET_ARCH):"$(VERSION)" .

If you folks are using some internal tooling to run make container somehow then that needs to be adapted. Else, instead of creating an action, this PR can be modified to change the Makefile itself so that it runs docker buildx and not docker build (example is in https://github.com/SayakMukhopadhyay/cloudflared-docker-multiarch/tree/docker-mp).

[abelinkinbio]

Hi All - Thank you for all the discussion and documented workarounds while we worked to get this better scoped on our end. With that, this isn't something we'll be picking up in the next handful of sprints, but we are aiming to tackle this by the end of the quarter (mid/late September). We have a few other exciting features we're working on in the meantime, but will be sure to update this thread first in the event we're able to pick this work up sooner.

[SayakMukhopadhyay]

Thanks for the update @abelinkinbio. Would you be able to give some info if you folks are considering using GitHub Actions as your CI platform or would you continue using your internal CI? It would help us by not using our efforts in the wrong direction.

[sudarshan-reddy]

We’d most likely gravitate towards internal CI. That being said, that’s definitely not set in stone. If there’s a good case for gh actions, we can all definitely deliberate it.

[SayakMukhopadhyay]

I personally prefer Actions and I believe, most of the community does too. The advantage of actions is that the CI becomes available for the community to track and improve upon. If you go that route @maggie0002's fork or the related PR #618 would be great examples of how to achieve it.

If for whatever reason, you folks want to keep your existing CI, look at https://github.com/SayakMukhopadhyay/cloudflared-docker-multiarch/tree/docker-mp where I attempted to change the makefile to update the way the container was created. This should be a good starting point for this.

[sudarshan-reddy]

There's now a dedicated arm64 image for cloudflared on Cloudflare's dockerhub: https://hub.docker.com/layers/cloudflared/cloudflare/cloudflared/latest/images/sha256-68706b30662250384737e069faa1eeb146fa2c98e2ae3a3d4be8e4394f43fba3?context=explore

[SayakMukhopadhyay]

There's now a dedicated arm64 image for cloudflared on Cloudflare's dockerhub: https://hub.docker.com/layers/cloudflared/cloudflare/cloudflared/latest/images/sha256-68706b30662250384737e069faa1eeb146fa2c98e2ae3a3d4be8e4394f43fba3?context=explore

Thanks a lot for taking care of this. I have just one suggestion. I see that the release images are tagged with arm and amd separately. It would be great if a single image is generated for a version containing both the architectures, like how latest is. This will have the benefit of having people like me who pin the image version and uses multiple architectures.

[sudarshan-reddy]

I'm looking into it. This is something that we'll bring out in the near future. The temporary workaround I'd recommend until then is to use latest if that's okay.

Out of curiosity, why do you pin a certain version of cloudflared?

[markis]

There's now a dedicated arm64 image for cloudflared on Cloudflare's dockerhub: https://hub.docker.com/layers/cloudflared/cloudflare/cloudflared/latest/images/sha256-68706b30662250384737e069faa1eeb146fa2c98e2ae3a3d4be8e4394f43fba3?context=explore

This is great! Thank you so much! Could we also get linux/arm/v7 as well?

[SayakMukhopadhyay]

I'm looking into it. This is something that we'll bring out in the near future.

Thanks :)

Out of curiosity, why do you pin a certain version of cloudflared?

Well, I am using containers in a kubernetes cluster and I try to keep them as immutable as possible. Hence, pinning a version. On that note, I noticed that the containers are also released with the commit sha as a tag and that image contains both arm and amd in a single image.

[sudarshan-reddy]

True. That's an alias to latest and a reasonably good workaround to pinning.

[firecow]

Can we push an image, containing both arch'es, please 😄

[sudarshan-reddy]

The ‘latest’ tag already has this.

[firecow]

@sudarshan-reddy We use renovatebot to bump our dependencies and doesn't run latest in production.
So it would be epic to have it on the exact version images as well.

You have in fact broken our current upgrade routine, by changing the versioning scheme 😄

I thought cloudflared was still on 2022.8.0, because I rely so heavily on renovatebot..

[SayakMukhopadhyay]

We use renovatebot to bump our dependencies and doesn't run latest in production.

Same here for me. I always run non-latest tags as that prevents any breaking changes from creeping in.

[sudarshan-reddy]

docker pull cloudflare/cloudflared:2022.10.2 should now work without needing a suffix.

Let me know if this solves your problems.