fix: Cross-compilation to ARM64

[raskyld]

Introduction

We should be able to build lemmy from amd64 to arm64.

All the heavy work has been off-loaded:

1. The cross-toolchain is now built from raskyld/lemmy-cross-toolchains,
   a. in the future we could/SHOULD migrate it to the LemmyNet orga
   b. it does NOT use musl but the more traditional (even though messy) glibc.
2. The compilation is done on native amd64, so we have respectable performance.

The only moment virtualisation would be involved is when building the runner container but it does really trivial things like downloading apt packages and creating non-root user.

I am not familiar with woodpecker, but I can give it a try.

Issues

Solves #3102.

[raskyld]

Now that I am about to update woodpecker CI, we need to take a decision: the cross-compilation step must be executed on an amd64 platform but the actual runner (with apt to add libs) need to be run on arm64 using the QEMU builtin capabilities of buildx.

This means I cannot just invoke the wookpecker image https://codeberg.org/woodpecker-plugins/docker-buildx as-is. I first need to make an intermediate image that will build lemmy_server (the builder from our Dockerfile but as a standalone) using the cross-toolchain.

Maybe named and published as: lemmy-builder-arm64 which would contain the precompiled lemmy_server ?

Then creating the arm64 runner by copying the binary from lemmy-builder-arm64, buildx would spawn a qemu VM for that but as I said earlier the tasks performed by the VM would be really small.

Are you fine with that plan? @Nutomic @dessalines.

[raskyld]

Also, I noted your willingness to avoid dependencies with MS / GitHub.
The current cross-toolchains images are stored on ghcr which is GH's registry.
I can also hosts those images on a Scaleway registry server if that would be better for you.

Your call!

[raskyld]

I think that should do! I will spawn a GH Codespace (bcs I run on arm) and try to build the images :)

[raskyld]

Ok, I had a typo + variables are not expanded in COPY --from= arg... so I will need to tricks around it...

[raskyld]

Fixed! :)

[dessalines]

I've started a tag to make sure that our builders will be able to handle this, and that everything works well with woodpecker.

[raskyld]

I merged main as the pipeline is now passing!

[dessalines]

I'll start another tag build to make sure it passes.

@raskyld in the future, avoid rebase, as it rewrites history, and makes collaboration difficult. Just merge from main.

[raskyld]

I'll start another tag build to make sure it passes.

@raskyld in the future, avoid rebase, as it rewrites history, and makes collaboration difficult. Just merge from main.

Yep! It's what I did 71c8f4e after you asked me to

[dessalines]

K, another tag build started. 🤞

[dessalines]

@raskyld Any idea what this error might be?

https://woodpecker.join-lemmy.org/repos/129/pipeline/4012/22

[raskyld]

Yeah that's because $TARGETARCH resolves to arm64 not just arm and I fixed it on one machine but didn't commit it on the other one (I had to switch back and forth between arm and amd machines). I comited the fix

[dessalines]

Thanks, I've started another build.

[raskyld]

I needed to use a temporary layer to be able to copy --from= it, but I used the wrong base image... 🙃 Sorry!
The last commit should fix it.

[dessalines]

Failed again.

[raskyld]

Gosh, I hate this "eventually working" workflow. Sorry @dessalines !
I will check it during the day, it seems like the builder image is indeed pushed there: https://hub.docker.com/r/dessalines/lemmy-builder-arm64/tags, it also contains the linux/amd64 platform, so why it fails to download the image is unclear for me atm

[raskyld]

The buildkit log:

#10 [linux/arm64 internal] load metadata for docker.io/********/lemmy-builder-arm64:0.19.0-alpha.10
#10 ERROR: no match for platform in manifest: not found

seems to say it tries to download the builder for linux/arm64 but the builder is a cross compilation toolchain meant to be run on linux/amd64, that's why it doesn't exist for linux/arm64. I think I just need to enforce the correct platform in the FROM clause using --platform=linux/amd64, I am gonna try to push a commit with this change.

[dessalines]

Sweet, nother tag started.

BTW don't worry about these. I've had like 20 builds failed when trying to tweak and get CI working. Once it finally works, then you usually don't need to worry about it again.

[dessalines]

Here's the error: https://woodpecker.join-lemmy.org/repos/129/pipeline/4046/22

[dessalines]

Woohoo it worked!

https://hub.docker.com/layers/dessalines/lemmy/0.19.0-alpha.12/images/sha256-30a8409830ea8468745ede0cd349770cf02085ebbb6ef220fc9679026636bd3c?context=explore

[raskyld]

To be sure the binary has been linked correctly, we should do some integration test using the resulting docker image. Any hint on how to do that? I have an arm machine so I can test it

[dessalines]

There's not an easy way to do it with the binary, but you can pull down and run the tests on your arm machine:

git clone https://github.com/LemmyNet/lemmy
cd lemmy/api_tests
./run-federation-test.sh

[kroese]

This Dockerfile renames lemmy_server to lemmy and then sets the entrypoint to lemmy_server?

I cannot imagine that that would work (but I havent tried it to be honest).

[raskyld]

You are right, that's why I wanted to test running the image, it indeed cannot work.

Also we exchanged mails with @kroese, he pointed out that we could avoid using a two-step build process:

If we use FROM --platform=linux/amd64 on the cross toolchain stage, the builder should actually run the build process on native host but run the rest in QEMU, so we won't need two images anymore.

This would make the main Dockerfile more heavy but would avoid the burden of building two images.

[kroese]

I now created a pull request ( #4202 ) which fixes the entrypoint and removes the need for the second image.