Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Add pre-compiled ARM binaries for releases #1379
I wanted to use concourse ci workers on my raspberry pi 3 for doing QA with x86 firmware.
Best Regards, Zaolin
@brownjohnf we have some getting started with contributing under our quickstart guide for engineer contributors: https://github.com/concourse/concourse/blob/master/CONTRIBUTING.md
Not sure if that has enough specific detail to get you started though...
/cc @vito to see if he has any more resources
I also tried to take a look at that and to be honest I struggled quite a bit.
I looked both at the existing pipelines at https://github.com/concourse/pipelines to run with my own concourse setup and tried to build things locally on my machine (and various combinations thereof). And didn't get very far... I have a local concourse setup I can experiment with, it just seems to duplicate the whole pipeline touches on so many custom things (versioning in s3, all the bosh stuff maybe?) that shouldn't be necessary just to build and yet they seem so integrated.
So any help would really be appreciated. I might be able to help out with various things in that context since we do have a new requirement for building arm stuff here. The concrete thing I need really is a concourse binary built with GOARCH=arm.
There are armv7 and aarch64 builds available that work with caveats.
I had a go on this and I went down a deep rabbit hole but I finally managed to cross compile both an
So I took the path of building everything from scratch using a bash script that bootstraps a concourse binary. The bash script more or less follows what the pipeline does. The idea is that this stage0 concourse binary can then be used on an actual arm machine to run the normal build pipeline and produce the concourse binary.
There are some build issues with both armv7 and aarch64 of which I think I fixed all of them and some of them are already upstreamed:
You can find the bootstrap repo here https://github.com/resin-io/concourse-arm. To cross compile it all you need is a linux system with a working Go and Docker installation. You can also find pre-compiled binaries in the Github releases page.
I haven't yet tested the arm64 version but I believe it will work without issues.
Unfortunately there are still some runtime issues for armv7 builds which are due to a bug of golang's syscall module. If you attempt to run concourse on 32bit hardware with user namespaces enabled you'll get:
A potential way forward (fixing go aside) would be to use runc's libraries that has already fixed this issue opencontainers/runc#1819 . It's still the wrong datatype but at least it doesn't overflow.
The official concourse binaries currently ship with the following resource types embedded:
Of those I only cross-compiled:
This means that the current binaries are not yet able to run the full concourse pipeline and build itself since it uses more resource types but you can definitely use it if your pipelines don't need those resource types. Depending on the resource type, cross compiling could be as simple as switching the base image. I expect bosh to be the most tricky one.
Currently the build process changes the projects in a way that breaks normal amd64 builds. So there is work left to be done to have a multi-arch build process that can be upstreamed.
I just pushed a couple of PRs for the int overflow issue
and published a new arm binary as v3.14.1-rc2 :)
It now initialises correctly and I can load the UI. Haven't done further testing or actual builds yet
The actual task I'm trying to solve is to make concourse build (and publish) docker images for me.
This is a bit new to me but I'll keep you posted once I get that working (and especially if I don't get it working :-)). Thanks for the offer, I just need to find the time to read up on the multi arch docker thing a bit first.
I have some updates on my multi arch journey :-)
One thing I thought would be great for this whole thing is to build all the images involved multi-arch aware. I.e. the images I want to use in concourse should exist for both amd64 and aarch64 architectures and this is easily accomplished by using alpine based images. Wherever they are built the architecture is properly propagated and the right repos are set up and the packages for the right architecture are installed. I don't know how this works for other base images (I do expect some issues here though, and at some point I think it'll be inevitable to have one Dockerfile per architecture or lots of if statements in the Dockerfiles).
I build a modified dind image that supports multi arch which in turn can be used to build other docker images. It's a bit of a chicken and egg problem, but once the initial image is bootstrapped you're ready to go.
Then I made a concourse pipeline to use this image with to tasks, one for amd64 (i.e. the normal job) building a xxx:amd64 image and one that has a tag specified so the right worker is used (aarch64 is the tag I use for the aarch64 workers) to build a xxx:aarch64 image. There's a third task building a manifest and pushing the manifest for both images (xxx -> xxx:amd64, xxx:aarch64).
I'm quite new to the multi-arch stuff in docker, but it seems to me that the only viable way forward is to add architecture tags to all images anyway (so the manifests can refer to them later).
There's one thing I noticed: when concourse pulls the image for the aarch64 task, i.e. the one that is run on the aarch64 host, it seems to ask for the amd64 image explicitly (via the architecture setting in docker). So my assumption is that the code somewhere has some custom logic that falls back to the amd64 architecture rather than propagating the architecture of the worker in a generic way. I'll double check that if I find the time, it's just a theory so far. When I pull the image without tags from inside the build container the right image is pulled (via the docker binary). The easy workaround for now is to add aarch64 tags to both the task and to the image used in the task.
Ideally this would be integrated in the docker image resource somehow, but I don't see how the manifest stuff can fit there, maybe a docker-manifest resource. If both docker-image resource and docker-manifest resources had the same set of architecture tags for which they can check this might work nicely (the docker-image resource has a multi-arch flag that adds these tags and the manifest resource uses the outputs and builds manifests for all known architectures).
I'll try to post updates when I have time to look for the architecture propagation code.
I have a short update on my previous ramblings about the architecture propagation code being buggy. That is not true. It works as intended, I had mistagged my images. It happens a lot with building multi-arch images :-), it seems to be the main challenge to keep things consistent.
Anyway. I've been using this rogue worker for aarch64 for almost three months from the binary and so far it's been very stable and worked well together with the concourse-web instance (4.0.0).
So is there any chance of producing official arm 64 binaries? And is there anything I can do to make that easier (even with my limited understanding of the concourse build process)?
Hey @neumayer, I followed your journey with great interest. Currently I think of using concourse on my raspberry pi 3. So stumbled over this issue. Can you give use a short description or mini tutorial how you achieved your goal? Perhaps we could manage to create a PR to integrate the changes into the build pipeline.