Support linux/arm

[bradrydzewski]

this issue will discuss a strategy for an official arm image for drone.

• ability to define platform in yaml
• ability to route builds to agents by platform
• publish linux/arm agent
• publish linux/arm64 agent
• publish linux/arm git plugin
• publish linux/arm64 git plugin
• publish linux/arm docker plugin
• publish linux/arm64 docker plugin
• develop strategy for defining multi-platform yaml configurations

cc @firecyberice @sokoow @djmaze

[sokoow]

Cool initiative, +1. One thing to note here is that currently, there are two main arm platforms in use, older 32 bit one is called armhf (RPi 1 + 2, odroid U/XU), and newer is called aarch64/arm64 (RPi 3, odroid C).

I think that by meaning 'arm', you mean 'armhf' here?

[bradrydzewski]

@sokoow yes I think starting with armhf makes sense, and getting the drone agent and git plugin ported would be the most important.

I'm thinking one of the first things we can do is create a const.go file to store values that may be architecture specific, such as busybox for the ambassador image, or plugins/git for the git clone image.

sample const.go file:

const (
  BaseImage = "busybox"
  GitImage = "plugins/git"
)

sample const_linux_arm.go file:

const (
  BaseImage = "armhf/alpine"
  GitImage = "plugins/git:arm"
)

[bradrydzewski]

in response to #1766 (comment)

Sqlite3 seems currently not to work maybe because of the static build? :-)
The binary has been cross-compiled on amd64 with make deps gen build-cross.
I suppose even the docker image can be built on amd64 as there is no RUN defined.

@firecyberice drone uses CGO to compile with sqlite. This means that if you compile on amd64 for arm that CGO will be disabled and as a result, the C sqlite library won't be compiled into the binary.

so in order to build an official drone arm image, with sqlite support and server support, we would need to build drone on an arm environment. We should be able to build an arm image by modifying the Dockerfile.armhf

FROM centurylink/ca-certs

EXPOSE 8000

# default to mysql?
ENV DATABASE_DRIVER=mysql
ENV GODEBUG=netdns=go

ADD release/linux/arm/drone /drone

ENTRYPOINT ["/drone"]
CMD ["server"]

this will still require some agent modifications to change the ambassador container, as mentioned above. And it will still require arm versions of at least the git plugin. But perhaps it is a good start?

[alkar]

I setup drone on ARM over the weekend. It wasn't so hard and I'm planning to document the process sometime this week. Most of the needed work seems to be already in place. In summary:

1. I had to build drone for ARM in order to replace the ambassador image (busybox:latest). I could have used the official cross-compiled binary (sacrificing SQLite support which should be alright in a production environment) if the ambassador image was parametrised.
2. I built three of the plugins for this POC: drone-docker, drone-git and drone-slack. Again, nothing special other than picking the appropriate base ARM images in their Dockerfiles.
3. Using the --namespace and --privileged on the agent, I was able to start drone and write .drone.yml files without having to override plugin images inline.

If you want to take a look, this is a simple example. All the components used are part of the organisation.

I think the main challenge is finding proper ARM base images that are dependable (eg. I had to use armhfbuild/docker:1.10-dind for the drone-docker plugin, since I couldn't find anything latest than that). There are various organisations on dockerhub offering ARM images but I believe all of them are working on a best-effort basis.

It also looks like it'd be possible to integrate ARM agents and restrict builds on certain architectures, so this could potentially be part of the official pipelines (using constraints and the --docker-os / --docker-arch flags on the agent).

[tboerger]

@alkar you can also see a lot of arm plugins at https://github.com/armhf-drone-plugins

[bradrydzewski]

I had to build drone for ARM in order to replace the ambassador image (busybox:latest). I could have used the official cross-compiled binary (sacrificing SQLite support which should be alright in a production environment) if the ambassador image was parametrised.

@alkar if you are interested, I would love a pull request to extract the ambassador image name as a constant into separate files, using build tags as described in #1767 (comment)

It also looks like it'd be possible to integrate ARM agents and restrict builds on certain architectures, so this could potentially be part of the official pipelines (using constraints and the --docker-os / --docker-arch flags on the agent).

yes these parameters are in place so that we can eventually have separate queues per operating system and architecture. This should land in 0.6

[bradrydzewski]

I built three of the plugins for this POC: drone-docker, drone-git and drone-slack. Again, nothing special other than picking the appropriate base ARM images in their Dockerfiles.

I'm guessing for slack we could use centurylink/ca-certs as the base image and cross-compile the slack binary for arm. This image is completely empty except for certificates, and the slack plugin doesn't have any external dependencies.

The Git and Docker plugins will be a bit more tricky since we'll need to install git and other software when we build the image. This means we'll therefore need to build the images on arm machines, which I don't have setup yet.

So we can possibly setup automated builds for Slack arm images today using my existing build infrastructure. The others we might have to manually push in the short term.

[alkar]

@alkar you can also see a lot of arm plugins at https://github.com/armhf-drone-plugins

@tboerger strange I didn't come across those, thanks!

@alkar if you are interested, I would love a pull request to extract the ambassador image name as a constant into separate files, using build tags as described in #1767 (comment)

@bradrydzewski I think I might be able to throw in a few hours this weekend.

I'm guessing for slack we could use centurylink/ca-certs as the base image and compile the slack binary for arm. This image is completely empty except for certificates, and the slack plugin doesn't have any external dependencies.

That sounds like an easy win!

The Git and Docker plugins will be a bit more tricky since we'll need to install git and other software when we build the image. This means we'll therefore need to build the images on arm machines, which I don't have setup yet.

I'd be more than happy to donate an agent node to the cause.

[djmaze]

I think the main challenge is finding proper ARM base images that are dependable (eg. I had to use armhfbuild/docker:1.10-dind for the drone-docker plugin, since I couldn't find anything latest than that).

I will have a look at updating that image for 1.12 tonight.

[firecyberice]

@alkar
There is also https://github.com/armdrone with some older versions of drone.

I like your idea in the last paragraph for working with constraints on docker-arch :-)

I think all drone plugins should use the same base image e.g. https://hub.docker.com/r/armhf/alpine/ for arm and the official counterpart https://hub.docker.com/_/alpine/ for amd64.

Or maybe use a self build image with ca-certs that exists in the same namespace like the plugins and can also be used as ambassador image.

[tboerger]

The best option should be to define arch specific default ambassador images but to also provide a flash for the agent to override the image

[alkar]

@djmaze that's great, thanks!

@firecyberice I saw that but I thought it's based off the 0.4 version of drone?

And I agree that drone plugins should use the respective images for ARM, but as I said, that might be a bit difficult because someone would have to provide those base images and make sure they are equivalent to the x86 images.

[firecyberice]

@alkar The images mentioned in my previous post are from the upstream (official) docker library even the arm ones :)

[djmaze]

@alkar FYI, armhfbuild/docker is now updated for 1.12.

[bradrydzewski]

I just submitted pull request #1771 that allows us to change the ambassador image based on os and architecture. Can I get a code review just to make sure I'm using the correct armhf ambassador image?

This pull request will allow you to start the agent with the DOCKER_OS and DOCKER_ARCH parameters ad will choose the correct ambassador accordingly:

docker run -e DOCKER_OS=linux -e DOCKER_ARCH=arm drone/drone:0.5 agent

Once merged, this should allow you to use the official arm binary to start the agent on an arm server, connect to a drone server on an amd64 machine, and run builds. Note that this will still require you to provide your own arm plugin implementations:

platform: linux/arm

plugins:
  clone:
    image: <name of custom drone git plugin compiled for arm>

... but hopefully this is an important first step toward native arm support

[alkar]

@djmaze thanks for the update, much appreciated!

@bradrydzewski I just had a look at your PR, it's a good starting point indeed, thanks!

I'm afraid I won't have time to test this out until next week but I'll post back when I do so.

[alkar]

This is a bit of a late response but amidst holidays and a busy work schedule, I've finally managed to put it all together in a (hopefully) user-friendly format in armswarm/drone-on-arm.

I've now got a drone setup on ARM which works without any hacks and which uses the most recent versions of base ARM images.

[bradrydzewski]

anyone have some quick / easy instructions to install qemu-arm-static on osx? homebrew maybe? I came across this blog post, which could help me quickly port the git and docker plugins
http://blog.hypriot.com/post/first-touch-down-with-docker-for-mac/

[firecyberice]

@DieterReuter can you give a hint how to setup qemu on a mac?

[firecyberice]

@bradrydzewski I have an idea for providing the plugins for another architecture.
If the DRONE_PLUGIN_NAMESPACE setting can be used as prefix for all plugins even git and the base ambassador image then a Dockerhub namespace can easily provide all plugins for that specific architecture.
Maybe the ambassador image should be an alpine image and named e.g plugins/ambassador on all platforms and then used as base image for all plugins.
Right now the git plugin docker image is hardcoded as plugins/git:latest so it does not work on arm.

[bradrydzewski]

@firecyberice I'm going to start playing around with this in the coming weeks. I might do something similar to what you suggested, providing DRONE_IMAGE_GIT=plugins/git and DRONE_IMAGE_AMBASSADOR=busybox as the defaults.

In the Dockerfile.arm, for example, we could override these values to use arm specific images.

[alkar]

I might be completely off here (been a while since I worked on drone & ARM) but I thought the namespace flag works already? I mean I managed to setup drone on ARM using a custom namespace (IIRC the ambassador image was hardcoded, however).

Git and Drone plugins should already work using namespace

[bradrydzewski]

@alkar we removed namespace a few weeks ago. You are now required to provide the full plugin name, for example, instead of image: git you must specify image: plugins/git

the reason we stopped automatically prefixing image names is because it caused a bit of confusion among users. For example image: docker could be interpreted as plugins/docker or library/docker.

[vielmetti]

Any progress on this since November?

I will note that the requirement for a busybox image for aarch64 might be met by https://hub.docker.com/r/aarch64/busybox/ as mentioned in the original item.

[maggicl]

For everyone out there, i managed to compile succesfully drone on arm using this custom Dockerfile.armhf:

FROM armhf/alpine:3.5

EXPOSE 8000
ENV DATABASE_DRIVER=sqlite3
ENV DATABASE_CONFIG=/var/lib/drone/drone.sqlite
ENV GODEBUG=netdns=go

RUN apk add --no-cache ca-certificates

ADD release/drone /drone

ENTRYPOINT ["/drone"]
CMD ["server"]

Firstly, you need to compile drone yourself. Instructions here are fine even for ARM.

Then, use the following docker-compose.yml as a template for yours. Consult the official documentation for more information.

version: '2'

services:
  drone-server:
    image: <your-built-image-tag>
    ports:
      - 23456:8000
    volumes:
      - drone:/var/lib/drone
    restart: always
    environment:
      - DRONE_OPEN=false
      - DRONE_SECRET=${DRONE_SECRET}
      - DRONE_ADMIN=<your-drone-username>

  drone-agent:
    image: <your-built-image-tag>
    command:
      - agent
      - --namespace=<your-docker-registry-userename>
      - --privileged=<your-built-docker-plugin-image-tag> # These lines are for the docker plugin
      - --privileged=<your-built-docker-plugin-image-tag>:*
      - --timeout=30m
      - --docker-os=linux # Declare explicitly os and architecture, otherwise it will default to x86-64
      - --docker-arch=arm
    restart: always
    depends_on: [ drone-server ]
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_SERVER=ws://drone-server:8000/ws/broker
      - DRONE_SECRET=${DRONE_SECRET}
      - DOCKER_OS=linux # Same as above
      - DOCKER_ARCH=arm

volumes:
   drone:

Then, build your git plugin. Clone this repo and edit Dockerfile.armhf FROM instruction using the more standard armhf/alpine:3.4 image. Build as shown in the repo the binary and then the docker image. In your .drone.yml files you should specify explicitly that you want to use your custom built plugin.

pipeline:
  clone:
    image: <your-built-git-plugin-image-tag>

The docker plugin is more tricky. You firstly need to build a dind (docker-in-docker) image that will be the base image of the plugin. Clone this repo (which is mine) and docker build it. Then clone the docker plugin repo and edit Dockerfile.armhf FROM instruction in order to use your custom built dind image. Build the plugin and tag it with the same name you gave in the drone agent --privileged option.

Hope you find this helpful.

[tboerger]

This is not really useful because drone lacks done stuff for multi arch support. Beside that somebody is already maintaining a drone port for arm

[maggicl]

The setup shown in my previous post works fine: it can run basic pipelines. The problem is that several images should be mantained and built in order to get the thing up and running.

[tboerger]

At https://discuss.drone.io/t/drone-ported-to-arm/55/10 you can see an old thread related to arm builds

[NicolasPetton]

Beside that somebody is already maintaining a drone port for arm

@tboerger do you have a link?

[tboerger]

@tboerger do you have a link?

Looks like they have not migrated it to 0.5 yet :(
https://github.com/armhf-docker-library/drone

[djmaze]

@NicolasPetton @tboerger Yes, this is currently a bit of a mess since we haven't updated the automated builds yet, but I manually built 0.5 versions of the most important images already. I just pushed the latest ones: armhfbuild/drone, armhfplugins/git and just now armhfplugins/docker.

I will update the code for the automated builds soon, when I find some time.

See armhfbuild/traefik for a working example where these are in use.

[firecyberice]

Maybe it is even possible to build multiarch Images so it does not matter on which platform the image is used because the docker-engine decides which arch to pull.

• Howto
• manifest-tool

[vielmetti]

Note that the portainer project has multiarch images to good effect.

[bradrydzewski]

Note that the portainer project has multiarch images to good effect.

Publishing mutli-arch images does not mean Drone supports multi-arch. If we built and published a Drone windows image, for example, Drone would still not be able to execute a pipeline for a windows project. This is being worked on, but requires code changes and other pre-requisites in place.

[bradrydzewski]

locking this issue. please direct message me on gitter for updates

[bradrydzewski]

I have started publishing linux/arm and linux/arm64 agents on dockerhub in preparation for the 0.8 release. See https://hub.docker.com/r/drone/agent/tags/

I will continue posting to this thread as I have more information to share, specifically with regards to official git and docker plugins for arm.

[bradrydzewski]

there are now official armhf images for the git and docker plugins. They are tagged as plugins/git:linux-arm and plugins/docker:linux-arm respectively.

clone:
  default:
    image: plugins/git:linux-arm
    depth: 50

pipeline:
  publish:
    image: plugins/docker:linux-arm
    repo: foo/bar
    dry_run: true

[bradrydzewski]

there are now official arm64 images for the git and docker plugins. They are tagged as plugins/git:linux-arm64 and plugins/docker:linux-arm64 respectively.

clone:
  default:
    image: plugins/git:linux-arm64
    depth: 50

pipeline:
  publish:
    image: plugins/docker:linux-arm64
    repo: foo/bar
    dry_run: true

[bradrydzewski]

I am closing this issue as complete. We now have official agent, git and docker images for armhf and aarch64 architectures compatible with drone 0.8 (release candidate should be available this week).

armhf and aarch64 agents:

docker run drone/agent:linux-arm
docker run drone/agent:linux-arm64

armhf git and docker plugins:

image: plugins/git:linux-arm
image: plugins/docker:linux-arm

aarch64 git and docker plugins:

image: plugins/git:linux-arm64
image: plugins/docker:linux-arm64

When using drone with multiple architectures you should specify the target architecture in your yaml file as demonstrated below

platform: linux/arm

platform: linux/arm64

Example yaml configuration for an arm pipeline:

platform: linux/arm

clone:
  default:
    image: plugins/git:linux-arm
    depth: 50

pipeline:
  build:
    image: arm32v6/alpine
    commands
      - uname -a

At this time I am not publishing the drone server for arm architectures. You can run the drone server on amd64 and connect agents running on arm architectures. I may reconsider in the future, but if you need to run the drone server on arm you will need to manually compile or download a community supported drone server image.

[djmaze]

@bradrydzewski This is really awesome, thank you for your work!

It would be even nicer if we pushed multiple-platform images for the plugins someday, so there would be no extra configuration needed.

Also, +1 for drone server on arm.

[bradrydzewski]

@djmaze thanks :) I definitely would like to push multi-platform images if they can simplify the tooling. Although maybe we could have a plugins/docker:multi-arch plugin that abstracts away the complexity ...

edit: looks like there are plans to support this capability in mainline docker docker/cli#138

[HelloEdit]

I'm a bit late but will drone server support the architecture of Raspberry Pi (arm) ?
(If it is already the case, I do not know absolutely where it is documented)

[bradrydzewski]

@HelloEdit we have experimental arm and arm64 images available for the agent only, as well as the git and docker plugins. The server image is amd64 only. The arm images are not yet officially supported (by me) and not yet documented. So if you would like to use them you will need to through this comment thread for usage details. I would start with #1767 (comment).

[atomi]

I built drone-server on an rpi a few weeks ago. I don't see why it shouldn't build on arm.

[HelloEdit]

I am not against the images of the community, but they are no longer update :(
Why just do not make available a Dockerfile.rpi ?

[bradrydzewski]

@HelloEdit these images are official and are updated frequently, on every commit to master

drone/agent:linux-arm
drone/agent:linux-arm64

plugins/git:linux-arm
plugins/git:linux-arm64
plugins/docker:linux-arm
plugins/docker:linux-arm64
plugins/gitter:linux-arm
plugins/gitter:linux-arm64
plugins/slack:linux-arm
plugins/slack:linux-arm64

The server is the only image for which an official Docker image is not provided, but you can build the drone server for arm using the existing Dockerfile in the root of the repository. No modifications to the Dockerfile are required.

Simply run the following commands on arm, from the root of the repository:

go get github.com/drone/drone/cmd/...
go build github.com/drone/drone/cmd/drone-server -o release/drone-server
docker build -t drone/drone .

When official documentation is available and official server images are available I will post updates here. In the mean time I am locking the issue since this discussion should probably be moved to our community channel at https://gitter.im/drone/drone