Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker images for ARM32v7 and ARM64v8 #5031

Merged
merged 1 commit into from Apr 15, 2019

Conversation

@johanneswuerbach
Copy link
Contributor

commented Dec 21, 2018

Build and push docker images for linux arm32v7 and arm64v8.

Fixes prometheus/promu#89

Building
The ARM images are building using binfmt_misc, which is the same mechanism also used by Docker for Mac to support building ARM images, some details are here https://www.ecliptik.com/Cross-Building-and-Running-Multi-Arch-Docker-Images/

While the circle docker executor doesn't natively support building ARM images, the machine executor has almost all the necessary dependencies pre-installed and allows privileged execution, which is required to setup binfmt_misc

The setup itself is done viadocker run --privileged linuxkit/binfmt:v0.6, which is the same script also used inside Docker for Mac. (Source https://github.com/linuxkit/linuxkit/tree/master/pkg/binfmt)

As the default machine image is fairly old and binfmt_misc requires a Kernel 4.8+, use a more recent machine image https://circleci.com/docs/2.0/configuration-reference/#machine

Publishing

The two ARM images would be published to prom/prometheus-arm32v7-linux and prom/prometheus-arm64v8-linux, so those 2 docker hub / quay.io repositories would need to be created.

To simplify image consumption a multi-arch manifest is created using https://docs.docker.com/edge/engine/reference/commandline/manifest/, which allows docker pull prometheus/prometheus to pull the respective amd64, arm32v7 and arm64v8 image automatically.

Requires prometheus/busybox#19 to have an arm32v7 and arm64v8 base image.

Test

Build images locally and pushed them to my registry using:

promu crossbuild -v -p "linux/amd64 linux/armv7 linux/arm64"
make docker DOCKER_REPO=johanneswuerbach
make docker-publish DOCKER_REPO=johanneswuerbach

Using them on macOS via Docker for Mac:

$ docker pull johanneswuerbach/prometheus:docker-arm && docker run --rm -it johanneswuerbach/prometheus:docker-arm --version
docker-arm: Pulling from johanneswuerbach/prometheus
Digest: sha256:c8212d84d2d79f32d9b824d9235d3b4ab178e21c3a762a643c8abe12145a2d22
Status: Image is up to date for johanneswuerbach/prometheus:docker-arm
prometheus, version 2.6.0 (branch: master, revision: 2e725a195a17155dfd1e172a1e1205fc7d6986ec)
  build user:       root@682d9280fbbf
  build date:       20181221-21:56:26
  go version:       go1.11.4

Using them on a Raspberry Pi 3:

$ docker pull johanneswuerbach/prometheus:docker-arm && docker run --rm -it johanneswuerbach/prometheus:docker-arm --version
docker-arm: Pulling from johanneswuerbach/prometheus
Digest: sha256:c8212d84d2d79f32d9b824d9235d3b4ab178e21c3a762a643c8abe12145a2d22
Status: Image is up to date for johanneswuerbach/prometheus:docker-arm
prometheus, version 2.6.0 (branch: master, revision: 2e725a195a17155dfd1e172a1e1205fc7d6986ec)
  build user:       root@682d9280fbbf
  build date:       20181221-22:07:33
  go version:       go1.11.4

while the current latest fails as expected:

$ docker pull prom/prometheus && docker run --rm -it prom/prometheus --version
Using default tag: latest
latest: Pulling from prom/prometheus
Digest: sha256:1ffbf5d3c6476384905e8f57c98ac0611f328af68bedb909ec3f350d7e18b134
Status: Image is up to date for prom/prometheus:latest
standard_init_linux.go:190: exec user process caused "exec format error"

Feedback more then welcome :-)

@brian-brazil

This comment has been minimized.

Copy link
Member

commented Dec 21, 2018

This only seems to be creating docker images, this should be consistent with other parts of our builds such as tarballs on github. We should also avoid providing 32bit versions of Prometheus, as that won't work well.

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Dec 21, 2018

@brian-brazil I'm not entirely sure I can follow. This PR mainly builds and publishes the already existing linux/armv7 (32bit) and linux/arm64 as docker images and the tarballs already exist for those.

The main purpose of this PR is to provide docker images for architectures besides amd64 (the only one provided today) as requested for example here prometheus/promu#89

I added ARM images here as a starting point, but other architectures should be fairly easy to add.

avoid providing 32bit versions of Prometheus

The linux/armv7 binary is already being published today and this mainly makes using prometheus easier on a Raspberry Pi 3 as now users can just pull prom/prometheus and don't have to fallback to a prom raspberry fork like https://hub.docker.com/r/napnap75/rpi-prometheus/, which is actually the main reason why I created this PR.

@discordianfish

This comment has been minimized.

Copy link
Member

commented Dec 22, 2018

Nice, LGTM but see my code comment.

Makefile.common Outdated Show resolved Hide resolved
@brian-brazil

This comment has been minimized.

Copy link
Member

commented Dec 22, 2018

This PR mainly builds and publishes the already existing linux/armv7 (32bit) and linux/arm64 as docker images and the tarballs already exist for those.

Ah, I must have been thinking of Debian. We should drop the 32bit builds one way or the other, as they limit you to only a few GB of data.

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Dec 22, 2018

only a few GB of data

I would assume this isn't really a concern for most 32bit ARM use-cases as those are mainly Raspberry PIs and similar small compute platforms afaik, which have limited processing and storage capabilities anyway.

E.g. I'm currently running prom ARM32v7 on a Raspberry PI 3 with storage provided by a 16 GB sd card to collect some home automation stats and don't think I'll actually ever reach > 1 gb of collected data, but still find the prom platform with the variety of exporters providing a lot of value for my small use-case.

ARM setups with potentially more data might be in datacenter / cloud environments, but for example those new AWS ARM instances https://aws.amazon.com/blogs/aws/new-ec2-instances-a1-powered-by-arm-based-aws-graviton-processors/ are ARM64v8 afaik, similar to https://www.scaleway.com/virtual-cloud-servers/#anchor_arm and https://www.packet.com/cloud/servers/c1-large-arm/

@carlosedp

This comment has been minimized.

Copy link

commented Dec 22, 2018

I already build the Docker images for both ARM and ARM64 on Travis using some patches to the Dockerfiles for most Prometheus images. Maybe something might help:
https://github.com/carlosedp/prometheus-ARM

@discordianfish

This comment has been minimized.

Copy link
Member

commented Jan 2, 2019

To follow up on this:

1.) I think we should provide docker images for both platforms
2.) This is the right place to add the functionality
3.) ..but it needs to be added in a way that works for all repos

@johanneswuerbach johanneswuerbach force-pushed the johanneswuerbach:docker-arm branch to 8071a3f Jan 2, 2019

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Jan 2, 2019

@discordianfish I pushed 8071a3f, which changes this PR to also use an approach similar to prometheus/node_exporter#1207

As a test prometheus/node_exporter#1207 pushes docker images for 4 different architectures, while this PR only pushes three of them (no ppc64le) and the only difference is in DOCKER_ARCHS https://github.com/prometheus/prometheus/pull/5031/files#diff-b67911656ef5d18c4ae36cb6741b7965R15, while the circle.yml and Makefile.common are identical.

As DOCKER_ARCHS defaults to AMD64, other repositories should support the same Makefile.common without any changes, except calling make docker-publish explicitly for each to be released tag in CI. I can happily revert this, but I find the implicit pushing of multiple tags a bit dangerous and accompanying make docker-manifest only works with an explicit tag.

3.) ..but it needs to be added in a way that works for all repos

I can open PRs for more repos if you think the above looks good.

@discordianfish

This comment has been minimized.

Copy link
Member

commented Jan 3, 2019

Nice! In general this looks good to me, see inline comments though.
Also agree on calling make docker-publish explicitly with the ARCHS to build for.

The only thing I'm not sure about is the BIN_PATH stuff.. That seems a bit brittle. Maybe we should get rid of that, pass ARCH (and maybe OS) as build-arg and leave it to the Dockerfile to figure out where to get the binaries from, e.g:

ARG ARCH="amd64"
ARG OS="linux"
FROM quay.io/prometheus/busybox-${ARCH}-${OS}:latest
...
COPY .build/${OS}-${ARCH}/prometheus  /bin/prometheus
COPY .build/${OS}-${ARCH}/promtool       /bin/promtool

To make this nice we'd need to rename the default busybox image for this though.

Alternatively we could only use the build args for the base image and set docker context to .build/[os-arch], e.g: docker build -f ./Dockerfile .build/linux-amd64. But since we have to use the build args to figure out the base image somehow, it probably makes sense to use the former.

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Jan 3, 2019

When choosing those names, my main reasoning was staying consistent with the official docker naming https://github.com/docker-library/official-images#architectures-other-than-amd64, but I can happily adjust it to be inline with promu.

Is there an example repo, where you think the BIN_PATH approach would break?

@discordianfish

This comment has been minimized.

Copy link
Member

commented Jan 4, 2019

@johanneswuerbach I just find it nice if it's up to the Dockerfile to reference the things it copies in. I'm fine either way though but not maintainer of this repo. So up to @brian-brazil @juliusv and @fabxc

@brian-brazil

This comment has been minimized.

Copy link
Member

commented Jan 4, 2019

I defer to you on Docker-related stuff, @SuperQ and @simonpasquier may care on the build front.

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Jan 4, 2019

To recap: the remaining issue is, whether the released docker images should follow the promu style GOOS-GOARCH naming, which would allow to only use two build args OS and ARCH.

Example resulting docker file:

ARG ARCH="arm64"
ARG OS="linux"
FROM quay.io/prometheus/busybox-${ARCH}-${OS}:latest
...
COPY .build/${OS}-${ARCH}/prometheus  /bin/prometheus
COPY .build/${OS}-${ARCH}/promtool       /bin/promtool

or whether the docker images should follow the official docker image naming https://github.com/docker-library/official-images#architectures-other-than-amd64 (e.g. armv8 vs arm64), which would require to inject three build args: os, promu arch and docker arch.

Example resulting docker file:

ARG PROMU_ARCH="arm64"
ARG DOCKER_ARCH="armv8"
ARG OS="linux"
FROM quay.io/prometheus/busybox-${DOCKER_ARCH}-${OS}:latest
...
COPY .build/${OS}-${PROMU_ARCH}/prometheus  /bin/prometheus
COPY .build/${OS}-${PROMU_ARCH}/promtool       /bin/promtool

I don't really have a strong opinion (or any concerns) with any of the two and will happily change this PR to the preferred solution.

@discordianfish

This comment has been minimized.

Copy link
Member

commented Jan 5, 2019

I tend towards using only OS/ARCH. If I understand this correctly, with the docker manifest stuff this doesn't really matter much anyway.

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Jan 13, 2019

@discordianfish , last open question. What would you do for amd64?

Re-push them as prometheus-amd64-linux or leave them as they are today?

@jmreicha jmreicha referenced this pull request Feb 1, 2019
@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Feb 7, 2019

@discordianfish, any update on the above?

@simonpasquier simonpasquier self-requested a review Feb 8, 2019

@simonpasquier

This comment has been minimized.

Copy link
Member

commented Feb 8, 2019

LGTM for the Makefile/CI parts. AFAICT it will be safe to update Makefile.common in the other projects even without enabling the multi-arch builds. Thanks for all the efforts @johanneswuerbach!

@Adirio

This comment has been minimized.

Copy link

commented Mar 11, 2019

The main difference is from a management perspective. Organizations need to be created and a password for them stored/saved/memorized. Repositories need to be created but no password required. Labels get auto-created so no need to manage them.

From the user point of view the only difference is how he should refer to the images as I wrote before:

Option 1 would look like prom-ARCH/LIBRARY:VERSION (e.g. prom-arm32v7/prometheus:latest).
Option 2.1 would look like prom/LIBRARY-ARCH:VERSION (e.g. prom/prometheus-arm32v7:latest).
Option 2.2 would look like prom/LIBRARY:ARCH-VERSION (e.g. prom/prometheus:arm32v7-latest).
In any of the options there would be the multi-arch manifests under prom/LIBRARY:VERSION (e.g. prom/prometheus:latest) that will be used most of the times.
The most used option will be the common prop/prometheus:latest that is valid for the 3 options and when needing to specify the architecture or the version any of the 3 options works fine if it is documented.

Both are two trivial differences so I would decide based on the following criteria:

  • If following the official aproach is the most important criteria of these 3, go for option 1.
  • If avoiding to create new organizations/repositories is the most imporatant criteria of these 3, go for option 2.2.
  • If none of the above criterias are that important, stick with option 2.1 as this PR would not need to be changed.
@SuperQ

This comment has been minimized.

Copy link
Member

commented Mar 11, 2019

Thanks for the explanation.

  • Creating new orgs would be a huge pain I would like to avoid. No to Option 1
  • Creating libraries is only slightly annoying.

In either Option 2.1, the user has to specify non-amd64 architecture in the pull. This is annoying for the user, but I don't see there being much difference for the user between 2.1 and 2.2.

@Adirio

This comment has been minimized.

Copy link

commented Mar 11, 2019

Thanks for the explanation.

You are welcome

  • Creating new orgs would be a huge pain I would like to avoid. No to Option 1
  • Creating libraries is only slightly annoying.

Option 2.1 & 2.2 remaining

In either Option 2.1, the user has to specify non-amd64 architecture in the pull. This is annoying for the user, but I don't see there being much difference for the user between 2.1 and 2.2.

I guess you mean either Option 2. And thats not right, I guess I was not clear enough. In either Option 2 you would have the prom/prometheus:latest with the multi-arch manifest, so no need to specify the arch. But if for any reason someone wanted to specify the arch they would need to write it the way I presented.

P.S.: I can not think of a single use case of referencing the specific archs directly when you can specify the multi-arch manifest and forget about it. You still need to have them so that the multi-arch can refer to them, but the user will always use the multi-arch manifest. You can consider prom/prometheus:latest as the proxy that redirects to either prom/prometheus-arm32v7:latest or prom/prometheus:arm32v7-latest depending on which option you choose.

@johanneswuerbach

This comment has been minimized.

Copy link
Contributor Author

commented Mar 11, 2019

P.S.: I can not think of a single use case of referencing the specific archs directly when you can specify the multi-arch manifest and forget about it.

cross-building is a least one use-case, which doesn't work with only a multi-arch manifest. E.g. in this PR FROM quay.io/prometheus/busybox-${OS}-${ARCH}:latest needs to be done and using just FROM quay.io/prometheus/busybox:latest won't work as there is no way to force a different architecture then the local one afaik.

@kfox1111

This comment has been minimized.

Copy link

commented Mar 11, 2019

@Adirio

This comment has been minimized.

Copy link

commented Mar 11, 2019

I was thinking from the user perspective, the developer of other images would need to target the specific archs thats true.

Kevin, prometheus is actually mirrored in quay.io and docker hub, and obviously both the specific images for each architectures and the multi arch manifest are mirrored. But I don't see that as being a use case.

What I was talking is that no user should target the specific images over the multi arch manifest.

@simonpasquier

This comment has been minimized.

Copy link
Member

commented Mar 15, 2019

Pondering on the discussion and I think that @johanneswuerbach has a point with users leveraging the Prometheus image as a base image for building another image. As Ben noted, creating per-arch repositories is a bit painful but it shouldn't happen too frequently. Hence my inclination would be to go with option 2.1.

@Adirio

This comment has been minimized.

Copy link

commented Mar 15, 2019

Pondering on the discussion and I think that @johanneswuerbach has a point with users leveraging the Prometheus image as a base image for building another image.

I think my point was not clear. I was never suggesting not having arch specific images, you need them for the multi-arch manifest. I was just saying that the most common case will be linking to the multi-arch manifest and letting the node determine which image it wants. So the differences writting 1, 2.1 or 2.2 are minimal. As you all said when building a docker image you need to reference the specific architecture but image developers are expected to have a stronger background than the common user.

@discordianfish

This comment has been minimized.

Copy link
Member

commented Mar 25, 2019

If docker registries would be just http servers.. Remember these discussions vividly :).
Either way, looks like we need to decide on something so I say let's go with one org and one repo per app-arch as @johanneswuerbach initially suggested.

@discordianfish

This comment has been minimized.

Copy link
Member

commented Apr 10, 2019

Docker seems to have allowed 'lazy' repo creation by just pushing to some name, so I don't think we need to create anything anymore, correct? So I'd be all up for merging this now. We can still iterate on this later to come up with an even more convenient way.

@kfox1111

This comment has been minimized.

Copy link

commented Apr 10, 2019

yeah. pushing to a non existing repo creates it.

@Adirio

This comment has been minimized.

Copy link

commented Apr 11, 2019

Docker seems to have allowed 'lazy' repo creation by just pushing to some name, so I don't think we need to create anything anymore, correct? So I'd be all up for merging this now. We can still iterate on this later to come up with an even more convenient way.

I think that the issue with repo creation was on quay.io not on dockerhub

I would also merge though

@SuperQ

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

Ok, I created the required repos on quay.io. I think this is ready for merge.

@SuperQ SuperQ merged commit 8dfd829 into prometheus:master Apr 15, 2019

4 checks passed

DCO DCO
Details
ci/circleci: build Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

beorn7 added a commit that referenced this pull request Apr 15, 2019

Docker images for ARM32v7 and ARM64v8 (#5031)
Build and publish ARM32v7 and ARM64v8 docker images.

Signed-off-by: Johannes Würbach <johannes.wuerbach@googlemail.com>
@alexellis

This comment has been minimized.

Copy link

commented Apr 25, 2019

Hi all, I can't seem to find the arm/arm64 images.

Can you tell me where they are going to be published?

Thanks,

Alex

no-arm

@johanneswuerbach johanneswuerbach deleted the johanneswuerbach:docker-arm branch Apr 25, 2019

@simonpasquier

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

@alexellis you'd have to wait for v2.10.0 as this PR has been merged after v2.9.0 got released. Multi-arch images are already available for master though:

image

matthiasr added a commit to prometheus/statsd_exporter that referenced this pull request May 17, 2019

Update CircleCI config for prometheus/prometheus#5031
multi-arch \m/

Release 0.10.2 with this. Hopefully.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>

matthiasr added a commit to prometheus/graphite_exporter that referenced this pull request May 17, 2019

Update release build config for prometheus/prometheus#5031
to properly handle multi-arch.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>

matthiasr added a commit to prometheus/influxdb_exporter that referenced this pull request May 17, 2019

Update release build config for prometheus/prometheus#5031
to properly handle multi-arch.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
@tamsky tamsky referenced this pull request May 24, 2019
@devinrsmith devinrsmith referenced this pull request Jul 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.