Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker: ARM images doesn't work since v6.4.x #19585

Closed
theWaldschrat opened this issue Oct 2, 2019 · 27 comments · Fixed by #19798

Comments

@theWaldschrat
Copy link

@theWaldschrat theWaldschrat commented Oct 2, 2019

What happened:
Grafana 6.4.X ARM in Docker does not work on Raspbian Buster.
Error message when trying to run the container:
/run.sh: line 80: /usr/share/grafana/bin/grafana-server: No such file or directory

What you expected to happen:
Obvious: Grafana runs without error.

How to reproduce it (as minimally and precisely as possible):
docker run grafana/grafana

Anything else we need to know?:
Error is caused by a lib-c mismatch: Grafana is build with ld-linux-armhf.so, but Alpine base image contains only ld-musl-armv7.so.

Environment:

  • Grafana version: 6.4.X
  • Data source type & version: InfluxDB
  • OS Grafana is installed on: Raspbian Buster with Docker
  • User OS & Browser: Windows / Firefox
  • Grafana plugins: None
  • Others: None
@hugohaggmark

This comment has been minimized.

Copy link
Member

@hugohaggmark hugohaggmark commented Oct 3, 2019

Thanks for reporting this @theWaldschrat we'll investigate it further

@marefr

This comment has been minimized.

Copy link
Member

@marefr marefr commented Oct 3, 2019

@theWaldschrat what device are you using? Is that 32- or 64-bit architecture (armv6m armv7, armv8 etc)?

@marefr

This comment has been minimized.

Copy link
Member

@marefr marefr commented Oct 3, 2019

Maybe we need to include https://pkgs.alpinelinux.org/package/edge/main/armhf/libc6-compat in Grafana docker image?

apk add --no-cache --repository=http://dl-cdn.alpinelinux.org/alpine/edge/main libc6-compat

@theWaldschrat can you confirm that above solves the prioblem? Harder for us to verify without access to an actual ARM device.Docker based arm image maybe a possibility, but appreciate if you can help us out here. Thanks

@theWaldschrat

This comment has been minimized.

Copy link
Author

@theWaldschrat theWaldschrat commented Oct 3, 2019

The device is a Raspberry Pi 4B. Technically it's an ARM64v8, but Raspbian by default runs a 32 bit kernel and userland, so ARM32v7.

uname -a
Host OS:
Linux raspberrypi 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l GNU/Linux
Grafana 6.3.6 image:
Linux 97f0bb9a456d 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l armv7l armv7l GNU/Linux
Grafana 6.4.X (latest) image:
Linux 84a01cb75816 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l Linux

I didn't build a lot of Docker images yet, so I'm not able to try above command in a new image, at least not quickly. But what I did do: Run a grafana:latest container with entry point bash and user root:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana
Here's the result:

  • Manually running /run.sh gives the same error message, so that's confirmed.
  • Run above command: No error.
  • Again run /run.sh: New error messages:
Error relocating /usr/share/grafana/bin/grafana-server: __memset_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __memcpy_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __vfprintf_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __fprintf_chk: symbol not found
  • Running ldd /usr/share/grafana/bin/grafana-server doesn't complain about missing libraries anymore, but gives the same results as above.

I'm not an expert, but I guess the lib-c are still not compatible.

@marefr

This comment has been minimized.

Copy link
Member

@marefr marefr commented Oct 3, 2019

@theWaldschrat thank you very helpful.

Just to verify some additional things can you specifically try these to verify that you get the same problem:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm32v7-linux:6.4.1

docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm32v7-linux:6.4.0-beta1

Just to be sure, can you also try to run and start grafana-server:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm64v8-linux:6.4.1

@theWaldschrat

This comment has been minimized.

Copy link
Author

@theWaldschrat theWaldschrat commented Oct 3, 2019

First two all do the same as previously described.
Running /run.sh or directly /usr/share/grafana/bin/grafana-server doesn't make a difference.

The third one doesn't even start with an arch mismatch:
standard_init_linux.go:211: exec user process caused "exec format error"

@ata4

This comment has been minimized.

Copy link

@ata4 ata4 commented Oct 3, 2019

I have the same issue and had to downgrade to version 6.3.6, so it seems that all Alpine-based 6.4.x images are broken for ARMv7.

@marefr

This comment has been minimized.

Copy link
Member

@marefr marefr commented Oct 3, 2019

Thanks. After you entered bash can you try and install the musl-dev packade using apk add?

@ata4

This comment has been minimized.

Copy link

@ata4 ata4 commented Oct 3, 2019

musl-dev installs fine for me, but doesn't have an impact on the problem, with or without libc6-compat.

@ata4

This comment has been minimized.

Copy link

@ata4 ata4 commented Oct 3, 2019

Installing the glibc apks from https://github.com/armhf-docker-library/alpine-pkg-glibc/releases allows grafana-server to start. If I understand the issue correctly, it's better to statically link the binaries with musl, though.

@theWaldschrat

This comment has been minimized.

Copy link
Author

@theWaldschrat theWaldschrat commented Oct 3, 2019

It's the idea of Alpine to statically link to musl instead of dynamic glibc linking. It's faster, smaller, more stable and potentially more secure. At least that's what they say.
But as far as I can see, Grafana is build outside the target image linking against glibc, so it's probably the best idea to either install glibc as above or use a different base image that already includes glibc.

@protik77

This comment has been minimized.

Copy link

@protik77 protik77 commented Oct 5, 2019

Considering this change effectively broke the docker image for ARM devices, I was expecting something better than a tag of "needs investigation".

Sigh! The curse of "agile development", I guess.

@aknuds1

This comment has been minimized.

Copy link
Contributor

@aknuds1 aknuds1 commented Oct 9, 2019

I can reproduce the error in OS X, even though it looks a bit different from yours:

$ docker run --platform arm grafana/grafana
/lib/ld-linux-armhf.so.3: No such file or directory

I will see if I can fix it.

@aknuds1

This comment has been minimized.

Copy link
Contributor

@aknuds1 aknuds1 commented Oct 9, 2019

I might have a lead on the root cause for this issue, hoping I can fix it by tomorrow.

@aknuds1

This comment has been minimized.

Copy link
Contributor

@aknuds1 aknuds1 commented Oct 10, 2019

Working to solve this by building musl binaries in addition to glibc ones.

@JasonSwindle

This comment has been minimized.

Copy link

@JasonSwindle JasonSwindle commented Oct 22, 2019

Just ran into this as well. My system is aarch64 (RockPro64) and I'm getting the same error:

/run.sh: line 80: /usr/share/grafana/bin/grafana-server: No such file or directory
@marefr marefr added this to the 6.4.4 milestone Oct 22, 2019
@marefr marefr changed the title 6.4.X Docker c-library mismatch on ARM (Raspi) Docker: ARM images doesn't work since v6.4.x Oct 22, 2019
@marefr

This comment has been minimized.

Copy link
Member

@marefr marefr commented Oct 22, 2019

Based on work in #19798 we've pushed a tag named dev-musl to grafana docker hub repositories. We've only been able to test run the arm and arm64 docker images using emulation so we ask kindly anyone to help out testing the arm and arm64 docker images to verify they work as expected. No manifest have been pushed to grafana/grafana so if you want to try arm or arm64 you have to manually specify correct repository, see below.

linux/amd64:
docker run <args> grafana/grafana:dev-musl

linux/arm64:
docker run <args> grafana/grafana-arm64v8-linux:dev-musl

linux/arm:
docker run <args> grafana/grafana-arm32v7-linux:dev-musl

Please note that these images are based on current developer branch (master/Grafana v6.5.0-pre) of Grafana so if you want to test with an existing Grafana installation please remember to backup your existing data.

Test scope:

  • Verify that container can successfully run and using docker logs <image name> shouldn't output anything unexpected like errors.
  • If possible, try to add/connect to a datasource and render a dashboard/panel
  • Optional: Verify that you can download/install plugins
  • Provide feedback in this issue or PR #19798 by making a comment telling which device/architecture you've used and the outcome.

Thanks in advance

@iwittkau

This comment has been minimized.

Copy link

@iwittkau iwittkau commented Oct 22, 2019

$ uname -a
Linux black-pearl 4.14.70-hypriotos-v7+ #1 SMP Sat Sep 22 05:54:18 UTC 2018 armv7l GNU/Linux

LGTM running on a Raspberry 3B

  • Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors.
  • If possible, try to add/connect to a datasource and render a dashboard/panel
  • Optional: Verify that you can download/install plugins
@JochenLutz

This comment has been minimized.

Copy link

@JochenLutz JochenLutz commented Oct 22, 2019

SBC: Cubietruck (aka CubieBoard 3)

$ uname -a
Linux fernia 4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l armv7l armv7l GNU/Linux

LGTM

  • Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors.
  • If possible, try to add/connect to a datasource and render a dashboard/panel
  • Optional: Verify that you can download/install plugins
@marefr

This comment has been minimized.

Copy link
Member

@marefr marefr commented Oct 22, 2019

Thanks a lot for the quick responses and help. Very appreciated.

@aknuds1

This comment has been minimized.

Copy link
Contributor

@aknuds1 aknuds1 commented Oct 22, 2019

@gcgarner

This comment has been minimized.

Copy link

@gcgarner gcgarner commented Oct 23, 2019

uname -a
Linux raspberrypi4 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

raspberry pi 4b

docker -v
Docker version 19.03.4, build 9013bf5
  • Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
  • if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb with graph panel
  • Optional: Verify that you can download/install plugins
@pgolm

This comment has been minimized.

Copy link

@pgolm pgolm commented Oct 23, 2019

LGTM: raspberry pi 4

$ uname -a
Linux worker-3 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux
  • Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
  • if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb, prometheus, loki)
  • Verify that you can download/install plugins (raintank-worldping-app, grafana-kubernetes-app, devopsprodigy-kubegraf-app, grafana-piechart-panel)
Backend Platform Squad automation moved this from Under review to Done Oct 24, 2019
jmb12686 added a commit to jmb12686/raspi-docker-stacks that referenced this issue Oct 25, 2019
@krystian-wojtas

This comment has been minimized.

Copy link

@krystian-wojtas krystian-wojtas commented Oct 26, 2019

LGTM

rockchip rock64

$ uname -a
Linux rock64 4.4.132-1072-rockchip-ayufan-ga1d27dba5a2e #1 SMP Sat Jul 21 20:18:03 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
  • Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
  • if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb, prometheus, loki)
  • Verify that you can download/install plugins (raintank-worldping-app, grafana-kubernetes-app, devopsprodigy-kubegraf-app, grafana-piechart-panel)
@SySfRaMe

This comment has been minimized.

Copy link

@SySfRaMe SySfRaMe commented Oct 27, 2019

Docker skill level
Noob
Grafana skill level
Noob
-uname -a
Linux SwingerPictureServer 4.19.75-v7+ #1270 SMP ........... armv71 GNU/Linux
HW
Raspberry pi 3B
compose.sh file:
docker run
--name Grafana_test
-p 3001:3001
-e "GF_SERVER_ROOT_URL=http://:3001"
-e "GF_SECURITY_ADMIN_PASSOWRD="
--mount type=bind,source="/home/pi/DockerConf/Grafana/test/config",target="/etc/grafana":ro
grafana/grafana-arm32v7-linux:dev-musl

LOG file:
warn: msg="phantomJS is deprecated and will be removed in the future release...

@marefr marefr modified the milestones: 6.4.4, 6.5 Oct 28, 2019
@marefr

This comment has been minimized.

Copy link
Member

@marefr marefr commented Oct 28, 2019

Thanks all. We've merged this to master, but decided to include this fix in Grafana v6.5.0, to be released in a couple of weeks. Until then you may use nightly builds if you want to run Grafana v6.5-pre ARM-compatible Grafana docker images by using the grafana/grafana:master tag.

@herm

This comment has been minimized.

Copy link

@herm herm commented Nov 10, 2019

Please add a note to docker hub so it is easier to find this issue. If you are currently pulling grafana/grafana you still get a non-functional image on armhf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.