prometheus: 2.22.0 failing on RPI4

[sashasimkin]

Maintainer: @aparcar
Environment: linux/arm64, RaspberryPI4 model B, snapshot

Description:

Prometheus version 2.22.0 is failing on RPI4 with a segfault. The same version built by go1.15.3 for arm64 arch has worked on the same RPI.

See more details in prometheus/prometheus#8589

I see that golang version is set to 1.16 in here: https://github.com/openwrt/packages/blob/master/lang/golang/golang/Makefile, but I suppose that many other packages depend on that version, and setting it to 1.15.3 most likely not something that will be accepted :(

Does it make sense trying to specify golang version to 1.16.2 for the whole build as suggested in prometheus/prometheus#8589 (comment)?

[aparcar]

I'll just update to the newest release which seem to fix the issue with later Golang versions:
#15237

[aparcar]

I updated Prometheus, could you please test 2.25.2?

[sashasimkin]

The binary is definitely working, thank you! But I'm getting some weird behavior :(

When I run it myself, using the command line and all parameters supplied by hand - it works well and I can see metrics in grafana, yet if I do start it as /etc/init.d/prometheus start - I see only flatline and no data after a while, like this:

while in logs it says:

Fri Mar 26 01:21:33 2021 daemon.info procd: Instance prometheus::instance1 s in a crash loop 6 crashes, 0 seconds since last crash

Later I figured out that it was most likely due to the permissions and that prometheus was not able to access /data directory under user prometheus because I've created it with prometheus running as root.

This is how I figured out it's starting with user prometheus:

root@home /53# PROCD_DEBUG=1 /etc/init.d/prometheus start
{ "name": "prometheus", "script": "\/etc\/init.d\/prometheus", "instances": { "instance1": { "command": [ "\/usr\/bin\/prometheus", "--config.file=\/etc\/prometheus.yml", "--storage.tsdb.path=\/data", "--web.listen-address=127.0.0.1:9090" ], "user": "prometheus", "file": [ "\/etc\/prometheus.yml" ], "respawn": [ "3600", "5", "5" ] } }, "triggers": [ ], "data": { } }

So I did the following steps to confirm:

1. Uninstalled prometheus
2. Backed up config
3. rm -rf /data
4. installed prometheus

yet, in logs I see this after the last installation:

Fri Mar 26 01:47:06 2021 daemon.info procd: Instance prometheus::instance1 s in a crash loop 6 crashes, 0 seconds since last crash
Fri Mar 26 01:48:04 2021 daemon.info procd: Instance prometheus::instance1 s in a crash loop 6 crashes, 0 seconds since last crash

and no metrics visible in grafana.
During those actions I noticed that data directory wasn't created:

root@home /53# stat /data
stat: can't stat '/data': No such file or directory

So I did create it with needed perms:

root@home /53# /etc/init.d/prometheus stop
root@home /54# mkdir /data && chown prometheus:prometheus /data
root@home /53# stat /data
  File: /data
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: b302h/45826d    Inode: 46081       Links: 2
Access: (0755/drwxr-xr-x)  Uid: (  112/prometheus)   Gid: (  112/prometheus)
Access: 2021-03-26 01:55:09.000000000
Modify: 2021-03-26 01:55:09.000000000
Change: 2021-03-26 01:55:09.000000000

root@home /53# /etc/init.d/prometheus start

After starting it like this - I can see the data in grafana. And the permission issue confirmed:

root@home /55# sudo -u prometheus mkdir /test
mkdir: can't create directory '/test': Permission denied

Yet this succeeds:

root@home /54# sudo -u prometheus mkdir /var/test
root@home /53#

So here's a small PR to make the experience better.

#15265

Lmk if I got something wrong there or maybe it's related to the image I'm using (it's wulfy's RPI4 image).

Btw, thanks for the amazing work you did to bring prometheus to openwrt world!