prometheus: Update default tsdb path

[sashasimkin]

Use /var/prometheus-data instead of /data, because user prometheus doesn't have permissions to create directory in root filesystem, while it's totally fine to be in /var

Maintainer: @aparcar
Compile tested: -
Run tested: linux/arm64, RaspberryPI4 model B, snapshot - by editing /etc/config/prometheus manually

Description: Use /var/prometheus-data instead of /data, because user prometheus doesn't have permissions to create directory in root filesystem, while it's totally fine to be in /var

related: #15236

[aparcar]

So /var is mounted in RAM, after every reboot your data would be lost. Having it in /data forces the user to think at least for a second on where to store things. Ideally on an external drive or USB stick, not so ideally in memory or some RPi flash chip.

What do you think?

[sashasimkin]

Oh, I didn't know that /var is mounted in RAM!

The problem with /data, is that it doesn't give any feedback on why prometheus is crushing :(

Is it maybe possible to add a remark about that on the OpenWRT packages page, or name the dir something like /pick-proper-location-for-data :D

Unless, /data is available on other devices, or some common mount-point for more durable than sd storage?

[sashasimkin]

Is it possible to document this quirk somehow, or am I missing some fundamental piece about OpenWRT?

[erdoukki]

Is it not a better place like /srv/data ?

[aparcar]

So /var is mounted in /tmp and wiped every time you reboot. /srv sounds better to me, however I don't know how it's mounted at this point.

[erdoukki]

/srv is default in rootfs, and may be switched by user in extroot or in any mountpoint...

[sashasimkin]

thanks @erdoukki , that makes sense, and I think I've done /srv/data(or other, more specific mountpoint) for myself in the end.

I've updated the path in PR to be in /srv/data/prometheus, will test on my installation soon.

[aparcar]

Please update me and I'll merge this.

[sashasimkin]

Just had time to boot up my installation and check:

OpenWRT(+wulfy's) image: OpenWrt SNAPSHOT r16357-f8669c174e

/srv/data doesn't exist, and when pointing prometheus tsdb data to /srv/data/prometheus - the service will appear running, but no data can be seen in grafana and not even /srv/data is created.

So I attempted to configure it with /srv/prometheus-tsdb path, and no luck either :(

---

I just realized that first message doesn't mention why it's fine for data to be in /var.
It boils down to sticky bit set on /var/ directory, so when prometheus creates it - it has his user's ownership.

There's no such thing is /srv.

---

So I'm thinking that instead of trying to figure out best place for user's data - maybe it's better to just document that by default installation is broken and users need to think and configure the package as they please?

UPD: So, as a final touch - this is what I've done and it works as expected and survives reboots:

1. Point tsdb path to /srv/prometheus
2. mkdir /srv/prometheus && chown prometheus:prometheus /srv/prometheus

[erdoukki]

Why not doing these stuff in the install script ?

[sashasimkin]

The only con I see is that it'll be burning PRI's flash chip by default while being in /srv, unless users have exroot or external USB mounted over that path.

Sounds good to me though, let me update the PR.

UPD: Though, I'm not sure where's the install script lives 😅

[sashasimkin]

I've pushed latest changes as per our discussion, but decided to create the directory if it doesn't exist on each prometheus service start.

This way it'll be also properly handled for cases when you move data to a USB partition.

(In fact, this is what prometheus tries to do currently, but fails because not having enough permissions. Here we create the dir as root and grant permissions to prometheus user.)

[sashasimkin]

I've fixed the requested changes - can you merge it 🙏 ?