docs: monitoring.md Prometheus troubleshooting contradicts compose.yaml's user: "0" setting

[Sertug17]

Summary

The "Prometheus is restarting with a permissions error" section in docs/monitoring.md instructs operators to run:

sudo chown -R 65534:65534 "$ARC_MONITORING"/prometheus-data

However, the compose.yaml provided in the same guide sets user: "0" on the Prometheus service, which runs Prometheus as root. Root always has write access to the bind-mounted ./prometheus-data directory, so the described UID 65534 write-permission error cannot occur when following the guide as written.

The troubleshooting step applies only to operators who remove user: "0" (a reasonable security hardening step), but the guide does not explain this connection, making the section misleading.

Fix

Update the troubleshooting section to note that user: "0" in the provided compose config prevents this error, and that the chown step is relevant only when Prometheus runs as a non-root user.

[osr21]

Confirmed — the report is accurate, and I think the root cause is slightly sharper than described, which changes what the best fix looks like.

Verified

Both lines are present as quoted: user: "0" at docs/monitoring.md:158 and the chown -R 65534:65534 at docs/monitoring.md:283.

The UID is correct, which is worth stating because the fix depends on it. Pulling the prom/prometheus image config straight from the registry:

User      : 'nobody'
WorkingDir: '/prometheus'
Volumes   : {'/prometheus': {}}

and from /etc/passwd in the base layer:

nobody:x:65534:65534:nobody:/home:/bin/false

So 65534 is genuinely the image's default user, and the chown target is right — for the case where Prometheus is not forced to root.

This was contradictory from inception, not drift

Worth knowing before deciding how to fix it: both lines arrived in the same commit.

$ git log --oneline -S'user: "0"'  -- docs/monitoring.md   ->  a96d0ed docs: add single-host monitoring guide
$ git log --oneline -S'65534'      -- docs/monitoring.md   ->  a96d0ed docs: add single-host monitoring guide

monitoring.md has only three commits total, and the other two are markdown-fence fixes. So this isn't a stale section left behind when user: "0" was added later — the guide shipped self-inconsistent on day one. That suggests the troubleshooting text was carried over from a non-root context rather than written against this compose file.

That guess is supported by the repo's own templates disagreeing with each other:

• crates/quake/templates/local/compose-monitoring.yaml.hbs sets no user: on Prometheus, so it runs as nobody — the chown advice is correct and necessary there.
• crates/quake/terraform/templates/monitoring/compose.yaml.hbs uses user: "0:0", matching the docs.

The structural cause is line 69, and it strengthens the report

The guide creates the data directory itself, as the invoking user:

mkdir -p "$ARC_MONITORING"/prometheus-data      # docs/monitoring.md:69

So the directory is owned by the operator's UID — not root, and not 65534. This is what makes the contradiction structural rather than cosmetic:

• With user: "0" (as shipped) root ignores the ownership, so the error can't fire.
• Remove user: "0" and it fires immediately on first start, not as some rare edge case, because a nobody process cannot write to a directory owned by UID 1000.

Note this is not the usual "Docker auto-created the bind mount as root" situation — the guide pre-creates it, so removing user: "0" is the only way to reach that troubleshooting section. The report's reading is correct.

One caveat on "cannot occur"

For a stock rootful Docker daemon — the guide's implicit assumption — "cannot occur" holds. Two deployments break that, and both are worth a word because in neither does the documented chown help:

• --userns-remap: container root maps to a subordinate host UID (e.g. 231072), which has no write access to an operator-owned directory. The error still occurs with user: "0", and chown 65534 does not fix it.
• SELinux (RHEL/Fedora/CentOS): the bind mount ./prometheus-data:/prometheus carries no :z/:Z label, so access is denied regardless of UID. Again chown is not the remedy.

(Conversely, on an NFS-backed directory with root_squash, root maps to 65534 and the documented chown coincidentally is correct.)

So the section is not merely misleading about when it applies — it is also the wrong remedy for the cases where a root-mode permission error genuinely can happen.

On the proposed fix

Documenting the user: "0" connection resolves the contradiction, but it locks in running Prometheus as root, and I'd push back on that being the direction. This is a metrics scraper with network_mode: host and a host bind mount; root buys nothing here except avoiding a chown.

Inverting the fix resolves the contradiction and removes the privilege:

1. Drop user: "0" from the compose file.
2. Move the ownership fix into the setup step next to the mkdir, where it belongs:

mkdir -p "$ARC_MONITORING"/prometheus-data
sudo chown -R 65534:65534 "$ARC_MONITORING"/prometheus-data

The troubleshooting section then becomes a coherent recovery step for the same, single configuration.

A cleaner variant that avoids sudo altogether: use a named volume instead of a bind mount. Docker initialises a named volume's ownership from the image's user, so nobody can write with no manual chown and no root. The tradeoff is that the data is less convenient to inspect from the host, which may be why the bind mount was chosen — worth a deliberate call either way.

One adjacent thing

image: prom/prometheus is unpinned, so it tracks latest. Since the correctness of the 65534 advice is coupled to that image's default user, a future image change would silently invalidate whichever fix is chosen here. Pinning a tag in an operator-facing guide would be worth doing at the same time.