Skip to content

docs(security): record Loki's unbounded volume, and test the control (#189) - #319

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/loki-unbounded-residual-189
Sep 5, 2026
Merged

docs(security): record Loki's unbounded volume, and test the control (#189)#319
Gerrrt merged 1 commit into
mainfrom
gerrrt/loki-unbounded-residual-189

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Closes #189 via its option 3, and adds the test that option 3 turns out to depend on.

Why this option

The issue's own conclusion, followed through: option 1 (authenticate the ingest ports) is the real answer, and it is #182's entire scope — both ports, every pushing Alloy agent, a credential to distribute and rotate. Doing it here would close #182 by accident with #189's narrower framing as the record of why. Option 2 bounds the blast radius without bounding the volume, and it is a host/LVM change that buys more of the thing there is most of: 53 GiB free against ~22 MB/day.

So: record it, and make #182's second justification explicit.

What SECURITY.md gains

A row in the exposure table beside the existing Prometheus/Loki residual, and a section carrying the part that was only ever in the issue:

Rate limits are the obvious response and they do not solve it. The lab produces roughly 0.012 MB/s of logs, so even a deliberately tight 0.5 MB/s ingestion_rate_mb is 40x headroom — and 0.5 MB/s still fills 59 GiB inside a day and a half. Any limit loose enough not to drop real logs is loose enough to fill the disk.

That is the durable half. Without it the next person notices the gap, reaches for ingestion_rate_mb, and calls it fixed.

The measured position is recorded too — 461 MB on 2026-08-31, 549 MB on 2026-09-04 — so drift is distinguishable from noise later without re-deriving the rate. Unbounded is a different property from close.

The control had no test

HostDiskWillFillIn24h is the whole control under this option. host.rules.yaml had no test file at all, so it was as unverified as ContainerCpuThrottled was in #185 — and for the same reason nobody noticed.

It is not in #185's position, though: node_filesystem_avail_bytes and node_filesystem_size_bytes report for prometheus, oracle and Saruman right now, so the input exists and the only question was whether the expression handles it.

host.test.yaml answers that with three cases and mutation testing:

Mutation Caught by
Drop the predictive half (becomes a static <30% threshold) the full-but-stable case
Drop the 30% guard (predictive half alone) the roomy-but-filling case
Invert the prediction's sign the firing case

Two corrections from doing it

  • My first mutation run reported two of the three "not caught", and it was wrong. The replacement text did not match the file verbatim, so promtool ran against an unmodified rule and passed. A mutation test that does not mutate looks exactly like a rule that cannot be broken — worth knowing, since that is the failure mode this whole class of test exists to prevent.
  • Once applied properly, the 30%-guard mutation genuinely was not caught. The roomy-but-filling case evaluated at 2h, which is precisely when the alert would first fire without the guard, so it reported silence either way. Moved to 4h, and the comment in the file says why so nobody "tidies" it back.

Checks

validate.sh — 48 PASS, 0 FAIL, 2 SKIP. promtool test rules now covers 10 files in observability, up from 9. yamllint caught a 162-char annotation line, and check_docs.py caught the coverage counts moving (28→29 tested, 28→27 not) — the third time that sentence has shifted this week, and the third time the guard from #212 caught it rather than a person.

🤖 Generated with Claude Code

…189)

#189 asks for one of three things chosen and written down. This is option 3,
and it is the one the issue itself leans toward once its own reasoning is
followed: the real answer is authentication in front of the ingest ports, which
is #182's whole scope, so #189 is a second justification for that work rather
than a separate piece of it.

The argument worth preserving is that rate limits look like a fix and are not.
The lab produces ~0.012 MB/s, so a deliberately tight 0.5 MB/s limit is 40x
headroom and still fills 59 GiB in a day and a half. Any limit loose enough not
to drop real logs is loose enough to fill the disk. That reasoning existed only
in a GitHub issue; in SECURITY.md it stops the next person reaching for
ingestion_rate_mb and calling it done.

Measured position recorded with it: loki-data was 461 MB on 2026-08-31 and
549 MB on 2026-09-04, ~22 MB/day against 53 GiB free. Unbounded is a different
property from close, and a later reader should be able to tell drift from noise
without re-deriving the rate.

HostDiskWillFillIn24h is the entire control under this option and had no test —
host.rules.yaml had no test file at all. Accepting a residual on the strength of
an unverified rule is #185's mistake with the stakes moved, so host.test.yaml is
added with it: fires on a disk that is filling, quiet on one that is full but
stable, quiet on one that is filling with room to spare.

The inputs are real, unlike #185's CFS counters — node_filesystem_avail_bytes
reports for prometheus, oracle and Saruman today — so the question was only
whether the expression handles them, and all three mutations now fail the suite:
dropping the predictive half, dropping the 30% guard, and inverting the
prediction's sign.

The roomy-but-filling case evaluates at 4h rather than 2h. Without the guard the
projection first goes negative at 1h and the alert fires at 2h, so an eval there
sits on the boundary and reports silence whether the guard exists or not — a
quiet case that cannot fail is what this file exists to avoid.

observability.md's coverage counts move again with the new tests, 28 to 29
tested and 28 to 27 not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 26147ff into main Sep 5, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/loki-unbounded-residual-189 branch September 5, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Loki is now the only unbounded store on the disk Authenticate the Prometheus and Loki ingest ports

1 participant