Skip to content

fix(observability): stop the UPS dashboard reporting a battery that isn't there - #30

Merged
Gerrrt merged 1 commit into
mainfrom
fix/ups-no-battery-honesty
Aug 19, 2026
Merged

fix(observability): stop the UPS dashboard reporting a battery that isn't there#30
Gerrrt merged 1 commit into
mainfrom
fix/ups-no-battery-honesty

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Makes the missing UPS battery visible in the monitoring instead of hidden by it.

What changed

  • A banner panel at the top of the UPS & Power dashboard.
  • The four fabricated panels retitled and given descriptions.
  • Output power marked as not reported by this NMC.
  • The ups.rules.yaml header rewritten to state that its battery rules cannot fire.

No rule is disabled or altered. Every one becomes correct the moment a pack is fitted.

Why

mjolnir has no battery pack fitted. The management card does not report that —
it fabricates a healthy one:

upsBatteryStatus              2     (batteryNormal)
upsEstimatedChargeRemaining   100
upsEstimatedMinutesRemaining  63
upsBatteryVoltage             480   (48.0 V)
upsAlarmsPresent              0

So the dashboard shows a green battery status and 63 minutes of runtime that do
not exist, and UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and
UpsAlarmPresent all sit comfortably inside their thresholds and will stay there.

The missing battery is known and tracked in docs/roadmap.md. The hazard this
addresses is a different one:
the monitoring currently offers positive
reassurance about it. A green UPS panel reads as "about an hour of runtime" at
the exact moment that is most wrong. A quiet ups group is not evidence of
runtime, and nothing in the stack said so.

UpsOnBattery, UpsOverloaded and UpsBatteryTemperatureHigh read mains-side
or load-side values and are unaffected — though UpsOnBattery would be
reporting an outage already underway rather than warning of one.

The rules header also records a trap worth not discovering the hard way: do not
run a UPS self-test to force an honest reading.
A self-test transfers the load
to battery, and with none fitted that is precisely the outage it exists to warn
about. upsTestResultsSummary is 6 (noTestsInitiated) and should stay there.

Blast radius

One dashboard and one rules file’s comments. No alert expression, threshold,
route or receiver changes. No config, credential or segmentation change.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

scripts/check_dashboards.py   5 dashboards OK (80 panels, 79 PromQL expressions)
promtool check rules          SUCCESS: 7 rules found
yamllint --strict             clean
./scripts/validate.sh         all checks passed

The new panel is type: text, which check_dashboards.py exempts from the
targets requirement; every existing panel was shifted down 4 rows and the
overlap check passes.

  • make validate passes
  • Deployed to the lab — Grafana provisions dashboards from files, so this
    lands on the next make up; no reload needed for rules comments
  • Docs updated (the rules file header is the documentation here)

…sn't there

mjolnir has no battery pack fitted. Its management card does not report that —
it fabricates a healthy one:

  upsBatteryStatus              2     (batteryNormal)
  upsEstimatedChargeRemaining   100
  upsEstimatedMinutesRemaining  63
  upsBatteryVoltage             480   (48.0 V)
  upsAlarmsPresent              0

So the dashboard shows a green battery status and 63 minutes of runtime that do
not exist, and UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and
UpsAlarmPresent all sit comfortably inside their thresholds and will stay there.

The missing battery is known and tracked. The hazard this fixes is different:
the monitoring currently gives positive reassurance about it. A green UPS panel
reads as "an hour of runtime", at the exact moment that is most wrong.

Adds a banner to the dashboard, marks the four fabricated panels in their titles
and descriptions, and rewrites the rules-file header to say plainly that those
rules cannot fire for this condition. Nothing is disabled: every rule becomes
correct the moment a pack is fitted.

Also marks Output power, which this NMC firmware reports as 0 regardless of
actual draw while Output load reads 15%.

Records the trap too: do not run a self-test to force an honest reading. A
self-test transfers the load to battery, and with none fitted that is the outage
it exists to warn about.

Refs docs/roadmap.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 5852358 into main Aug 19, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the fix/ups-no-battery-honesty branch August 19, 2026 04:00
Gerrrt added a commit that referenced this pull request Aug 19, 2026
fix(observability): stop the UPS dashboard reporting a battery that isn't there
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant