feat(server): warn in the UI when the device is under-voltage - #3285
Conversation
Surfaces power-supply problems that previously only showed up as unexplained display glitches and stray text on screen. - Read the kernel's rpi_volt hwmon sensor instead of vcgencmd: the raspberrypi-hwmon driver clears the firmware's sticky bits every 2s, so get_throttled's since-boot flags are not a durable record - Works unchanged on Balena, which ships no host agent: sysfs reads the same from inside an unprivileged container - Latch the since-boot history in Redis, keyed on boot_id so it resets on reboot rather than persisting via the Redis volume - Watch the sensor with poll()/POLLPRI in the celery worker, since a sampler would miss the short dips a marginal supply produces - Show a persistent banner on every page plus a System Info card, in plain language, leading on the official Raspberry Pi supply - Expose under_voltage on /api/v2/info - collect_debug.sh now captures /api/v2/info rather than deriving device state in shell, keeping raw sysfs and the kernel ring buffer as fallbacks for when the stack cannot answer Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3285 +/- ##
=========================================
Coverage ? 90.39%
=========================================
Files ? 81
Lines ? 9228
Branches ? 989
=========================================
Hits ? 8342
Misses ? 656
Partials ? 230 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds kernel-hwmon based under-voltage detection (via rpi_volt sysfs), latches “seen since boot” in Redis, and surfaces the state consistently across the web UI, /api/v2/info, and debug bundles—so operators get an actionable warning instead of unexplained display glitches.
Changes:
- Introduces
anthias_common.undervoltage(sysfs detection + Redis latch) and a Celery-side background watcher to keep the latch current. - Adds an always-on-page under-voltage banner plus a System Info “Power supply” card with plain-language guidance and technical detail behind a disclosure.
- Exposes
under_voltageon/api/v2/infoand updatescollect_debug.shto snapshot the info endpoint (with sysfs/dmesg fallbacks).
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_undervoltage.py | Unit tests for hwmon discovery, alarm reads, and Redis latch behavior |
| tests/test_template_views.py | Template-level tests for the global banner and System Info power card rendering |
| src/anthias_server/lib/undervoltage_watcher.py | New poll()-based watcher thread for under-voltage transitions and latch refresh |
| src/anthias_server/celery_tasks.py | Starts the watcher on Celery worker_ready |
| src/anthias_server/app/templates/system_info.html | Adds “Power supply” card + technical details disclosure |
| src/anthias_server/app/templates/_power_warning.html | New global under-voltage banner partial |
| src/anthias_server/app/templates/_layout.html | Includes the power warning partial above <main> |
| src/anthias_server/app/static/sass/_styles.scss | New design tokens + styles for the banner and power card |
| src/anthias_server/app/page_context.py | Adds navbar power_warning context and System Info power context |
| src/anthias_server/api/views/v2.py | Adds under_voltage to /api/v2/info (+ schema) |
| src/anthias_server/api/tests/test_info_endpoints.py | Tests for the new under_voltage info field |
| src/anthias_common/undervoltage.py | New sysfs-based under-voltage reader + Redis latch implementation |
| bin/collect_debug.sh | Captures /api/v2/info snapshot and adds under-voltage sysfs/dmesg fallbacks |
Suppressed comments (1)
src/anthias_common/undervoltage.py:225
record_observation()incrementscounton every call whereactive=True, even if the latch was already active. Because the watcher also resamples on a timeout, a sustained under-voltage condition will inflatecountover time (e.g., +1 everyRESAMPLE_INTERVAL_S) even though no new “dip” occurred. Ifcountis meant to represent the number of under-voltage events/dips (as the UI copy implies), it should only increment on a rising edge (previously inactive → active), whilelast_seencan still be refreshed wheneveractiveis true.
if not state['seen_since_boot']:
state['seen_since_boot'] = True
state['first_seen'] = now
state['last_seen'] = now
state['count'] = state['count'] + 1
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Follow-up from local review of the under-voltage warning. - count was incremented on every truthy reading, so the watcher's 30s resample rendered "power dropped too low 21 times" for one brown-out that never went away; it is now edge-triggered off the latch, which also stops a worker restart recounting - treat POLLERR/POLLHUP and an empty sysfs read as errors: a removed hwmon device made poll() return instantly forever, and an empty read was mapped to "healthy", clearing a live warning - persist a live alarm seen only by a page render, so the banner no longer disappears on recovery when the watcher is down - guard the stored count coercion; a non-numeric value raised before the write that would overwrite it, killing the feature for good - cache the hwmon path scan, which ran on every page render - add tests for the watcher, which had none Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/anthias_server/lib/undervoltage_watcher.py:121
- The watcher logs "Under-voltage alarm cleared" on the first read (and after any exception) because
previousstarts asNoneandactive != previousis true even when the device is simply healthy. This creates misleading/noisy logs; only log transitions after an initial baseline has been established.
if active != previous:
_log_transition(active, state)
previous = active
src/anthias_common/undervoltage.py:276
- If the kernel boot id can't be read (
get_boot_id()returns None), the latch is persisted withboot_id=Noneand will then survive reboots (becausestored.get('boot_id') == boot_idstays true). That contradicts the intended "reset on reboot" behavior; consider degrading to a non-persisted, live-only state when boot id is unavailable.
if boot_id is None:
boot_id = get_boot_id()
state = _load_latch(redis_client, boot_id)
Found on a real Pi 4: the first reading of each watch pass differs from the initial None, so a device that never browned out logged "Under-voltage alarm cleared." about 30s after every boot. It reads as though an alarm had happened, in the one log a support engineer greps. A device that starts up already in alarm is still logged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/anthias_common/undervoltage.py:276
record_observation()currently persists a latch even whenboot_idisNone(by writing'boot_id': None). With the_load_latch()fix above this stored value becomes dead data; it’s also safer to avoid writing a latch at all when the boot id is unknown, since we can’t guarantee it will reset on reboot. Return a best-effort in-memory state instead.
if boot_id is None:
boot_id = get_boot_id()
state = _load_latch(redis_client, boot_id)
src/anthias_common/undervoltage.py:218
- If
boot_idcannot be read (e.g./procunavailable),_load_latch()will accept any previously-stored payload whoseboot_idis alsoNone, which makes the latch persist across reboots (contrary to the “reset on reboot” behavior described in the module docstring). Treat an unknown boot id as “cannot trust persisted latch” and always return an empty state.
This issue also appears on line 273 of the same file.
def _load_latch(redis_client: Any, boot_id: str | None) -> dict[str, Any]:
"""Read the stored latch, discarding it if it predates this boot.
Redis persists to a volume, so without the boot-id check a
device that browned out last week would still be showing the
From Copilot review. A stored `boot_id: null` compares equal to a current unknown `None`, so a device that could not read its boot id would treat an old latch as current and the warning would never reset again. - discard a stored latch outright when the boot id is unknown - skip persisting in that case, and log why - correct the watcher comment: the fd is held for the life of the inner loop, and only reopened on the recovery path Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Third point from the review (the suppressed one on The trap is sharper than just dead data: a stored Fixed on both sides so they cannot disagree:
Degrading to "live readings only" is the safe direction here: it can under-report history, never invent it. Two regression tests cover the discard and the skipped write. All three points now addressed. Full gate green: 1812 tests, |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (1)
tests/test_undervoltage_watcher.py:176
- Avoid using
# type: ignore[method-assign]here by using a smallFakePollsubclass that overridespoll()for this test. This keeps mypy clean without suppressions and avoids runtime monkeypatching of a method.
poller = FakePoll([])
poller.poll = flip # type: ignore[method-assign]
|
Re-validated on the Pi 4 testbed against the current head
Testbed restored to |
|
Now validated on all seven testbeds at
Banner correctly absent while healthy on all seven, and no board logged a spurious Two things this fleet run adds over the single-board run:
The 512 MB Pi 3 A+ and the 1 GB Rock Pi 4 are known-fragile. The Rock Pi 4 sat at 0 MB swap free throughout, so it got a server-only restart to avoid adding pressure and its watcher path was exercised with a direct All seven restored to their pinned |
Second review pass. The POLLERR guard added last round broke the event-driven path outright: kernfs_generic_poll() returns DEFAULT_POLLMASK|EPOLLERR|EPOLLPRI on every genuine change notification, so a real brown-out arrives as POLLPRI|POLLERR and was raising, logging a stack trace, sleeping 60s and dropping the transition. Measured as mask 0xa on a Pi 5 (kernel 6.18) and confirmed with a negative control: the old guard drops a real kernel event, the new one processes it. - believe POLLERR only when POLLPRI is absent, the standard sysfs idiom - tests now script the mask the kernel really returns; the previous bare-POLLPRI mask is what let this through - skip the Redis write when nothing changed; the 30s re-sample was ~2,880 appendonly-fsynced writes a day on a healthy device - treat an unreadable latch as unknown, not empty, so a transient Redis error can no longer erase a real brown-out record - log the missing-boot-id warning once per process, not every 30s - raise collect_debug's API timeout above get_node_ip()'s ~80s wait, so the snapshot is captured on hosts with no running host agent Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Second review pass found a serious one that the hardware run had missed, because that run never induced a real sensor event. The Measured on a Pi 5 (kernel 6.18) with a
The tests missed it because Four smaller fixes in the same commit:
1818 tests, ruff, mypy all clean. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/anthias_server/app/page_context.py:50
- _parse_iso() claims to return an aware datetime, but datetime.fromisoformat() will successfully parse a naive ISO string (no offset) and return a naive datetime. Passing a naive datetime into Django’s humanize/naturaltime (with USE_TZ enabled) can raise offset-naive/offset-aware comparison errors during template rendering. Consider rejecting naive timestamps (or making them explicitly UTC) so template rendering can’t break on a malformed/stale latch value.
if not isinstance(value, str):
return None
try:
return datetime.fromisoformat(value)
except ValueError:
return None
src/anthias_server/app/page_context.py:72
- _power_warning() fully suppresses exceptions from undervoltage.get_state() with no logging. If this path starts failing in production, the banner will silently disappear (potentially hiding real under-voltage) and operators/support will have no signal that the diagnostic itself is broken. Logging once per process keeps the “never break rendering” property without losing debuggability.
try:
state = undervoltage.get_state(_redis)
except Exception:
# Never let a diagnostic break page rendering.
return None
Every state change arrives as a kernel POLLPRI notification on both edges, so the poll timeout is only a backstop for a missed event. A 30s timeout woke the worker and re-read sysfs to learn nothing ~2,880 times a day on a device that was behaving. Nothing user-facing depends on the cadence: get_state() reads the live attribute on every page render and writes back on disagreement, so the UI is current to the request, not to the last poll. - RESAMPLE_INTERVAL_S 30 -> 3600 - route the unreadable-latch warning through the same once-per-boot throttle as the missing-boot-id one, since page renders hit it too Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/anthias_common/undervoltage.py:339
- record_observation() updates the Redis latch via a non-atomic read/modify/write (GET in _load_latch → local mutation → SET). Because both the watcher thread and page renders can call this concurrently (and the server may run multiple workers), two callers can observe the same inactive→active edge and each increment
count, inflating the occurrence counter and potentially clobberingfirst_seen/last_seen.
To keep count reliably edge-triggered under concurrency, consider making the latch update atomic (e.g., WATCH/MULTI with retry, or a Lua script that reads the JSON, applies the transition rule, and writes back in one round trip; alternatively store active/count in separate Redis keys and use INCR on a successful inactive→active transition).
state, readable = _load_latch(redis_client, boot_id)
prior = dict(state)
was_active = state['active']
state['active'] = active
if active:
now = _now_iso()
if not state['seen_since_boot']:
state['seen_since_boot'] = True
state['first_seen'] = now
state['last_seen'] = now
if not was_active:
state['count'] = state['count'] + 1
tests/test_undervoltage_watcher.py:188
- This test assigns to
poller.polland suppresses the type error with# type: ignore[method-assign]. The repo guidelines discouragetype: ignorewhen a small refactor can avoid it; here the override can be expressed via a tiny FakePoll subclass, keeping mypy clean without suppressions.
poller = FakePoll([])
poller.poll = flip # type: ignore[method-assign]
with (
From the low-confidence review comments. - _parse_iso() stamped naive values as UTC. We only ever write aware strings, but fromisoformat accepts a naive one and naturaltime then compares it to a naive LOCAL now: a dip one minute ago rendered as "3 hours from now" on a US-Eastern player. Not the TypeError the review predicted, Django's humanize handles naive values, just a nonsense relative time in the banner. - _power_warning() swallowed every exception silently, so a broken probe was indistinguishable from a healthy supply and would have hidden a real under-voltage with no signal. Logged once per process. - replace a `type: ignore[method-assign]` in the tests with a FakePoll subclass, per the repo's no-suppressions rule. - tests pinning the concurrency behaviour: two callers observing the same edge converge on one increment rather than stacking, and seen_since_boot is never downgraded by a stale writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Went through the four suppressed (low-confidence) review comments. Three were real and are fixed in 812a7e1; one I believe is incorrect, with a test now pinning the behaviour. Fixed — naive latch timestamp ( So a stale or hand-edited latch would put "most recently 3 hours from now" in the banner on any non-UTC device. Naive values are now stamped UTC (which is what the writer emits), explicit offsets are preserved, and there's a test that fails without the fix. Fixed — silent exception swallow ( Fixed — Not fixed — non-atomic latch update ( 1823 tests, ruff, mypy all clean. |
|



Surfaces under-voltage in the web UI instead of leaving it to show up as
unexplained display glitches.
Prompted by this forum thread,
where stray characters along the edge of the screen turned out to be
kernel console messages triggered by a marginal power supply. The
accepted answer there suggests redirecting stderr to
/dev/null, whichcannot work: those messages are kernel
printkoutput, not something auserspace process emits. Nothing here suppresses anything.
Why not
vcgencmd get_throttledIt is no longer a reliable source, and not only because it is missing
from the server image.
The
raspberrypi-hwmondriver polls the same firmware property every2 seconds and clears the sticky bits as it goes (it sends
value = 0xffff). On every current Raspberry Pi OS and balenaOS Pikernel, the "has occurred since boot" bits (16-19) are wiped moments
after they are set, so anything reading them sees a field the kernel
just zeroed. The kernel owns this state now.
So detection reads the kernel's own view instead:
/sys/class/hwmon/hwmonN/in0_lcrit_alarmon the hwmon device namedrpi_volt.Balena
The sysfs read is what makes one code path work everywhere. It needs no
host agent, no
/dev/vciomapping, nolibraspberrypi-binand nocompose changes, so Balena fleets (which ship no
anthias_host_agent)are covered by exactly the same code as docker-compose installs.
Since-boot history
Because the driver keeps clearing the firmware bits, "has this happened
since boot" is not available from the hardware at all. Anthias latches
it in Redis, keyed on
/proc/sys/kernel/random/boot_idso it resets onreboot rather than persisting through the Redis volume and warning
about a brown-out that was fixed weeks ago.
Detection is event-driven, not sampled. The driver calls
hwmon_notify_event()on every change, raisingPOLLPRIon theattribute, so the watcher blocks in
poll()and is woken on eachtransition. That matters because the attribute only reflects a
2-second window: a marginal supply produces bursts of short dips, and
any sanely-spaced sampler would walk straight past them. The poll also
carries an hourly timeout, but that is only a backstop to re-sync if an
event were ever missed. It costs no CPU while idle and runs in the
celery worker, a single long-lived process on both compose and Balena.
Nothing user-facing depends on that cadence either:
get_state()readsthe live attribute on every page render and writes back on
disagreement, so what an operator sees is current to the request.
What an operator sees
A persistent banner on every page (not just System Info, because
someone whose screen is glitching opens the Schedule page to look at
their content), plus a Power supply card on System Info. Red while it
is happening, amber once power recovers but the supply has already
proven it cannot hold up.
Copy is plain language and leads on using the official Raspberry Pi
power supply, stated as the fix rather than one option among four. Safe
to name outright, since the banner only ever renders when the
rpi_voltsensor exists. Sensor and kernel detail stay on System Infobehind a disclosure.
Boards with no sensor report "Not monitored" rather than implying a
healthy supply, and the API exposes
supportedfor the same reason: itis the only field separating "we checked and it is fine" from "we
cannot check".
Writes
A healthy device never writes to Redis at all. Observations that change
nothing are skipped, which matters because Redis runs
--appendonly yesand everySETis fsynced to the SD card. Writing constantly tothe card would be poor manners in a feature whose entire purpose is
avoiding card corruption.
An unreadable latch is treated as unknown, not empty, so a transient
Redis error cannot overwrite a real brown-out record and tell the
operator the supply is fine. Device-level faults (an unreadable boot
id, a Redis that will not answer) are logged once per boot rather than
on every wake-up and page render.
Also in here
under_voltageon/api/v2/info, reading the same state the UIrenders from so the two surfaces cannot drift.
collect_debug.shnow captures/api/v2/infointoapi-info.jsonrather than re-deriving device state in shell, so the bundle tracks
the API as more diagnostics move into the UI. Raw sysfs and the
kernel ring buffer stay as fallbacks for the case that prompts most
debug bundles: a stack too broken to answer HTTP.
vcgencmd get_throttledis kept but relabelled, since bits 0-3 and the thermalreadings are still valid. Its timeout is 90s, above the ~80s
get_node_ip()can block waiting on the host agent, because amissing host agent is itself a fault the bundle exists to diagnose.
Testing
1819 unit tests,
ruff check+ruff format --check, andmypyacross 181 files all clean.
Validated on all seven testbeds
Deployed to every board in the fleet, with the in-container blob md5s
asserted equal to the commit's blobs on each one. Every board runs
image
bada3fb-<board>, an ancestor of this branch's base with zerointervening changes to any overlaid file, so nothing unrelated rides
along.
supportedhwmon1hwmon1hwmon1hwmon1hwmon2start() -> FalseKernel 6.18.34 on every Pi. On all seven the banner was correctly
absent while healthy, and no board logged a spurious
Under-voltage alarm cleared.after boot.open question when this was written.
rpi_voltonhwmon2, nothwmon1(rp1_adctakeshwmon1). Matching on the device name rather than a fixed indexis what makes Pi 5 work; a hardcoded
hwmon1would have worked onsix boards out of seven and silently read an ADC on the newest one.
report
supported: false, render "Not monitored" rather thanimplying a healthy supply, and start no watcher thread at all.
Pi 5 by driving the actual
_watch_loopagainst acgroup.eventsattribute, which goes through the same
kernfs_generic_poll()pathas the hwmon alarm but can be triggered on demand. With the poll
timeout set to an hour so only a genuine event could wake it, the
loop observed the event and logged no failure. A negative control
with the previous code dropped the event and logged
Under-voltage watcher failed.poll()genuinely blocks. On Pi 4 and Pi 5 it sat the full3000 ms and returned no events after the initial read, confirming
the "first read clears the latched POLLPRI" assumption. Had that
been wrong the watcher would have spun a core at 100% on every Pi.
active: truesaw the watcher correct it on its next wake-up whilepreserving
countand history. Celery's whole MainProcess burned0.120 s over 40 s (0.3% of one core) including beat, so the watcher
itself is below that.
_watch_loopagainst a controllable file: a brown-out held across~6 wake-ups counted 1, recovery kept
seen_since_boot, and asecond distinct dip counted 2.
get_boot_id()forced toNone, the live reading stayed accurate,nothing was written to Redis, and the explanatory warning logged. A
planted
boot_id: nulllatch carryingseen_since_boot: true, count: 7was correctly discarded tofalse / 0rather than trusted.amber for a recovered one, on the Schedule page rather than only on
System Info.
The 512 MB Pi 3 A+ and the 1 GB Rock Pi 4 are both known-fragile under
memory pressure. The Rock Pi 4 sat at 0 MB swap free throughout, so it
got a server-only restart to avoid adding pressure and its watcher path
was exercised with a direct
start()probe instead. Neither board wasdestabilised (Rock Pi 4 ended at 109 MB available, unchanged).
All seven boards were restored to their pinned images, the overlay
confirmed gone from both containers, the test latch deleted, and the
locks released.
The seven-board sweep ran at
ba5b9621. Everything after it is thereview round below, which changed the poll-mask handling (separately
verified on hardware, above), the Redis write discipline, log
throttling, and the re-sample constant. The per-board sensor and
rendering results are unaffected by those.
What review and hardware testing changed
Four defects that unit tests alone did not surface:
Under-voltage alarm cleared.after everyboot, because the first reading of a pass differs from the initial
None. It reads as though an alarm had occurred, in exactly the loga support engineer greps. Found on the Pi 4.
boot_id: nullcompared equal to a current unknownNone, so a device that could not read its boot id would treat anarbitrarily old latch as current and never reset the warning again.
POLLERRwas treated as fatal.kernfs_generic_poll()returnsDEFAULT_POLLMASK | EPOLLERR | EPOLLPRIon every genuine changenotification, so a real brown-out arrives as
POLLPRI|POLLERRandwas raising, backing off 60s and dropping the transition. Measured
as mask
0xaon a Pi 5. The tests missed it because the fake pollerscripted a bare
POLLPRI, a mask the kernel never returns.GETfailure erased history, by falling backto an empty state that was then persisted.
Not covered
A genuine brown-out was not induced: that needs a deliberately weak
supply on a board someone can physically reach, and forcing one on a
shared testbed risks the SD-card corruption this feature warns about.
The notification mechanism is verified end to end on hardware (see
the
cgroup.eventstest above, same kernel path); what is untested isthe specific firmware condition that triggers it, which rests on the
raspberrypi-hwmondriver callinghwmon_notify_event(). The hourlyre-sample, plus the live read on every page render, both cover the case
where an event is missed.
🤖 Generated with Claude Code