feat(server): warn in the UI when storage is failing

[vpetersson-bot]

Issues Fixed

No tracking issue. Follow-up to the under-voltage warning (PR 3285), which covered the first of the two failures a Raspberry Pi actually dies of. This covers the second. Both previously showed up only as unexplained behaviour: the screen keeps playing while every change the operator makes silently fails to stick.

Description

SD cards have no health register, so unlike under-voltage there is no single sensor to read. The verdict is assembled from four container-readable sources, so Balena fleets are covered without a host agent:

• ext4 superblock error counters. Durable: they survive reboots and are cleared only by fsck.
• A write check — write, fsync, POSIX_FADV_DONTNEED, read back, compare. Needed because the counters go quiet once a filesystem stops: nothing is going wrong any more because nothing is being written.
• eMMC life_time / pre_eol_info where the board has them.
• Card identity from the MMC/SD registers.

The device is resolved from /proc/self/mountinfo rather than assumed, so the bind mount, Balena's volume and an x86 SSD all work with no per-platform case.

SMART extends this to x86/arm64, where a SATA/NVMe device reports wear nowhere in sysfs. Reading it needs an ioctl, so anthias-viewer (the one container privileged everywhere, including both Balena templates) samples it and publishes a TTL'd Redis fact the server reads — the same division of labour as HDMI-CEC. It folds into the existing wear_pct/pre_eol fields, so the UI and API need no separate SSD branch.

Six banner states, because the fixes differ: a full card, bad blocks, wear and a dying card need different advice, and the noun follows the hardware so an x86 player isn't told to replace a memory card. The under-voltage banner is generalised into a shared .device-alert block so the two stack cleanly, power first, without repeating each other.

Also: storage on /api/v2/info, a System Info card, and a storage.txt section in the debug bundle.

Validated on hardware, which changed the code five times

Driving this against a real ext4 broken with the kernel's trigger_fs_error, and against the full testbed fleet, found bugs no unit test would have:

• A stopped filesystem is spelled emergency_ro, not ro, on kernels 6.18 and 7.0. Writes failed with EROFS while both the ro option and statvfs's ST_RDONLY stayed clear. read_only was returning False on a filesystem refusing every write.
• privileged: true on a container does not mean the process is privileged. start_viewer.sh drops to the unprivileged viewer user, so smartctl returned "Permission denied" and every board would have reported its drive unsupported. Fixed with one NOPASSWD sudoers rule scoped to that binary.
• Bad blocks are not wear. The x86 testbed's SSD has zero wear, a PASSED self-assessment, and 8 reallocated/pending sectors. The wear copy would have claimed it had used "most of" its rated writes.
• 0 is falsy in a Django template, so a genuine 0% wear reading rendered as "unknown".
• SD and eMMC manufacturer IDs are different namespaces. Vendor tables now come from mmc-utils, which keeps two tables for exactly that reason.

Regression sweep across all seven boards (x86, pi5, pi4, pi3-64, pi3a-32, pi2, rock4): every board reports ok, no false positives, write check 1.2-16.6 ms, nothing left behind, smartctl correctly absent on the SBCs. The Rock Pi 4 exercised the eMMC wear path.

The docstring's claim that Raspberry Pi OS mounts root errors=remount-ro was also wrong and is corrected: every Pi reports Errors behavior: Continue, so a dying Pi card limps rather than stopping. The Rock Pi 4 is the opposite. Both endings are real, which is why both signals are read.

Checklist

• I have performed a self-review of my own code.
• New and existing unit tests pass locally and on CI with my changes.
• I have done an end-to-end test for Raspberry Pi devices.
• I have tested my changes for x86 devices.
• I added a documentation for the changes I have made (when necessary).

End-to-end on x86: viewer collects SMART, publishes to Redis, server folds it in, /api/v2/info returns the real drive's figures and the page renders the right banner. Pi coverage is the fleet sweep above plus the emergency_ro test on a loopback filesystem, never a testbed rootfs.

Ops finding: the x86 testbed's SSD has 4 reallocated + 4 pending sectors at 2226 power-on hours. testbed-qa already documents EIO wedging sshd on that board; this is the same drive. Worth replacing before it takes the board out mid-burn-in.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds end-to-end “storage health” diagnostics to Anthias, surfacing impending storage failure (ext4 error counters, write/read-back checks, eMMC wear, and viewer-sampled SMART) in the web UI, /api/v2/info, and the debug bundle, so operators get actionable warnings before changes start silently failing to persist.

Changes:

• Introduces storage-health detection (+ latch) and SMART collection, with a Celery watcher thread updating Redis state and the viewer publishing SMART facts.
• Adds UI surfaces: stacked device-alert banners across all pages and a new Storage Health card on System Info.
• Extends /api/v2/info and the debug bundle to include storage health, backed by substantial new test coverage.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/image_builder/utils.py	Installs smartmontools only on boards that can boot from SMART-capable storage.
tests/test_template_views.py	Adds banner + System Info rendering tests and stubs storage health for determinism.
tests/test_storage_watcher.py	Adds unit tests for the new storage-health watcher thread.
tests/test_storage_health.py	Adds extensive tests for storage probing, write-checking, latching, and SMART merge behavior.
tests/test_smart.py	Adds tests for SMART parsing, smartctl execution behavior, and Redis publish/read.
src/anthias_viewer/init.py	Adds viewer-side SMART sampling loop that publishes TTL’d Redis facts.
src/anthias_server/lib/storage_watcher.py	Adds Celery-worker thread to keep storage-health latch current and log transitions.
src/anthias_server/celery_tasks.py	Starts the storage watcher on Celery worker_ready.
src/anthias_server/app/templates/system_info.html	Adds a new Storage Health stat card and unifies “verdict” styling.
src/anthias_server/app/templates/_storage_warning.html	Adds a new storage warning banner rendered on every page.
src/anthias_server/app/templates/_power_warning.html	Refactors power banner to use shared “device-alert” styling/container.
src/anthias_server/app/templates/_layout.html	Switches to a shared _device_alerts.html include above <main>.
src/anthias_server/app/templates/_device_alerts.html	New wrapper template stacking power + storage banners.
src/anthias_server/app/static/sass/_styles.scss	Introduces .device-alert component styles and shared .status-verdict styles.
src/anthias_server/app/page_context.py	Adds storage warning/card context generation and state shaping for templates.
src/anthias_server/api/views/v2.py	Adds storage to /api/v2/info plus OpenAPI schema description.
src/anthias_server/api/tests/test_info_endpoints.py	Adds API tests covering the new storage payload and failure modes.
src/anthias_common/storage_health.py	New core storage-health module: mount resolution, ext4 counters, write-check, latch, SMART merge.
src/anthias_common/smart.py	New SMART module: viewer-side sampling via smartctl + normalization + TTL’d Redis publishing.
docker/Dockerfile.viewer.j2	Adds a sudoers rule enabling the unprivileged viewer to run smartctl as root.
bin/collect_debug.sh	Adds storage.txt (ext4 counters, MMC identity/wear, SMART, kernel storage errors) to debug bundles.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 88.77698% with 78 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (master@0bf0790). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/anthias_common/storage_health.py	88.15%	32 Missing and 13 partials ⚠️
src/anthias_common/smart.py	88.96%	10 Missing and 6 partials ⚠️
src/anthias_server/lib/storage_watcher.py	85.96%	4 Missing and 4 partials ⚠️
src/anthias_server/app/page_context.py	90.19%	4 Missing and 1 partial ⚠️
src/anthias_server/celery_tasks.py	42.85%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3286   +/-   ##
=========================================
  Coverage          ?   90.29%           
=========================================
  Files             ?       84           
  Lines             ?     9920           
  Branches          ?     1097           
=========================================
  Hits              ?     8957           
  Misses            ?      709           
  Partials          ?      254           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (5)

src/anthias_viewer/init.py:2668

• SMART sampling resolves the backing disk via storage_health.probe(), but calling it with no data_dir uses storage_health.default_data_dir() (HOME-based). In the viewer process HOME is for the viewer user, so this can resolve the wrong mount (often the container root/overlay) and either skip SMART or report the wrong device. Pass the same config dir the server uses so both containers agree on the backing disk.

                facts = storage_health.probe()
                disk = facts.get('disk')
                # Only ever a real answer on SATA/NVMe. SD and eMMC
                # have their own registers and smartctl has nothing to
                # say about them, so most of the fleet skips this.
                if disk and facts['media']['kind'] == 'disk':
                    smart.publish(r, smart.collect(f'/dev/{disk}'))

src/anthias_server/api/views/v2.py:1149

• The OpenAPI schema for /api/v2/info.storage.media says wear fields are only populated on eMMC, but storage_health also populates wear_pct/pre_eol from SMART for SATA/NVMe disks (kind == 'disk'). This makes the schema description misleading for API consumers.

                                    'What the device is. `kind` is '
                                    '`sd`, `emmc`, `disk` or '
                                    '`unknown`; the wear fields are '
                                    'populated on eMMC only, since '
                                    'SD cards do not report health.'

tests/test_template_views.py:3742

• This autouse fixture patches get_state() to always return the same healthy dict instance. page_context._storage_card() mutates the dict it receives (adds/overwrites keys like warn/kind/parsed timestamps), so returning a shared instance can leak state between tests and make ordering matter. Return a fresh copy on each call.

    with mock.patch(
        'anthias_server.app.page_context.storage_health.get_state',
        return_value=healthy,
    ):
        yield

src/anthias_server/lib/storage_watcher.py:90

• When storage status is STATUS_WEAR, this warning logs wear=%s%% directly from state['media']['wear_pct']. For SMART-driven wear warnings, wear_pct can legitimately be None (e.g., failed self-assessment or bad sectors without a vendor wear attribute), which would log as wear=None% and be confusing. Format wear as "unknown" when wear_pct is None.

    elif status == storage_health.STATUS_WEAR:
        logger.warning(
            'Storage on %s is nearing end of life (wear=%s%%, pre_eol=%s).',
            state.get('device'),
            state.get('media', {}).get('wear_pct'),

src/anthias_server/api/views/v2.py:1179

• The /api/v2/info OpenAPI schema for storage.media is missing the wear_is_exact field, but anthias_common.storage_health.read_media_info() always returns it (True for NVMe, False otherwise). This makes the documented response shape incomplete.

                                    'wear_pct': {'type': ['integer', 'null']},
                                    'pre_eol': {'type': ['string', 'null']},

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/anthias_viewer/init.py:2668

• storage_health.probe() defaults to $HOME/.anthias, but the viewer runs as the viewer user and $HOME may not be the same directory the server uses for Anthias data/config. That can make the viewer resolve and publish SMART for the wrong backing device (e.g., the container rootfs) and disagree with the server/UI about which disk is being described. Pass the same config dir the server uses when probing.

                facts = storage_health.probe()
                disk = facts.get('disk')
                # Only ever a real answer on SATA/NVMe. SD and eMMC
                # have their own registers and smartctl has nothing to
                # say about them, so most of the fleet skips this.
                if disk and facts['media']['kind'] == 'disk':
                    smart.publish(r, smart.collect(f'/dev/{disk}'))

src/anthias_server/app/templates/system_info.html:186

• The failing-storage System Info card says “Replace it…”, which is incorrect/unclear for non-replaceable storage (e.g. soldered eMMC) and also ambiguous for x86 (“it” could be the player). Since the banner at the top already gives the correct hardware-specific action, this sentence should just refer the operator to that warning.

          {% if storage.errors_new %}{{ storage.errors_new|intcomma }} storage
          error{{ storage.errors_new|pluralize }}{% else %}A storage error{% endif %}
          since this player last restarted.
          {% endif %}
          Replace it and see the warning at the top of the page.
        </p>

src/anthias_server/lib/storage_watcher.py:92

• This log message renders wear=None% when the status is wear but the underlying signal is pre_eol (or other SMART evidence) without a numeric wear percentage. That produces a misleading/garbled message in support logs; prefer logging wear=unknown (or similar) when no percentage is available.

            'Storage on %s is nearing end of life (wear=%s%%, pre_eol=%s).',
            state.get('device'),
            state.get('media', {}).get('wear_pct'),
            state.get('media', {}).get('pre_eol'),
        )

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/anthias_server/lib/storage_watcher.py:126

• last_write_check is updated whenever write_check is true, even if storage_health.record_check() didn’t actually run the write check (it skips writes when the filesystem can’t be resolved). If supported temporarily flips false, this stamps the interval anyway and can delay the next real write check after recovery.

                # Stamped after the call, not before: on a card that
                # is struggling the check itself can take a while,
                # and the interval should be a gap between checks
                # rather than a deadline they overrun.
                last_write_check = time.monotonic()

tests/test_template_views.py:3742

• The autouse _healthy_storage fixture returns the same healthy dict instance for every call to storage_health.get_state. page_context._storage_card() mutates the returned dict (adds warn, kind, parses timestamps, etc.), so later requests/tests can observe cross-test state and become order-dependent/flaky.

    with mock.patch(
        'anthias_server.app.page_context.storage_health.get_state',
        return_value=healthy,
    ):
        yield

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud