Testing and CI

Testing and CI

How imPress is tested: the single test runner, every suite, the CI pipeline, the test strategy, and the hardware bench checklist. References: docs/guides/testing.md, docs/testing/TEST_STRATEGY.md, docs/testing/HARDWARE_VALIDATION.md.

Contents

• Running the tests
• The suites
• What each layer catches
• End-to-end and fault-injection tests
• Continuous integration
• Rules for new tests
• Hardware validation

---

Running the tests

python3 -m venv .venv && . .venv/bin/activate
pip install -r backend/requirements-dev.txt
python run_tests.py                        # every default suite, about a minute
python run_tests.py --list                 # what exists
python run_tests.py -s backend -s host     # selected suites (prefix match)
python run_tests.py --with-ui              # + Playwright browser tests
python run_tests.py --with-idf             # + ESP-IDF builds of all three projects (Docker if no IDF)
python run_tests.py --with-frontend        # + the React build
python run_tests.py -s backend --coverage reports   # + coverage
python run_tests.py --json report.json --markdown report.md

The runner prints a report with each suite's status, counts and duration, the slowest tests, and details of every failure.

Runner behaviour:

• Progress: long suites show a live status line (elapsed time and the latest output), plus a heartbeat in CI.
• Missing packages: if the interpreter lacks a package a suite needs, the suite fails once, with the exact pip install command, instead of hundreds of import errors.
• Interrupting: Ctrl-C stops the whole process tree cleanly, prints the report, and exits 130. --timeout SECONDS bounds every suite.
• Unavailable tools: a suite whose tools are missing (no C compiler, no Docker) is reported as SKIP with the reason.

The suites

Suite	What	Tests (v2.1)
backend	every router and service: auth and sessions, roles, classes, students, quizzes and polls, device API, presence, health, inventory, migrations, CSV imports, firmware registry and signing, deployments, device keys, end-to-end flows, fault injection, route guards	414
firmware-static	structural rules in firmware sources, plus firmware ↔ backend agreement	58
repo	the CI workflow, repository hygiene, documentation consistency, the install scripts, the runner itself	138
host:* (8 suites)	real firmware C compiled on your machine with AddressSanitizer and UBSan: the protocol (frames, SPI slots, FIFOs, JSON reader), mesh de-duplication, the gateway's SPI slave, WebSocket commands, configuration (including device keys and the boot counter), OTA offer logic and student set, and the student's identity rules	41
ui (optional)	Playwright with Chromium (or a local Chrome) against a real server: navigation, CSV imports, the firmware page and signatures, deployments, the module inventory, diagnostics and key reset; screenshots for review	15
idf:* (optional)	ESP-IDF v6.1 builds of class_c6, class_s3, student	3 builds
frontend (optional)	npm ci + vite build	1 build

The latest full local run, on varun/v2.1 at a1662c2 (2026-10-09, run_tests.py --with-ui): 666 passed, 0 failed, 0 skipped in 2 min 15 s. Backend line coverage: 88 % (measured before the #66 work).

What each layer catches

Layer	Catches	Can't catch
Host C tests	logic bugs in pure firmware code; memory errors and undefined behaviour (ASan/UBSan)	anything that needs drivers, radios or timing
Firmware structural tests	ordering and safety rules (verify before boot; no flash write before the HTTP status check; mark-valid only when healthy; every backend client gets the TLS settings); names agreeing between firmware and backend (heartbeat fields, reset reasons, partition sizes)	whether the code runs
ESP-IDF builds	compile and link errors; size regressions	runtime behaviour
Backend tests	every endpoint's contract, validation, access control, state machines, migrations	radio, SPI, real devices
Contract tests	the backend and firmware drifting apart (WebSocket frames, heartbeat fields, image constants)
End-to-end (simulated)	whole flows through a simulated gateway	real firmware behaviour
Fault injection	failed commits, lost responses, racing copies, offline gateways, corrupt downloads, power loss, unhealthy images	physical faults
Browser tests	pages render; actions reach the API; live refresh	visual design
Install smoke (CI)	the documented install path on a fresh machine, over HTTP and HTTPS	other operating systems

End-to-end and fault-injection tests

backend/tests/gateway_sim.py is a simulated gateway. It follows the firmware's documented behaviour:

• it registers and sends heartbeats;
• it buffers mesh messages and posts them in batches, keeping a batch on a 5xx response and dropping it on a 4xx, as the firmware does;
• it runs the OTA client's steps, with optional faults: a corrupted download, power loss after a step, or an image that fails its health check.

Test	Flow
test_e2e_quiz.py	class, then CSV questions, then a gateway linked by room code, then modules join, then quiz frames on the device socket, then answers relayed in batches (one duplicated by the mesh), then live counts on the teacher socket, then results. A late press is not counted
test_e2e_ota.py	a staged rollout to three rooms; a corrupted download refused, then retried; power loss mid-install, timed out and retried; an unhealthy image rolled back, with the rollout paused
test_fault_injection.py	a failed database commit (nothing stored, 503, the retry stores once); a lost response's resend (harmless); answers buffered while offline (counted only while the quiz is open); two copies racing (one stored); a malformed batch (4xx)

Fault injection found a real bug: a database error on commit used to be an unhandled 500. It is now a 503 with nothing saved.

Continuous integration

.github/workflows/ci.yml runs on every pull request, on pushes to the long-lived branches (main, v2, v3, varun/**), and on demand.

Job	What it runs
Repository and CI checks	the hash-locked install, actionlint, ruff, run_tests.py --suite repo
Dependency audit	pip-audit --strict, npm audit --audit-level=high
Backend tests	run_tests.py --suite backend with coverage (report uploaded)
Firmware structural tests	run_tests.py --suite firmware-static
Firmware host tests	run_tests.py --suite host (gcc, ASan/UBSan)
ESP-IDF v6.1 build (×3)	idf.py build and idf.py size for each project; images with SHA256SUMS uploaded for 7 days
ESP-IDF v6.1 signed and TLS builds	a throwaway signing key, a signed gateway build checked with verify_firmware.py --key; the gateway with public-CA TLS; the hub with an embedded throwaway CA
Frontend build	npm ci + Vite build
Browser UI checks	Playwright with Chromium; screenshots uploaded
Clean install + smoke test	install.sh on a fresh runner, start.sh, smoke_test.py over HTTP; then again over HTTPS with a throwaway certificate, checking that an untrusted certificate is refused
CI result	passes only if every job above passed: the one required check

Pipeline properties:

• Least privilege: the workflow can only read the repository.
• Pinned supply chain: every action is pinned to a commit SHA, the ESP-IDF image to a digest, and Python packages to a hash-locked set (backend/requirements-lock.txt). Frontend packages come from package-lock.json (npm ci).
• Cancellation: a newer push to the same branch cancels the running pipeline.
• Self-checking: tests/test_ci_workflow.py checks the workflow itself. Every job must have a timeout, every action must be pinned, every runner suite must be wired into CI, and the result gate must depend on every job.

The README shows one CI badge, for varun/v2.1. GitHub's badges are per workflow, not per job.

Release workflow

.github/workflows/release.yml publishes releases and the backend image (docs/guides/releases.md):

Trigger	What it does
CI passed on a push to an integration line <owner>/vX.Y	computes the release <owner>/vX.Y.N and the version (MAJOR.MINOR from VERSION, patch = pushes since vMAJOR.MINOR.0) and stamps it into the build; builds the backend image, runs smoke_test.py against it and pushes edge, sha-<commit> and <version>; tags the commit, builds the firmware, packages the bundle, firmware and documentation zips with checksums, and publishes the release with the merged pull requests listed. It needs the line's notes template, docs/releases/vMAJOR.MINOR.md
an issue changes, daily, or a manual run	sets each release Beta (open testing issues in a milestone that gates it: its line, a line it is built on, or its exact release) or Stable, redraws the status section (badges, alert, test table) and moves the image tags beta and latest

It never moves a tag or replaces a published release. tests/test_release.py checks its status rules (against a fake GitHub), its safety settings, the Dockerfile and the release notes.

Rules for new tests

1. Test behaviour through the API. Touch the database only to create states the API can't produce (time passing, a crash) or to check stored rows.
2. Never block forever. WebSocket tests broadcast a marker after the action and read until it arrives. Never sleep waiting for something.
3. Prove the test can fail: break the code it guards once (a mutation) and watch it fail before trusting it.
4. One source of truth across the boundary. When firmware and backend must agree, the test reads both (the frame fixture, schemas.py, partitions.csv, health.py) instead of copying a value.
5. Fix flakiness, don't retry it. A test that fails intermittently is a bug in the test or the code.
6. Simulation is not hardware. A passing simulated test never counts as hardware verification.

Writing a backend test:

from conftest import DEVICE, auth, login, make_class

def test_my_rule(client, db):
    h = auth(login(client))
    cid = make_class(client, h, activate=True)
    r = client.post("/api/device/register", headers=DEVICE, json={"mac_address": "48:F6:EE:00:00:01", "device_type": "c6"})
    assert r.status_code == 200

client starts the app on a fresh temporary database; db(fn) runs an async ORM function against it. Helpers: login, auth, make_user, make_class, make_student, upload_firmware, DEVICE (device-key headers), and gateway_sim.Gateway.

Hardware validation

As of v2.1, none of these checks has been run on real boards. Each check has a test issue with numbered cases, expected results, edge cases and pass criteria; the tracker is #94. The issues are in milestone v2.1.0 with the testing label, so the 2.1.0 release stays Beta until they are closed. Each check below has a procedure and pass criteria in docs/testing/HARDWARE_VALIDATION.md. Record a result there as passed | failed — <boards>, firmware <version + commit>, backend <version>, <date>, <who>; <notes>.

#	Check	Test issue	Status
H1	SPI link between hub and gateway (30 min, no CRC errors, hub link up)	#84	not run
H2	Student join and leave (connected, then seen within ~90 s)	#85	not run
H3	Multi-hop relay (a module out of the hub's range)	#85	not run
H4	Quiz answers end to end with 3 modules	#86	not run
H5	Burst: 30 / 100 / 300 modules pressing within 1 s	#87	not run
H6	First serial flash with rollback (gateway)	#84	not run
H7	Gateway OTA update	#88	not run
H8	Hub OTA update	#89	not run
H9	Gateway rollback (an image with a wrong SSID)	#88	not run
H10	Power loss mid-download	#88	not run
H11	Staged rollout to 3 gateways	#89	not run
H12	Reset reasons (power-on, panic, watchdog)	#90	not run
H13	Boot counter across power cycles	#90	not run
H14	Hub link loss (unplug SPI)	#90	not run
H15	Hub without PSRAM	#91	not run
H16	Reduced-flash hub profile on an 8 MB quad-flash S3	#91	not run
H17	Mesh range	#93	not run
H18	Power draw	#93	not run
H19	Endurance: a school day with 30 modules	#87	not run

The v2.1 security work adds checks to run on hardware too:

• Device keys: enrolment, and reset with re-enrolment.
• Signed firmware: a signed device accepts a signed image and refuses an unsigned one.
• Device TLS: a TLS handshake from a gateway and from a hub.