26 Testing and Validation

HealthBook AI (internal code name: BabyCare) is validated at three levels: a pytest suite that drives the FastAPI app in-process, a Playwright suite that drives the real web client against a running server, and CI jobs that type-check and bundle the Android app. All tests use synthetic data only (fictional names, @test.invalid emails, generated PDFs and images) and isolated temporary data directories. Many tests encode the product's clinical safety rules directly, such as "a scheduled medicine is never counted as given".

Responsibilities

• Prove that the safety invariants hold: OCR is not verification, prescribed ≠ given, due ≠ administered, attendance ≠ treatment, susceptibility ≠ received, and a reminder is never proof of a dose.
• Prove that family isolation and authorization hold on every router.
• Exercise both database dialects (SQLite and PostgreSQL) and both storage backends (local and a faked Supabase transport).
• Catch client regressions: request storms, phone-width layouts, end-to-end journeys.

Test counts

The counts below come from grepping def test_ in tests/*.py and test( calls at the start of a line in tests/browser/*.spec.js:

Suite	Files	Tests
pytest (tests/*.py)	16 test modules + conftest.py	130 functions
Playwright (tests/browser/*.spec.js)	10 specs	20 tests (a raw grep for test( finds 23, because request_storm.spec.js contains three regex .test( calls)

Three pytest tests are skipped conditionally. Two PostgreSQL staging checks in test_cloud.py need BABYCARE_TEST_PG_URL. The live-uvicorn import test in test_import_history.py runs on Windows, or anywhere BABYCARE_RUN_IMPORT_TEST is set.

Inputs and outputs

Input	Used by	Output
BABYCARE_DATA_DIR (temp)	All Python tests	Throwaway SQLite database, key and encrypted files
BABYCARE_TEST_DATABASE_URL	conftest.py	Whole suite against PostgreSQL
BABYCARE_TEST_PG_URL	test_cloud.py	RLS hardening and SQLite-to-PostgreSQL copy checks
scripts/browser_fixture.py	Playwright	Fictional vaccination PDF and synthetic stool JPEGs (EXIF, WhatsApp-style name, no timestamp) in .cache/fixtures
BABYCARE_TEST_URL	Playwright	Target server (default http://127.0.0.1:8000)

Isolation

tests/conftest.py runs before any test module imports the app, and does four things:

1. Points tempfile.tempdir at .cache/test-temp inside the workspace.
2. Sets BABYCARE_DATA_DIR to a fresh suite_* directory, so no household data directory is ever touched.
3. If BABYCARE_TEST_DATABASE_URL is set, copies it into DATABASE_URL, giving a PostgreSQL staging run. Otherwise it removes DATABASE_URL, giving SQLite in the temp directory.
4. Defaults BABYCARE_STORAGE=local.

Individual modules also create their own tempfile.mkdtemp data roots and use fastapi.testclient.TestClient(app). External services are replaced: Supabase Storage and the camera worker use httpx.MockTransport, email uses the dev-outbox provider (refused in production) through a monkeypatched OUTBOX, and the vision provider uses explicit fakes, so nothing touches the network.

Test inventory: pytest

File	Tests	What it proves
test_api.py	8	Authentication and family isolation; OCR has no side effects; reference vaccines are never verified; a given date clears due reminders; medicines need an original, and revision disables reminders; a visit creates no administration; cross-family source documents are rejected; ambiguous card dates are never decided
test_workflows.py	18	Atomic, idempotent review with a human check; prescriptions never auto-activate; dose and next appointment stay separate; wrong page and cross-family are rejected; duplicate vaccine acknowledgment; distinct courses; worker dedupe and acknowledgement do not administer; DST rejection; lab unit and specimen separation; viewer invitation and revocation; signed-file expiry; cookie CSRF; vision consent without network; persistent rate limit
test_review_confirm.py	10	Only included rows are validated; field-level errors carry row and page; ambiguous dates are never guessed; excluded or empty rows never block; transcription-only save; double confirm never duplicates; other families cannot confirm
test_care.py	9	Idempotent quick-add; diaper threshold language; stool photo observation is never a diagnosis; scheduled is not counted as given; grouping by the child's local date; calendars; observational analytics; corrections keep history; Ask routing
test_stool_intake.py	12	EXIF, WhatsApp file name and metadata timestamp priority; unknown time requires the parent; future dates rejected; batch cards; hash duplicate detection; AI values are only suggestions until confirmed; timezone conversion; stool and diaper created once; red or pale appearance only suggests clinical review; privacy
test_lifetime.py	8	Drug classification; distinct courses and exposure days; duplicates and unknown dates; fever documented, not inferred; parent entries labelled; isolation and viewer rules; Ask cites evidence; idempotent transcription import
test_history_api.py	3	The history router requires auth, reflects only stored records, and is isolated per family
test_import_history.py	6	source_ref provenance blocks duplicates; series without inference; 7, 1 and 0 day reminders only for confirmed dates; Ask never invents administration; cross-family rejection; idempotent bundle import that activates nothing
test_export.py	10	PDF structure, bookmarks and footer; selective E. coli scope; originals appended in order; multi-page PDFs embedded without rasterizing; prescribed is not administered; due date is not a given date; isolation; ZIP without database ids; in-memory deterministic output
test_share.py	6	Preset scope excludes unrelated records; ZIP contains only selected originals; manual, category and date scopes; doctor link PIN, expiry, revoke and isolation; unverified labels; mobile base64 GET
test_cloud.py	17	Local storage never overwrites; Supabase contract with private bucket only, refusal of a public bucket, HTTPS plus service key required; storage round trip; scheduler hidden without a key; scheduler never activates unconfirmed reminders; web push lifecycle and generic payload; VAPID keys; datasync copies ciphertext verbatim; full backup create, verify and restore; PostgreSQL hardening; SQLite-to-PostgreSQL copy
test_delivery_security.py	6	Worker retries and receipts; backup restore never overwrites; a missing key never generates a replacement; account deletion spares other families; invalid files and future administrations rejected; the independent worker persists notifications
test_password_reset.py	7	Full reset flow preserves data and signs out sessions; generic answer for unknown email; hashed, expiring, replaced tokens; weak passwords rejected; per-email limit; secrets never logged; fragment link on the public origin
test_profiles.py	4	Age groups and default modules; profiles of any age; per-profile vitals; HealthBook branding
test_camera.py	5	Credentials stripped and private network enforced; RTSP probe handling; analyser emits movement classes only; API hides credentials; simulated worker reconnects and ingests
test_migrations.py	1	A populated legacy database migrates, gets encrypted, and keeps its originals

Test inventory: Playwright

Spec	Tests	Journey
care.spec.js	2	Upload card, correct extraction, save verified history and appointment; responsive navigation and growth form
care_journal.spec.js	3	Quick-add, medicine GIVEN, stool photo, calendar; one-handed phone width; dialog survives re-render
review_confirm.spec.js	1	Precise errors with page jump; excluded rows; transcription-only save
stool_intake.spec.js	1	Batch upload with detected times and AI suggestions placed on the historical calendar
lifetime.spec.js	2	Tiles, evidence, calendar, parent entry, Ask; phone width
export_record.spec.js	2	Preview counts, selective export, PDF download; phone width
share.spec.js	1	Wizard, ZIP, secure link, doctor page, revoke
family_workspace.spec.js	1	Branding, adult profile with vitals, member switch
password_reset.spec.js	1	Generic message, reset page, sessions ended
request_storm.spec.js	6	Zero patient requests on login; no reload loops; one POST per add; one GET per day or child change; logout aborts loading; 429 gives a quiet cooldown

playwright.config.js sets testDir: ./tests/browser, a 45 s timeout, workers: 1, headless Chromium, trace: retain-on-failure and the list reporter. When CI is set, it starts uvicorn api.main:app on port 8000 with BABYCARE_DATA_DIR=.cache/browser-validation and waits for /api/health.

CI pipeline

flowchart LR
    P["Push to main, pull request,<br/>or manual dispatch"] --> B["backend job"]
    P --> W["web job"]
    P --> M["mobile job"]
    P --> APK["android-apk job<br/>main or dispatch only"]
    B --> B1["compileall"]
    B1 --> B2["pytest on SQLite"]
    B2 --> B3["pytest on PostgreSQL 17 service<br/>synthetic data"]
    B3 --> B4["pip-audit, non-blocking"]
    W --> W1["node check on web js"]
    W1 --> W2["browser fixtures"]
    W2 --> W3["Playwright Chromium"]
    M --> M1["TypeScript typecheck"]
    M1 --> M2["Expo export Android and iOS"]
    M2 --> M3["npm audit, non-blocking"]
    APK --> A1["expo prebuild"]
    A1 --> A2["Gradle assembleDebug artifact"]

Loading

The backend job installs tesseract-ocr, so the OCR paths run for real. Its PostgreSQL pass sets both BABYCARE_TEST_DATABASE_URL and BABYCARE_TEST_PG_URL and skips test_migrations.py, which targets the legacy SQLite file. Playwright traces are uploaded only on failure.

What is not validated

• Native Android runtime behaviour: CI type-checks, bundles and builds a debug APK, but runs no device or emulator tests.
• Real third-party delivery: Expo push, Web Push endpoints, email providers, the hosted Supabase bucket and the external vision provider are all faked.
• OCR accuracy on real documents (only fictional fixtures are used), PDF visual fidelity beyond structure checks, and camera analysis on real video streams.
• Load, concurrency at scale, accessibility audits, and cross-browser runs (Chromium only).
• Dependency audits (pip-audit, npm audit) do not fail the build.
• No test claims clinical correctness; the tests prove that the app records and labels information without inferring it.

Implementation

tests/conftest.py, tests/*.py, tests/browser/*.spec.js, playwright.config.js, pytest.ini, scripts/browser_fixture.py, .github/workflows/ci.yml

Related pages

Deployment and CI/CD · Security and Privacy Model · OCR and Human Verification Workflow · Limitations · Backend Architecture