10 Medical Document Ingestion

Medical document ingestion is how an original paper record, such as a prescription, discharge summary, lab report or vaccination card, gets into HealthBook AI (internal code name: BabyCare). The original file is the anchor for everything else. It is validated, deduplicated, encrypted and stored privately. Any structured data that later comes out of it is either a suggestion or a caregiver-confirmed record, and it keeps a pointer back to the document and page it came from.

Responsibilities

• Accept only real PDF and image files, decided by the file's bytes, never by its name or the client-supplied MIME type.
• Enforce size, page and pixel limits before anything is stored.
• Refuse an exact duplicate of a file already stored for the same profile.
• Encrypt the original before it reaches storage, and never expose a public URL.
• Record provenance (category, record date, title, source_ref, import_batch, notes) so that structured records can cite the original.
• Serve page images, thumbnails and back-links for review and timeline views.
• Support bulk import of a reviewed scan bundle without confirming any medical fact.

Inputs and outputs

Endpoint	Input	Output
POST /api/patients/{pid}/documents	multipart file, category, title, record_date, source_ref, import_batch, notes	document metadata (201)
GET /api/patients/{pid}/documents	none	list, newest record_date first, undated last
PATCH /api/documents/{id}	title, category, record_date, notes	updated metadata; earlier values snapshotted
GET /api/documents/{id}/download	none	decrypted original as an attachment, nosniff
GET /api/documents/{id}/pages/{page}	page number	PNG render of one page, at most 1800 px
GET /api/documents/{id}/thumbnail?w=	width 80 to 800	PNG (PDF page 1) or JPEG (image)
GET /api/documents/{id}/links	none	every record citing this document, plus review status
POST /api/documents/{id}/signed-access	none	single-file link valid for 120 s, bound to the session

The API never returns storage_name, sha256 or extracted_cipher (see as_json in api/main.py).

Upload validation

Check	Rule	Failure
Ownership	own_patient on the profile	403/404
Category	one of other, vaccination_card, prescription, lab, culture, discharge, hospital, birth, imaging, bill, admission, consultation	422
Size	at most 12 MB (the server reads 12 MB + 1 byte and rejects anything longer)	413
Magic bytes	%PDF-, the PNG signature, FF D8 FF (JPEG), or an ISO-BMFF ftyp box with brand heic, heix, mif1 or hevc	415
PDF	not encrypted, at most 35 pages (opened with PyMuPDF)	400
Image	at most 25 megapixels, then Image.verify() must pass; HEIC only if the optional decoder is available	413 / 415
Damaged file	any parser exception	400
Duplicate content	SHA-256 of the raw bytes already stored for this profile	409 "already uploaded as document #N"
source_ref	[A-Za-z0-9_.-]{1,40}, unique per profile	422 / 409

After validation the raw bytes are Fernet-encrypted (api/crypto.py) and written under a random name (secrets.token_hex(24) plus .enc). The local backend opens the file with O_EXCL, so an existing original can never be overwritten. In cloud mode the object goes to a private storage bucket that only the backend can reach (see Encryption and Private Storage). If the database insert fails afterwards, the encrypted blob is deleted. If storage is unavailable, the API returns 503 and saves nothing.

Ingestion flow

Diagram: Document ingestion to confirmed medical data

flowchart TD
    U["Caregiver upload or bundle import"] --> V["Validate magic bytes, size, pages, megapixels"]
    V --> H["SHA-256 duplicate check per profile"]
    H --> E["Fernet encrypt original"]
    E --> S["Private storage, random object name"]
    S --> D["Document row, status not_extracted"]
    D --> X["OCR or text extraction, local by default"]
    X --> T["Encrypted candidate transcription"]
    T --> C["Structured suggestions, verified false"]
    C --> W["OCR suggestion is not a confirmed medical fact"]
    W --> R["Human review against original page"]
    R -->|"included and checked"| K["Confirmed medical data with source page"]
    R -->|"excluded"| Z["Kept as transcription only"]
    K --> TL["History timeline"]
    K --> MD["Medicine courses as drafts"]
    K --> LB["Lab observations"]
    K --> VX["Vaccination rows"]

Loading

Extraction and review are covered in OCR and Human Verification Workflow. Even after review, a confirmed prescription row is still a medicine draft. It needs a separate prescription verification before any reminder exists (see Medicines and Administration Model).

Metadata and provenance

• Category drives which suggestion extractor runs. If a document is still other when OCR runs, classify() may assign a category from keywords.
• record_date is the document date. Undated documents remain undated and are listed separately in the history view, not given a guessed date.
• source_ref / import_batch record the bundle page identifier and the batch label. The history view marks documents whose notes start with "date inferred" as date_inferred, not document_date.
• PATCH may change only title, category, record date and notes. Only record_date may be set to null. Every change first writes a RecordRevision snapshot (api/history.py) and then an audit row.
• Back-links (/links) list the medicines, vaccines, observations, admissions (source or discharge document), visits, treatment administrations, growth rows and episodes that cite the document.

Bulk import (scripts/import_bundle.py)

This operator tool uploads a reviewed scan bundle through the normal authenticated API.

Safeguard	Behaviour
Target	only http://127.0.0.1, localhost or ::1
Checksums	every file's SHA-256 must match manifest.json, otherwise the import stops
Identity	--dob (or expected_dob) must equal the profile DOB, otherwise the import stops before any upload
Confirmation	type IMPORT unless --yes is passed; --dry-run sends nothing
Credentials	token from an environment variable, or an interactive prompt; nothing written to disk
Idempotency	reuses documents by source_ref, or by the 409 duplicate-hash response; vaccines, medicines, observations, admissions, growth rows and episodes are matched on natural keys
Never confirms	vaccines are created unconfirmed, medicines as drafts with no schedule, no reminders, no administration events; handwritten-page review drafts are saved with checked_against_original: false
OCR	--ocr runs local extraction only for documents with no stored text

Curated lab rows pass through their own verified value. The API still refuses verified for a lab row that has no source document.

Security boundaries

• Every document route resolves the row through own_resource / own_patient, so profile isolation applies (see Family Profile Isolation).
• Originals and transcriptions are encrypted at rest. Download filenames are sanitised.
• Signed file tickets expire after 120 seconds and stop working when the issuing session ends.

Clinical safety rules

• OCR extracted ≠ clinically verified. Upload and extraction never create a confirmed fact or a reminder.
• Structured records keep provenance: source_document_id plus source_page, and safe_doc() rejects a page outside the original.

Implementation

api/main.py (upload, list, PATCH, download, extract), api/storage.py, api/crypto.py, api/workflows.py (pages, signed access), api/history_api.py (thumbnail, links), api/history.py, scripts/import_bundle.py.

Related pages

OCR and Human Verification Workflow · Encryption and Private Storage · Database and Data Model · API Design · Security and Privacy Model