An evidence-first life-insurance underwriting assistant — a complete, runnable POC that pairs a deterministic, plan-aware rules engine with a citation-grounded RAG pipeline, an MLX LoRA fine-tune of Llama-3.2-3B, MLflow-tracked experiments, and a FastAPI review workbench. Built as a customer-proposal proof of concept; published here as a sanitized, fully synthetic snapshot.
Everything in this repo is fictional. The insurer ("Acme Life"), the regulator, the plans, the underwriting manuals, and every applicant are synthetic stand-ins authored for this demo. The engineering — and the experiment numbers in docs/RESULTS.md — are real.
make setup # python3 -m venv + core deps (fastapi, mlflow, pytest…)
make test # 34 unit/integration tests — engine, plan profiles, fixtures, API
make demo # 11-point acceptance gate, then the workbench on http://127.0.0.1:8800Cached mode serves deterministic, pre-generated rating sheets — the full UI flow (analyze → evidence panel → rating sheet → feedback → PDF export) works offline. Live model modes are opt-in (see Going live).
An underwriter picks a case; the workbench produces a six-field EMR rating sheet — risk factors, extra-mortality class, extra premium, decision type, refer-to-human, rationale — with every debit grounded in a cited clause of the governing manuals. The assistant recommends; the underwriter decides.
The design bet is evidence first: the LLM is never the source of truth for the arithmetic.
- Deterministic rules engine (
scripts/build_iteration1_artifacts.py) — the Numerical Rating Method: additive impairment debits → EMR class bands → decision + referral triggers. Pure functions, byte-reproducible, unit-tested against the manuals. - Plan-aware overlay (
data/plans/) — each plan declares its underwriting levers in a JSON profile (EMR acceptance cap, entry-age/sum-assured ceilings, medical mode). The same medical facts land differently across plans: a Class IV case is extra premium on an open endowment, referred on Horizon Term (cap III), declined on MicroShield (non-medical only). The profile verdict is enforced in code, after generation — the plan cap is never left to the model (serve/core/rate.py::_apply_plan_guard). - Hybrid RAG with citations (
rag/) — bge-m3 dense + BM25 → RRF → cross-encoder rerank over a jurisdiction-tagged knowledge base built from the synthetic manuals (docs/manual/) plus a clearly-labeled illustrative debit grid. Retrieval is pinned per impairment dimension so the clauses that fire are the clauses that get cited. - RAFT fine-tune (
ft/,data/sft/) — the SFT data puts the firing debit-grid rows (oracle) + distractors in the input and trains a chain-of-thought that derives the rating from the cited rows; 20% of rows withhold the oracle and train the model to flag the gap and refer instead of guessing. MLX LoRA on Llama-3.2-3B, tracked in MLflow with a champion/candidate registry and a divergence guard born from a real 2e-4 learning-rate blow-up at iteration 31 (ft/train_tracked.py; the full story is in docs/RESULTS.md). - Workbench (
serve/) — FastAPI JSON API + server-rendered UI: case intake, streaming analysis, before/after model toggle (base+RAG vs fine-tune+RAG), jurisdiction-tagged evidence panel, accept/edit/override feedback capture, audit log, PDF export. Every review emits an MLflow trace.
The fine-tune moved risk-class band accuracy (within ±1) from a best-base 0.442 → 0.958 on the 120-case gold set, refer-precision 0.53 → 0.94, decision-consistency 0.74 → 0.99. A four-variant SFT data bake-off (e0–e3) then probed phrasing robustness: the honest outcome was no new champion — the best variant cut the held-out author-style accuracy drop from −20.8 pts to −4.2 pts but missed the referral-safety bar. Numbers, tables and the learning-rate divergence post-mortem: docs/RESULTS.md.
Every committed fixture is a pure function of the code and manuals, and the test suite asserts it byte-for-byte:
make regen # kb chunks, gold sets, RAFT SFT data, cached UI cases (stdlib only)
make test # includes fixture-reproduction + manual↔engine consistency checksThe synthetic manuals are the RAG ground truth and the engine documentation —
tests/test_kb_integrity.py fails if they drift apart.
| Mode | Needs | Command |
|---|---|---|
cached (default) |
core deps only | make demo |
| RAG index | pip install -r requirements-rag.txt (~2GB HF models) |
make kb, then python rag/smoke_test.py |
mlx (in-process champion) |
Apple Silicon + requirements-ft.txt + a trained adapter in ft/adapters/ |
UW_UI_GENERATOR=mlx make serve |
live (Ollama) |
Ollama + make fuse (adapter → GGUF → acme-uw) |
make live |
| Train | requirements-ft.txt |
python ft/train_tracked.py (guarded; logs to MLflow) |
| Measured eval | RAG index + adapter | python eval/run_measured_eval.py, eval/fast_eval.py |
The eval harness also carries the robustness probes (held-out author styles, OOD paraphrases, retrieval-ablation matrix) reported in RESULTS.md.
scripts/ rules engine + deterministic data generators (SFT, gold, UI cases)
data/ plans (profiles + schema), generated fixtures, KB chunks
docs/ manual/ (synthetic governing manuals) · RESULTS.md (exported evidence)
rag/ build_kb, hybrid retrieve, pinned multi-query case retrieval, smoke test
ft/ train wrapper (divergence guard), MLflow registry, pyfunc wrapper
eval/ measured eval ladder, fast eval, robustness probes, MLflow helpers
serve/ FastAPI app + core seams (generator/retriever/renderer) + web UI
tests/ engine, plan-profile, fixture-reproduction, KB-integrity, API tests
This is a history-free public snapshot of a private customer POC. All customer-identifying content was replaced by a re-authored fictional universe (fresh manuals, plans and fixtures — regenerated, not redacted); applicant data was synthetic from day one. The project was built with Claude Code and OpenAI Codex as governed contributors under a written agent autonomy and cost-cap policy — spec-first, with the deterministic engine, fixture-reproduction tests and acceptance gate serving as the verification harness. MIT license.