Financial Planning RAG Assistant

A US personal-finance education assistant that answers only from official government publications, renders claim-level citations, abstains when the corpus does not support an answer, and refuses personalized advice before it costs an API call.

Live demo: https://financial-planning-assistant.streamlit.app

Held out from tuning: 0.891 status accuracy, 0.854 retrieval recall@5, 5/5 personalized-advice refusals, 0 over-refusals, 0 fabricated citations. Model-assisted answer grading over the same 55 held-out questions: 0.848 correctness, 0.894 faithfulness (caveats).

Every claim carries its own citation: publisher, publication, tax year, section path, and page number. The limitations are generated per answer, not boilerplate.

Not financial advice

This explains what official publications say. It cannot account for anyone's circumstances, and it declines questions that ask it to. A deterministic scope gate runs before retrieval and before any model call, so a request for advice cannot be reached by prompt injection — the refusal already happened.

Note the request counter: still 10 of 10 after the refusal. The gate ran before retrieval and before the model, so declining cost nothing.

Official sources

26 documents, 1,718 indexed passages:

• IRS — Publications 17, 505, 550, 590-A, 590-B, 969, 970, plus the retired 2023 edition of 590-A
• CFPB — credit reports, credit scores, debt collection and collector rights, budgeting, emergency funds
• SEC / Investor.gov — investing basics, asset allocation, fees, mutual funds, compounding, fraud types

Full inventory with hashes, versions, and parsing quality: docs/data_card.md.

Versioning and copyright

Every source is a work of the United States Government and is not subject to domestic copyright (17 U.S.C. §105). The evaluated index and the public index are identical — no published metric describes a corpus a reader cannot open, and tests/test_index_parity.py asserts it.

IRS publications are versioned by tax year, verified against the year printed on the downloaded title page rather than trusted from the manifest. Agency web pages are versioned by retrieval date, because they are continuously updated and carry no edition. The retired 2023 edition of Publication 590-A is indexed but excluded from normal answers; it exists so that conflicting-version handling is testable rather than merely asserted.

Architecture

Four of the six exits are refusals. That is the design: the system declines in more ways than it answers, and two of those refusals — the scope gate and citation validation — are deterministic code rather than model judgement. Source: docs/diagrams/architecture.mmd.

Hybrid retrieval. Dense search catches paraphrase, BM25 catches exact terms like 590-A and 401(k) — the tokenizer preserves intra-word hyphens for exactly this reason. Reciprocal rank fusion (k=60) combines them without needing comparable scores.

Gates. The scope gate is regex, not a model: something asked to police its own scope will sometimes say yes. The evidence gate requires the reranker logit to clear a tuned threshold, requires two distinct sources when a question needs synthesis, and returns source_conflict when one publication is retrieved under two editions.

That last branch is unreachable in the deployed app, and the diagram would be dishonest without saying so. Retrieval defaults to active_only=True, and the only publication indexed twice — Pub 590-A, 2023 and 2025 — has its 2023 edition marked inactive. The conflict path fires when the evaluation harness opens the filter for VS64 and VS67, which is how the branch is tested. A reader of the hosted app cannot reach it, and no metric here claims otherwise.

Generation. OpenAI Responses API with Structured Outputs via Pydantic text_format. citation_ids has min_length=1, so an uncited claim is unrepresentable at the schema level rather than discouraged in a prompt.

Citation validation. Every cited chunk ID must exist in the evidence actually supplied. A fabricated citation withholds the entire answer.

Results

Configuration was tuned on a 20-question development split and frozen before the 55 held-out questions were scored once. All safety categories — no-answer, version-sensitive, personalized-advice — are held out in full.

retrieval variant	recall@5	nDCG@5	precision@5	MRR@10
dense only	0.866	0.771	0.424	0.787
BM25 only	0.780	0.680	0.291	0.720
hybrid RRF	0.854	0.760	0.323	0.796
hybrid + rerank (deployed)	0.854	0.718	0.458	0.699

generation	held out (55)	dev (20)
status accuracy	0.891	1.000
abstention recall	0.900	—
version recall	0.800	—
personalized-advice recall	1.000	—
answerable questions that abstained	6	0

Dev figures are shown beside test figures rather than only the more favourable one. Dev retrieval recall@5 was 0.925 against 0.854 held out — that ~7 point gap is the honest cost of tuning on 20 questions.

Status accuracy measures whether the system answered or declined correctly, not whether the answers are right. A second model grades all 55 held-out answers claim by claim against the passages the generator saw — 0.848 correctness, 0.894 faithfulness, 0.076 unsupported claim rate, 1.000 safety. That is a secondary score: the grader shares a model family with the generator and shares some of its blind spots.

20 high-risk held-out answers were then reviewed by hand against their cited passages — 0.850 correctness, 0.925 faithfulness, 0.675 completeness, one answered question in ten carrying an unsupported claim, no safety failures (sheet).

Correctness and faithfulness track the model-assisted grade closely, which is mild evidence it is not wildly off. Completeness is where they part — 0.864 graded against 0.675 reviewed — and the review is the one to believe. It found two false abstentions that every automated metric here scores as correct behaviour, because status accuracy asks whether the system abstained and never whether it should have. Details in docs/evaluation_report.md.

Worth reading before trusting any LLM-graded number: the grading harness was wrong five separate times before it was right, and truncating passages to 900 characters alone moved the unsupported-claim rate from 0.076 to 0.403 with no change to the system. docs/evaluation_report.md records all five.

Full analysis: docs/evaluation_report.md.

Failure analysis

All six held-out failures are abstentions on answerable questions; the system never fabricated.

Two of them, VS62 and VS65, were described here as corpus gaps — questions asking for rationale the IRS never states. The human review disproved that. Pub 970 and Pub 505 do carry the evidence, and both questions retrieved their gold source at rank 1 and 2. They were refused by the evidence gate on a reranker logit — VS62 by 0.07 — before any model saw a passage. Blaming the corpus was the more flattering reading and it was wrong; only reading the retrieved passages caught it.

The remaining failures are retrieval misses on thin, fragmented agency pages.

Reranking buys precision and costs ranking quality — recall@5 ties at 0.854, precision@5 rises 0.323 → 0.458, MRR@10 falls 0.796 → 0.699. Only six passages reach the model, so distracting context is not free, and the reranker logit is what the evidence gate thresholds on. Recorded as a tradeoff rather than a uniform win.

The reranker threshold does not separate answerable from unanswerable questions. Measured logit ranges overlap (−7.63…+6.86 vs −9.87…−1.53); 90% is the ceiling from that signal alone. The architecture does not depend on it.

Three choices that look like mistakes and are not

Inference runs on ONNX, not PyTorch. Same model weights, different runtime. FastEmbed installs 180 MB against PyTorch's multi-gigabyte tree, and no torch appears anywhere in the resolved dependency graph — asserted in CI. The PyTorch stack does not reliably fit the hosted memory allocation, and it fails by not loading rather than by raising. Measured footprint: docs/deployment_memory.md.

Reranker thresholds are logits, not probabilities. The cross-encoder emits raw logits from roughly −11.5 to +11.4 (sbert_ce_default_activation_function is Identity). A threshold that reads like a confidence value is a category error. Zero is the trained decision boundary in the abstract and wrong on this corpus — it discards 37.5% of correct evidence, because short agency web pages score systematically below long IRS prose regardless of relevance. Tuned to −4.5 on the dev split. Measurements: docs/reranker_calibration.md.

max_output_tokens is 4,000 for an answer needing ~600. On a reasoning model the budget covers reasoning and visible output together. Sized for the answer alone, the model exhausts it while reasoning and returns status="incomplete" with no message item — every question abstains, and the report shows near-zero correctness beside perfect abstention recall while measuring nothing. The evaluation runner refuses to compute metrics if any response hits that state.

Privacy

Questions are sent to a language model to compose an answer from retrieved passages. They are not stored. No accounts, no analytics on question content, no raw query logging. Credentials live in Streamlit secrets or the environment.

Repository

src/financial_rag/   ingestion, parsing, chunking, indexing, retrieval,
                     answerability, prompts, generation, citations, service, evaluation
scripts/             build_corpus, build_index, run_evaluation, export_requirements
app/streamlit_app.py public demonstration
data/                source manifest, lock file, 75-question evaluation set
artifacts/index/     work index (private) and release index (deployed)
docs/                data card, evaluation report, safety, calibration, memory
docs/diagrams/       architecture.mmd, rendered to docs/images/architecture.png
tests/               139 tests

Reproduce

uv sync --dev
uv run python scripts/build_corpus.py --config configs/base.yaml
uv run python scripts/build_index.py --config configs/base.yaml
uv run python scripts/run_evaluation.py --mode tune --config configs/base.yaml
uv run python scripts/run_evaluation.py --mode retrieval --config configs/base.yaml
uv run streamlit run app/streamlit_app.py
uv run pytest

Generation evaluation calls the paid API and needs OPENAI_API_KEY:

uv run python scripts/run_evaluation.py --mode generation --config configs/base.yaml

Deployment

Streamlit Community Cloud reads neither pyproject.toml nor uv.lock, so requirements.txt is the deployment dependency file and CI installs from it in a bare venv. Regenerate with scripts/export_requirements.sh; tests/test_requirements.py fails on drift from the lock.

docker build -t financial-rag . && docker run -p 8501:8501 -e OPENAI_API_KEY=... financial-rag

Limitations

A 26-document corpus leaves many reasonable questions unsupported, and abstention in those cases reflects corpus coverage, not model capability. Citation validation proves a claim points at a real passage; it does not prove the claim faithfully summarizes it. Held-out recall@5 of 0.854 means roughly one answerable question in eight retrieves incomplete evidence. Deployment memory sits at 71% of the Community Cloud ceiling.

Full accounting: docs/safety_and_limitations.md.