A retrieval-augmented Q&A system over SEC 10-K risk factor disclosures, with grounded citations down to the specific filing and chunk, and an eval harness that measures whether it actually works.
Status: all three stages complete — ingestion + chunking, retrieval + generation (CLI and web UI), and an eval harness with 23 test questions.
Risk factor disclosures are long, dense, and change from filing to filing. An analyst asking "what cybersecurity risks does this company disclose" wants a specific answer traceable to a specific sentence in a specific filing, not a paraphrase that might be subtly wrong. For financial documents, an answer without a citation isn't useful; a wrong answer stated confidently is actively harmful. This project is built around that constraint: every answer must be traceable to the exact chunk it came from, and the system has to be willing to say "the filings don't say" rather than guess.
flowchart LR
A[SEC EDGAR] -->|fetch_filings.py| B[Raw 10-K HTML]
B -->|extract_risk_factors.py| C["Chunked risk factors<br/>chunks.jsonl"]
C -->|"embed_and_store.py<br/>BGE-small-en-v1.5"| D[("Chroma<br/>vector store")]
E["User question<br/>CLI / Streamlit"] -->|"retrieve.py<br/>query embedding + ticker filter"| D
D --> F["Top-k chunks<br/>+ metadata"]
F -->|generate.py| G["Claude Haiku 4.5<br/>answer only from context"]
G --> H["Cited answer<br/>+ GROUNDED yes/no"]
F -.eval harness.-> I["judge.py<br/>faithfulness + relevance"]
G -.eval harness.-> I
10-K Item 1A ("Risk Factors") filings for 8 companies across 4 sectors, pulled from SEC EDGAR's public API:
| Ticker | Company | Sector |
|---|---|---|
| AAPL | Apple Inc. | Technology |
| MSFT | Microsoft Corp | Technology |
| WMT | Walmart Inc. | Retail |
| TGT | Target Corp | Retail |
| JPM | JPMorgan Chase & Co | Financial Services |
| GS | Goldman Sachs Group Inc | Financial Services |
| JNJ | Johnson & Johnson | Healthcare |
| DAL | Delta Air Lines Inc | Airlines/Industrials |
For each company, ingestion/fetch_filings.py looks up the most recent 10-K via EDGAR's submissions API and downloads the primary filing document. Source metadata (CIK, accession number, filing date, source URL) is recorded per filing in data/raw/manifest.json so every chunk can be traced back to exactly which filing it came from.
10-K risk factor sections are already organized by the filer into discrete, self-contained units — typically a short bolded headline sentence ("The Company's reliance on third-party suppliers could disrupt operations...") followed by one or more paragraphs explaining that specific risk. Fixed-size or sliding-window chunking would cut across these units arbitrarily: splitting one risk's explanation across two chunks, or merging two unrelated risks into one. Both hurt retrieval precision, and both make citations weaker — a citation should point to "the chunk about supply chain risk," not "chunk 14 of 40, which happens to contain half of two different risks."
Chunking on the filer's own structure keeps each chunk topically coherent and gives every citation a clean boundary that means something.
10-K filings are produced by many different filing agents, so there's no single consistent HTML template. But one convention held across every filer inspected: the lead sentence of each risk factor is wrapped in bold, distinguishing it from the surrounding body text. Filers differ on how they mark it further (Apple also italicizes it, JPMorgan colors it, Microsoft doesn't distinguish it from section titles at all) — but bold is the consistent signal.
The extractor (ingestion/extract_risk_factors.py):
- Locates the real Item 1A → Item 1B span in the filing, filtering out the table of contents, in-body cross-references ("see Item 1A, 'Risk Factors'"), and — for filings that repeat a page header throughout the section — running header noise.
- Walks all bold spans in document order and classifies each one as a category header (a short or ALL-CAPS topical label like "Legal and Regulatory," not itself a chunk boundary) or a risk headline (a full sentence making a specific claim — a chunk boundary).
- Builds one chunk per detected headline: the headline plus everything up to the next boundary, tagged with whichever category header preceded it.
- A run only counts as a real heading/category candidate if it makes up most of its own line's text — this filters out filers (JNJ) that bold recurring defined terms like segment names inline within ordinary paragraphs, which would otherwise create false boundaries mid-sentence.
Fallback 1 — no reliable bold-headline structure detected. If a filing's ratio of section words to detected headlines is too high (suggesting the structural signal doesn't hold for this filer), the extractor falls back to grouping consecutive paragraphs up to a ~300-word target, always breaking between paragraphs rather than mid-paragraph. Microsoft's filing hits this path — its bold sub-headings don't reliably distinguish topical labels from actual per-risk boundaries, so paragraph-grouping is the more honest choice than forcing a bad structural fit.
Fallback 2 — a single risk factor chunk is too long. Some risk factors run 500+ words. If a detected chunk exceeds ~400 words, it's split into sub-chunks at paragraph boundaries, with the original headline re-prepended to every part after the first (the first part already contains it) so no sub-chunk loses its topic context.
| Ticker | Method | Chunks | Avg words | Range |
|---|---|---|---|---|
| AAPL | structured (bold headline) | 42 | 239 | 51–563 |
| MSFT | fallback (paragraph grouping) | 40 | 250 | 33–408 |
| WMT | structured (bold headline) | 55 | 265 | 26–481 |
| TGT | structured (bold headline) | 35 | 234 | 35–412 |
| JPM | structured (bold headline) | 62 | 238 | 110–360 |
| GS | structured (bold headline) | 84 | 251 | 16–424 |
| JNJ | structured (bold headline) | 27 | 236 | 102–364 |
| DAL | structured (bold headline) | 32 | 251 | 119–361 |
377 total chunks across 8 filings, 7 of which used the structural detector and 1 (Microsoft) correctly fell back.
Manually spot-checking chunk quality surfaced two remaining edge cases, both understood but not fully fixed (~0.5% of the corpus, 2 of 377 chunks):
- GS_..._rf033 is a stub chunk (just the headline, no body). Root cause: Goldman's filing includes a plain-weight "risk summary" bullet list that restates the same headline sentence verbatim before the real bold heading appears later. The offset-resolution logic (searching for each bold run's text within the flattened section text) occasionally resolves to the earlier, non-bold duplicate instead of the real one.
- WMT_..._rf046 has an orphaned lowercase fragment appended to an unrelated headline's chunk. Root cause: Walmart's HTML splits one sentence across a
<hr style="page-break-after:always">page-break element, and the boundary detection for a nearby headline landed inside that split.
Both stem from the same underlying design tradeoff: bold-run offsets are resolved by searching for each run's text within the flattened section text, rather than tracking character position exactly during a single DOM traversal. That search can occasionally resolve to the wrong occurrence when identical or near-identical text appears twice (a summary bullet + the real heading) or when a page break splits a sentence into two blocks. A fully robust fix would track bold-run character offsets directly during the same tree walk that builds the flattened text, rather than reconciling the two after the fact. Given this affects 2 chunks out of 377, I judged it not worth the added complexity for this stage, but it's a real limitation worth being able to explain rather than paper over.
mnemosyne/
├── data/
│ ├── raw/ # raw filing HTML + manifest.json (source metadata)
│ ├── processed/ # chunks.jsonl, chunking_stats.json
│ └── chroma/ # persistent local vector store
├── ingestion/
│ ├── companies.py # company/CIK universe
│ ├── fetch_filings.py # Stage 1a: EDGAR download
│ ├── extract_risk_factors.py # Stage 1b: chunking
│ └── embed_and_store.py # Stage 1c: embed + load into Chroma
├── retrieval/
│ ├── retrieve.py # Stage 2a: query embedding + Chroma search
│ ├── generate.py # Stage 2b: grounded generation + citation resolution
│ ├── ask.py # Stage 2c: CLI
│ └── app.py # Stage 2d: Streamlit web UI
├── eval/
│ ├── test_set.py # Stage 3a: 23 hand-grounded + adversarial questions
│ ├── judge.py # Stage 3b: LLM-as-judge (faithfulness + relevance)
│ ├── run_eval.py # Stage 3c: harness runner
│ └── results.jsonl # Stage 3 output: full per-question results
└── requirements.txt
cd mnemosyne
python -m venv venv
./venv/Scripts/pip install -r requirements.txt # (venv/bin/pip on macOS/Linux)
python ingestion/fetch_filings.py # downloads 8 filings from SEC EDGAR
python ingestion/extract_risk_factors.py # chunks Item 1A into data/processed/chunks.jsonl
python ingestion/embed_and_store.py # embeds chunks, loads into local Chroma DBEmbeddings use BAAI/bge-small-en-v1.5 (open, CPU-only, ~130MB) — chosen over all-MiniLM-L6-v2 for stronger MTEB retrieval scores at a similar size. Vector storage is a local persistent Chroma collection (data/chroma/) — no server, no hosting cost.
retrieval/retrieve.py embeds the query with the same BAAI/bge-small-en-v1.5 model used in Stage 1, adding BGE's required query-side instruction prefix ("Represent this sentence for searching relevant passages: ") — passages and queries are embedded asymmetrically in BGE's training, so skipping this measurably hurts retrieval quality.
A lightweight alias lookup (detect_tickers) scans the question for a company name or ticker and auto-scopes retrieval to it — e.g. "What cybersecurity risks does Apple disclose?" automatically filters to AAPL. This isn't NLP, just a dict of tickers and common aliases (JPMorgan/JP Morgan/Chase → JPM, etc.), but it reliably prevents the failure mode where a company-specific question pulls in a different company's chunks that happen to score marginally closer.
For comparison questions naming multiple companies ("compare supply chain risk across these three companies"), retrieval runs once per company and merges the results, rather than one pooled top-k query across all of them. A pooled query has no guarantee of balance — if one company's risk factors happen to embed slightly closer to the query, it can crowd out the others entirely, which defeats the point of a comparison. Per-company retrieval guarantees every named company gets representation.
retrieval/generate.py builds a numbered context block from the retrieved chunks and asks the model (Claude Haiku 4.5 — cost-effective and sufficient for grounded extractive Q&A, which doesn't need frontier-level reasoning) to answer using only that context, citing claims with bracketed reference numbers like [2].
The model never has to reproduce a chunk ID, ticker, or filing date in its answer — citations are resolved from the reference numbers back to the actual chunk metadata in code, not generated as text. This means a citation can't be subtly wrong (a mis-typed date, a hallucinated source): the model's only job is picking which excerpt supports a claim.
For honesty, the model is required to end every answer with an explicit GROUNDED: yes or GROUNDED: no line — a structured judgment kept separate from the prose, rather than trying to infer intent from hedge words in the answer (fragile, and hard to grade automatically in Stage 3). "no" means the retrieved excerpts didn't actually contain what was needed, regardless of how the answer reads.
Verified manually against all three example use cases from the brief:
- Single-company question ("What cybersecurity risks does Apple disclose?") — auto-detected AAPL, retrieved relevant chunks, answered with correct per-claim citations.
- Multi-company comparison (
--tickers AAPL,WMT,JPM "Compare supply chain risk language...") — balanced retrieval across all three, answer organized by company with citations traceable to each one's own filing. - Out-of-scope question ("What was Apple's total revenue last fiscal year?") — correctly refused, explained that revenue isn't in the Risk Factors section, cited nothing, and flagged itself as not grounded.
A suspected failure case that turned out not to be one: during manual testing, a broad cross-company query about AI regulation risk cited a JNJ chunk that looked, at a glance, like it was about general R&D/innovation risk rather than AI regulation specifically. It was flagged here as a likely mis-citation and carried into Stage 3's eval set to investigate properly. Tracing it back to the actual chunk text showed the citation was correct all along — that chunk's headline is about general innovation, but its body text separately and explicitly discusses AI risk, including the exact sentence the model cited. See Stage 3 below for the full account. Keeping this in the README uncorrected would have been the more comfortable thing to do; the honest version is that a surface-level judgment call by the person building the system was wrong, and only checking the primary source text caught it.
CLI:
python retrieval/ask.py "What cybersecurity risks does Apple disclose?"
python retrieval/ask.py --tickers AAPL,WMT,JPM "Compare supply chain risk language across these companies"
python retrieval/ask.py --show-chunks "What supply chain risks does Walmart face?"
python retrieval/ask.py # no query -> interactive promptWeb UI (Streamlit):
streamlit run retrieval/app.pySame retrieve() / answer_question() pipeline as the CLI — the UI is a thin wrapper, not a separate implementation. Sidebar lets you restrict to specific companies, adjust how many chunks are retrieved, and inspect the raw retrieved chunks. Verified end-to-end in-browser: question in, auto-detected company filter, retrieved chunks, cited answer, working links back to the real SEC filing.
Requires an ANTHROPIC_API_KEY in mnemosyne/.env (see .env.example).
Pulled directly from eval/results.jsonl — real output, not hand-edited.
Q: What cybersecurity risks does Microsoft disclose regarding cyberattacks?
Cyberattacks can be conducted by "individual and groups of hackers and sophisticated organizations, including nation-states, state-sponsored organizations..." [1] and can lead to reduced revenue, increased costs, and liability claims [2]... (truncated — full answer in results.jsonl, id q03)
GROUNDED: yes — Sources: [1]–[4] MSFT 10-K filed 2025-07-30 — source
Q: Compare cybersecurity risk disclosures between Apple and Microsoft.
Apple discloses that it "experiences malicious attacks and other attempts to gain unauthorized access to its systems on a regular basis" targeting "the confidentiality, integrity or availability of confidential information" [1]... Microsoft frames the same risk more broadly under cyberattacks and security vulnerabilities that could reduce revenue or harm its competitive position [5], [8]... (id q17)
GROUNDED: yes — Sources: [1] AAPL 10-K filed 2025-10-31; [5], [8] MSFT 10-K filed 2025-07-30
Q: What was Apple's total revenue for the last fiscal year?
"I cannot answer this question based on the provided excerpts. The excerpts provided are all from Apple's Risk Factors section of the 10-K filing and discuss various risks the company faces... None of these excerpts contain Apple's actual total revenue..." (id q19)
GROUNDED: no — Sources: none cited. Correct behavior — revenue isn't disclosed in Item 1A.
Q: What risk does this company's retail media advertising network face? (no company named)
"Target's Roundel retail media advertising network faces several risks [1]: 1. Loss of advertiser commitments — advertisers do not have long-term commitments with Roundel... 2. Dependence on vendor and seller base..." (id q22)
GROUNDED: yes — Sources: [1] TGT 10-K filed 2026-03-11 — source
Notably, no company was named in the question at all — retrieval found this from semantic similarity alone.
eval/test_set.py has 23 questions, not the 15-20 originally planned — 20 hand-grounded questions plus 3 adversarial additions explained below.
Questions 1-16 are single-company, each grounded in one specific chunk I read and verified before writing the question (expected_chunk_ids names the exact chunk, not a guess). Questions 17-18 are multi-company comparisons. Questions 19-20 are deliberately unanswerable from Item 1A (revenue figures, headcount) — the correct behavior is an honest refusal, not a retrieval hit, so they're excluded from the hit-rate denominator rather than counted as misses.
A limitation worth being upfront about: questions 1-20 were derived from actual chunk headlines, which makes retrieval and faithfulness easy by construction — of course the system can find a chunk when the question is phrased close to that chunk's own topic sentence. A real user doesn't ask that way. Questions 21-23 exist specifically to push past that:
- q21 — "What risks do companies face from AI regulation?" with no ticker filter at all. This is the exact query that produced the suspected JNJ mis-citation during Stage 2 manual testing (see above). Re-running it here to actually check the claim, rather than leaving a hedge in the README, is the point of including it.
- q22 — "What risk does this company's retail media advertising network face?" — no company named anywhere, testing whether pure semantic similarity (no ticker-detection crutch) can still find Target's Roundel disclosure specifically.
- q23 — "What percentage of Apple's revenue comes from Greater China?" — a specific-sounding figure that Item 1A doesn't actually contain (it's in the financial statements/segment footnotes), testing whether a plausible-sounding question tempts the system into fabricating a number.
- Retrieval hit rate: does the top-k retrieved set contain the chunk that actually answers the question? Only computed for the 19 questions that have a real
expected_chunk_ids— the 2 unanswerable questions and the 2 open-ended adversarial probes (q21, q23) have no single ground-truth chunk to check against, so they're excluded rather than auto-scored as misses (any()over an empty list is vacuously false, which would silently miscount them — this was a real bug in the first version of the harness, see below). - Faithfulness: does the answer state only what the retrieved excerpts support? Scored by an LLM judge (Haiku 4.5), spot-checked manually (see below).
- Answer relevance: does the answer actually address the question asked? Same judge call, same model, a separate field in the same response to keep eval cost down (one call scores both dimensions instead of two).
| Metric | Score |
|---|---|
| Retrieval hit rate | 19/19 (100%) — 4 questions excluded (2 unanswerable, 2 open-ended adversarial) |
| Faithfulness | 23/23 (100%) |
| Answer relevance | 23/23 (100%) |
Full per-question results: eval/results.jsonl. Run with python eval/run_eval.py (23 questions × 2 API calls each — generation + judge — both Haiku 4.5, well under $0.10 total).
Read this table skeptically, not as proof the system is flawless. A clean sweep on a test set built from the corpus's own headlines is the expected outcome, not an impressive one — retrieval and faithfulness are close to a solved problem when the question already echoes the source's own phrasing. The adversarial questions (q21-23) are the more interesting result precisely because they don't have a guaranteed answer, and here's what actually happened:
- q22 (no company named) still hit its target — retrieval found Target's Roundel chunk from semantic similarity alone, with no ticker-detection to lean on. This is real evidence the embeddings are doing their job, not just that the ticker-alias heuristic is doing all the work.
- q23 (Apple China revenue %) correctly refused — the system cited the one loosely-related sentence it had ("sales outside the U.S. representing a majority of total net sales") and explicitly said this doesn't answer a percentage-by-region question, rather than inventing a number that sounds plausible for a company this size.
- q21 (AI regulation, no ticker) resolved the earlier suspected mis-citation — this run retrieved a different JNJ chunk (
JNJ_..._rf014) than the one from the original Stage 2 test, likely due to Chroma's HNSW index being an approximate-nearest-neighbor search (a documented property: ANN retrieval isn't always bit-for-bit reproducible near decision boundaries, even with the same query embedding twice). Readingrf014's actual text directly: its headline is about general "innovation, development and implementation of new products," but its body separately states "the application of AI in our business is emerging and evolving alongside new laws and regulations that may entail significant costs or ultimately limit our ability to continue the use of these technologies" — the exact phrase the model's answer cited. The citation was correct. The original suspicion, written into this README in good faith after eyeballing a category label, was wrong. - The same q21 run also surfaced a real, previously-documented Stage 1 defect in action: one of the six retrieved chunks was
GS_..._rf033, the 16-word headline-only stub chunk flagged as a known chunking limitation in Stage 1 (a text-search offset ambiguity — see above). It's topically on-topic ("AI risks... adversely impact our business"), so it's not wrong that retrieval surfaced it. What's worth noting is what generation did with it: the answer never cited it. Every GS-attributed claim in the answer traces to the other GS chunk retrieved that run (GS_..._rf034, a properly-bounded 424-word chunk covering the same topic), and the empty stub was silently skipped rather than causing a hallucinated or garbled citation. That's the system being incidentally robust to a known upstream data-quality issue, not evidence the issue doesn't matter — Goldman's actual disclosure at that exact spot in the filing is still functionally invisible to anyone who only retrieves that stub, which is a real gap, just not one this particular query exposed.
The brief asks not to trust the LLM-as-judge blindly, so beyond the automated pass/fail, I manually re-read the full answer, retrieved context, and judge reasoning for q01-q04, q17-q23 (12 of 23) — every comparison and adversarial case, plus a sample of the single-company ones. In all 12, the judge's stated reason named specific claims and specific excerpts rather than generic language ("the answer directly quotes excerpt [1] and doesn't add unsupported claims" vs. a template like "the answer looks correct"), and every claim I independently traced back to source text checked out. I did not spot-check all 23 — this is a portfolio-scale eval, and 12 manually-verified cases is enough to trust the pattern without claiming exhaustive verification.
The first version of run_eval.py computed retrieval hit-rate as any(cid in retrieved for cid in expected_chunk_ids) for every question not labeled "unanswerable" — but q21 and q23 (labeled "adversarial-broad" / "adversarial-specific") also have an empty expected_chunk_ids list, since they're open-ended probes with no single correct target. any() over an empty list is always False in Python, so both were silently counted as retrieval misses rather than "not applicable," understating the hit rate as 18/21 (86%) instead of the correct 19/19 (100%). Fixed by excluding any question with an empty expected_chunk_ids, regardless of its category label, rather than gating on the category string. Caught by actually reading the per-question output instead of trusting the summary line — the same principle as the judge spot-check above, applied to my own code.
Everything in the original brief is built. Ideas for further work, roughly in order of value:
- Fix the Stage 1 chunking edge cases properly instead of documenting around them — the GS stub and WMT orphaned-fragment issues both trace back to resolving bold-run offsets via text search rather than tracking character position during the DOM traversal. A single-pass extractor that computes offsets directly would eliminate this class of bug rather than mitigating symptoms.
- A larger, blinder eval set — have someone other than the person who built the chunker write the test questions, so retrieval difficulty isn't implicitly calibrated to what the system's own chunking produces.
- Multi-year comparisons — the corpus only has each company's most recent 10-K; pulling 2-3 years per company would let a question like "how has Apple's China risk language changed since 2023" actually be answerable, and would stress-test citation dating (same company, multiple filing dates, must not conflate them).