ISC Document Intelligence & Knowledge Platform

An ACL-aware document extraction and retrieval pipeline for industrial supply-chain documents, running end to end over one document type (purchase orders) against a 20-document synthetic corpus. It enforces permissions at index and retrieval time rather than filtering results after the fact, and every extracted value carries a decomposed confidence signal and a source span rather than a bare score. This is the P1 milestone: one document type proven all the way through, not broad coverage — see "What it does not do" below for the honest edge of that claim.

What the evaluation found

The strongest result here is not that extraction scores well on this corpus — it does, but the corpus was generated by this system's own tooling to be readable, so that number describes a solved problem, not a general capability. The stronger claim is that the eval harness — not unit tests, not code review — found six distinct, measured defects during P1:

• rollup() reported 1.0 confidence on records containing real errors. It only walked top-level fields; a bad line-item price never reached it because line items live in a nested list fields() doesn't see. Fixed by adding all_fields().
• An ambiguous-date signal raised confidence instead of lowering it. _apply_check()'s two-path design treated any score at or above 0.5 as corroborating — including the 0.6 an ambiguous-date check returns to say this is genuinely unclear. 56 of 58 real P1-03 extraction errors sailed through at ~0.999 confidence this way. (ADR 0006)
• auto_accept_error_rate structurally excluded silently dropped fields. A field the model omits is wrapped ExtractedField.missing(), which is Confidence.certain() (1.0) by construction — a metric counting only wrong outcomes never saw a silent omission at all.
• Flattening a reconstructed table for tidiness destroyed the column alignment the model was reading, and every structural check still passed. Row count, ordinal sequence and cell presence all validated clean while po_018.pdf's extended_price field went from 8/8 correct to 8/8 missed. Only a live re-run against the model caught it. (ADR 0007)
• The index was purely additive, which would have inflated recall@k. Re-chunking a document added new vectors without removing the previous run's — orphaned chunks that any recall@k measurement would have silently counted. Fixed to purge orphans on every re-index.
• A citation that resolves is not a citation that supports. bind_citations() checks that a [n] marker points at a real, permitted chunk — not that the sentence beside it agrees with what's in it. Found live, not hypothesised: a P1-07 sample question cited a real, permitted table row while misstating the price in it. (ADR 0009)

Each one is measured, not asserted — see the ADR or docs/LIMITATIONS.md cited next to it for the numbers.

Quick start

conda env create -f environment.yml
conda activate Sai2608
cp .env.example .env          # add OPENAI_API_KEY
make slice                    # corpus -> ingest -> parse -> extract -> index -> eval

The env name is case-sensitive — conda activate sai2608 will fail. make slice costs real money (embedding + chat calls); budget a few cents on a clean cache, see Reproducibility below for the measured figure. make test (503 tests) runs with no network required.

What it does

• ingest — source PDFs → blob store + document stubs, deduplicated by content hash.
• parse — blobs → Document with blocks, line-item tables reconstructed structurally (not just text-flowed), parser provenance attached.
• extract — LLM structured extraction into PurchaseOrderRaw; every field wrapped with a decomposed Confidence and a source Span.
• index — chunk + embed into a local vector store (dense + BM25); every chunk carries its ACL terms, and re-indexing purges what it replaces.
• retrieve — dense and lexical search fused with RRF, filtered to the asking principal's permissions before ranking, on both paths.
• answer — draft a response, bind [n] citations to retrieved chunks, verify the cited chunk actually supports the sentence next to it, abstain rather than guess when it doesn't.
• eval — two harnesses against the live pipeline's own output, not fixtures: extraction (compare to gold, confidence calibration) and retrieval (recall/MRR, answer accuracy, abstention correctness, ACL-leak detection).

The numbers

Extraction — 20-document corpus (P1-03/P1-04 run)

• Auto-accept error rate: 0.0%.
• One real error across the corpus: lines.extended_price, 1 of 210 values (a 10x digit slip), caught at confidence 0.02 and correctly routed to review rather than auto-accepted.
• Review band: 0 wrong of 93.

This corpus was generated by this system's own tooling to be extractable — treat this as "this pipeline solves extraction on documents shaped like the ones it was built against," not extraction in general. Full caveats in docs/LIMITATIONS.md.

Retrieval — P1-09 full run, 56 gold questions (35 answerable), live index

Sample size, read before any figure below: 35 answerable questions over 20 documents. At this size, one flipped outcome moves any per-subtype figure by several percentage points — these numbers say where the system is weak, not a tight estimate of how weak. Unlike extraction, retrieval has been measured exactly once; this is a first honest reading, not a converged benchmark.

• recall@8: 0.971
• answer accuracy: 26/35 (74.3%)
• MRR: ~0.83 — reproduced at 0.833 in this tree and 0.830 from an independent clean-clone run; see Reproducibility below for why it's not a fixed figure.
• abstention precision: 0.862 (recall@8 and answer accuracy above are the headline numbers; this one was itself a measurement bug this milestone — the original definition scored every correct restricted/no-reader denial as imprecision and read 0.241 on the same outcomes)
• abstention recall (reason-aware): 0.750
• ACL leaks: 0 — the one metric this project treats as a hard gate, not a scored dimension alongside the rest.

The invariants

Enforced by tests and validators, not conventions:

• No module outside isc.llm imports openai. Enforced by AST inspection (tests/unit/test_import_hygiene.py) — swapping providers is a config change, not a refactor. (ADR 0002)
• A chunk cannot exist without ACL terms. AclSet.allow_terms is validated non-empty at construction — an unlabelled document is nobody's, not everybody's. (ADR 0004)
• Filtering is pre-ranking, on both the dense and lexical retrieval paths. Never post-filtered — post-filtering leaks through result counts and score distributions even when it drops the right documents. (ADR 0004)
• One ACL leak fails the run, regardless of every other metric. The retrieval eval harness's passed() gates on acl_leaks == 0 alone; make slice's eval stage exits non-zero if it happens.

What it does not do

No OCR (native-text only), no reranker (LOW_SUPPORT is disabled — measured, not guessed, see ADR 0008), no query rewriting, and the corpus is 20 synthetic documents, not collected ones. Full detail — including what the retrieval numbers above do and do not support, question by question — is in docs/LIMITATIONS.md; not duplicated here.

Reproducibility

Verified from a clean git clone, a fresh conda env, and no .cache/: make slice ran end to end for $0.0314 in real API cost. All 20 document ids matched this tree's exactly — content-addressing held across an independently built copy. recall@8 and answer accuracy reproduced exactly (0.971; 26/35); MRR landed at 0.830 against this tree's 0.833 — rank-order metrics among near-tied chunks vary slightly run to run because embedding calls are not bit-deterministic, confined here to line_item (split-table) chunks whose rows are textually close together. Quote MRR as ~0.83, not 0.833.

ADR index

Full reasoning lives in docs/adr/; one line each here.

• 0001 — Every non-obvious structural choice gets a written ADR, not a whiteboard reconstruction after the fact.
• 0002 — All model access goes through a provider port; only the OpenAI/Azure client modules may import openai, enforced by AST inspection.
• 0003 — Confidence is a propagated signal, not a float: it carries the Factors that produced it, not just a score.
• 0004 — ACL correctness is an index invariant, not a retrieval feature: permissions are enforced pre-ranking, never post-filtered.
• 0005 — Span location by string search, master-data matching without fuzzy fallback: provenance and dedup without bounding boxes.
• 0006 — Checks report an outcome, not just a score; dates use site context — fixes the ambiguous-date confidence bug above.
• 0007 — Chunk settings, settled before P1-08's gold exists — chunking frozen before gold chunk ids depend on it; documents the table-flattening regression above.
• 0008 — A rank-fused score cannot gate support, and raw similarity doesn't separate answerable from unanswerable questions on this corpus either — why LOW_SUPPORT is disabled.
• 0009 — Binding resolves a citation; it does not verify what it supports — citation binding vs. attribution verification, found live.

Layout

config/prompts/     versioned prompts, loaded by path
src/isc/common/     confidence, tracing, config, ids  ← becomes isc-core
src/isc/llm/        provider port + OpenAI/Azure impls ← becomes isc-core
src/isc/models/     domain models, storage-agnostic    ← becomes isc-core
src/isc/storage/    ports + local implementations
src/isc/<stage>/    ingest parse extract index retrieve answer eval
docs/adr/           why, not what
tests/adversarial/  the ACL suite

See docs/architecture.md for the stage pipeline diagram.