A pipeline + Q&A chatbot that ingests a repository of supplier contracts, understands each one, links related contracts into families, tracks true expiries, proactively notifies the right people, and answers questions in natural language with citations.
The one architectural idea: the LLM reads and writes prose; code does all the reasoning. The LLM is used in exactly two roles — extraction (read a messy contract → structured fields with confidence + evidence) and chatbot (choose which tested tool to call, then phrase the cited answer). Every fact — date math, amendment-moves-the-date, renewal-supersedes, MSA roll-forward, family graph, spend, routing, supplier matching — is deterministic, tested code. No probabilistic step is ever allowed to compute a fact.
Full design: ARCHITECTURE.md (Part 1 of the submission). Decision log: DECISIONS.md. Build story: BUILD_LOG.md. Assumptions: ASSUMPTIONS.md. Interactive architecture: architecture.html.
Walkthrough. As the brief allows ("a live walkthrough during the interview is fine too"), I'm happy to walk through the running system and the key architectural decisions live — the keyless run instructions below make that reproducible on any machine with no API key.
The LLM's extraction outputs and the chatbot's turns are recorded in cache/ and committed.
With LLM_MODE=cached, every LLM call replays from that cache — the whole system runs with
no API key. (Everything except extract and chatbot is deterministic code with no LLM at all.)
Tested on Python 3.12; targets 3.10+ (modern typing throughout — X | Y unions via
from __future__ import annotations, list[...]/dict[...] generics, stdlib only beyond the
three pinned deps).
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
export LLM_MODE=cached # keyless: replay the committed cache
# run the pipeline in order (ingest → extract → classify → link → resolve → evaluate+notify)
python -m src.ingest # 9 files → 9 documents (idempotent, content-hashed)
python -m src.extract # LLM → fields+confidence+evidence (cached replay, no key)
python -m src.classify # document_type + category + OpCo
python -m src.link # families from content refs
python -m src.resolve # R1–R8 → the curated truth
python -m src.notify # expiry buckets + anchor-aware notifications → outbox
python -m src.validate # the 5 invariant checks (exits non-zero on any failure)
python -m src.chatbot --demo # the 6 sample questions + "what can you do?" + a decline
# regenerate the grounded diagram from a fresh scan
python -m src.indexer && python -m src.viz # → architecture.htmlTo run against the live API instead (your own key): put ANTHROPIC_API_KEY=sk-… in a .env
(cp .env.example .env) and set LLM_MODE=live. Live runs also record to cache/, so a later
keyless replay works. Default model: claude-opus-4-8.
Ask one question: python -m src.chatbot "Which contracts expire in the next 60 days?"
Interactive: python -m src.chatbot (keeps short conversation history for follow-ups).
Change the as-of date: any stage takes --run-date YYYY-MM-DD (or RUN_DATE env; default
2026-07-06).
Correct a field (human-in-the-loop override, R8):
python -m src.override <contract_number> <field> <value> --reason "why this correction"
# e.g. python -m src.override CS-ORD-2041 effective_value 262500 --reason "amended by AMD-…-01"The override lane is how a reviewer corrects the curated truth. The CLI validates the field name
against the resolve whitelist (a bad field is refused loudly, listing the valid fields, and
writes nothing), checks the value coerces, resolves the contract number to its document, writes the
curation_overrides row, and tells you to re-run resolve to apply it. Overrides always win
(R8) and re-apply on every resolve, so a correction sticks. Symmetrically, resolve itself now
fails loud (non-zero) if it ever meets an override row targeting a field it cannot apply —
a silent skip would let a correction quietly vanish.
Run bash scripts/cold_check.sh before every push. It is the gate that proves the repo runs
for a grader on a clean machine, keyless:
bash scripts/cold_check.shIt clones the repo fresh from the local .git into a temp dir, builds a clean venv from the
pinned requirements.txt, runs the full pipeline (ingest → extract → classify → link → resolve
→ notify) + validate + chatbot --demo under LLM_MODE=cached (no API key), and exits
non-zero if any stage fails or any demo answer is not a grounded [ok] answer (a cache that
doesn't replay, or a broken tool, shows up as a [declined]/[hard_refuse] demo answer — exactly
what this catches). Because it clones committed state, it fails if the repo only ran thanks to an
uncommitted file, a local .env, a stale *.db, or an untracked cache entry — so a green
cold_check means the push is genuinely reproducible. The temp clone is removed on exit.
contract_intelligence/
├── src/ # the pipeline + surfaces (each stage is `python -m src.<name>`)
│ ├── ingest.py # 1 register raw text + content-hash + provenance
│ ├── extract.py ← LLM # 2 read prose → fields + confidence + evidence
│ ├── classify.py # 3 derive document_type / category / OpCo
│ ├── link.py # 4 families from content refs
│ ├── resolve.py # 5 R1–R8 cross-document rules → curated truth
│ ├── evaluate.py # 6 expiry buckets + anchor-aware due logic
│ ├── notify.py # 7 route + dedup → outbox
│ ├── validate.py # 5 invariant checks (the review gate)
│ ├── override.py # human-in-the-loop override CLI (R8; validated field writes)
│ ├── chatbot.py ← LLM # tool-calling, cited, self-aware Q&A
│ ├── tools.py # the deterministic fact tools the chatbot calls
│ ├── llm_client.py # live/cached transport (the only file that calls the API)
│ ├── telemetry.py # light per-run LLM cost/usage + stage breakdown
│ ├── grounding_audit.py # evidence-span audit over extractions
│ ├── indexer.py + viz.py # scan code+DB → architecture.json → architecture.html
│ ├── db.py # thin SQLite seam (swap point for Postgres)
│ └── prompts/ # named prompt templates (no inline prompt strings)
├── scripts/cold_check.sh # the pre-push cold-clone gate (fresh clone → pipeline → demo)
├── tests/ # 18 test files (pytest); run keyless with LLM_MODE=cached
├── db/schema.sql # the data model (raw / curated / reference layers)
├── reference/ # field dictionary + routing + supplier master (controlled vocab)
├── contracts/ # the 9 sample contracts (.txt — the source corpus)
├── cache/ # committed LLM recordings → keyless replay
├── docs/ # index/ (grounded architecture.json), audits/, telemetry/ (git-ignored artifacts)
├── outbox/ # generated notifications (git-ignored)
├── ARCHITECTURE.md · DECISIONS.md · ASSUMPTIONS.md · BUILD_LOG.md
├── requirements.txt · .env.example
└── architecture.html # the grounded, self-drawing diagram
The six sample questions (the acceptance suite — ARCHITECTURE.md §11)
Ask them via python -m src.chatbot --demo, or individually. Each answer is grounded in tool
results and cited [S1..Sn].
- Which contracts expire in the next 60 days? — distinguishes expiring (Meridian, 40d) from
action-needed (Verolt: expiry is 86d out, but its non-renewal notice deadline is only 56d
out — inside the window). This nuance is computed by the anchor-aware
find_action_items, not hoped for. - What is the notice period for the Cloudspire subscription, and the deadline? — 60 days; deadline 2027-07-02 (effective expiry 2027-08-31 − 60d).
- Everything related to the DataForge master? — the family:
DF-MSA-2023-07→DF-SO-1180(superseded) →DF-SO-1180-R1(renewal, current), with the correct edges. - Total annual committed spend, and how much in GBP? — recurring-annual: USD 322,500 (Cloudspire 262,500 amended + Verolt 60,000) + GBP 104,000 (DataForge renewal); Meridian 95,000 USD one-time reported separately; superseded/expired excluded. GBP portion = 104,000.
- Which auto-renew, and which have already renewed? — already renewed: DataForge master (R4 roll-forward) + DataForge renewal (R3). Auto-renews not yet: Cloudspire order, Verolt. Does not renew: Meridian, SOW.
- Which master governs SOW-CS-003? —
MSA-CS-2024-0012(Cloudspire).
Plus "what can you do?" → the capability manifest, and a deliberately-unanswerable question → an informative decline (why + what it can do).
Assumptions (per brief §9 — the live list is ASSUMPTIONS.md)
- FX: a fixed assumed rate (1 GBP = 1.27 USD) as of the run date, for single-currency totals only; native currencies reported alongside. No live FX.
- Run date: default
2026-07-06, configurable via--run-date/RUN_DATE. - Spend: "annual committed spend" = recurring-annual contract values; one-time engagements (SOWs, advisory) are reported separately, never summed into the annual figure.
- "Already renewed": a renewal whose term has begun (or a master rolled forward by R4) is the current governing contract; predecessors are superseded and excluded from spend.
- Notice basis: calendar days unless a contract states business days.
- Source system: the
contracts/folder is the source; a real CMS API swaps the ingest reader only.
This is a take-home; the how is as much the deliverable as the what.
- Staged build with per-stage tests. Nine stages (ingest → extract → classify → link →
resolve → evaluate → notify → chatbot → validate), each committed separately with its own test
file. 262 tests, all green, offline (a stub LLM stands in — the real API is never hit in
tests). The correctness core (
resolve, R1–R8) is tested against the exact §11 numbers. - The invariant suite gates every run.
python -m src.validateruns 5 source-derived checks that fail loud and double as the human-review surface: orphan edges, single-truth (one current raw value per field), expiry completeness, the confidence floor / fabrication guard, and idempotency (a re-run creates zero rows). Each is bite-tested against a deliberately-broken DB. - Raw vs curated. The
documents/extractionslayer is append-only — the audit trail of exactly what was ingested and what the model returned. Thecontractslayer is the single trusted truth, produced by the resolve rules. The chatbot and notifications only ever read the curated layer, never the raw guesses. - Confidence + evidence + provenance. Every extracted field carries a confidence, a verbatim evidence quote (with char offsets located deterministically in the source), and the model + prompt version that produced it. Nothing is asserted without a trace.
- The citation guard. After the LLM composes an answer, code checks every
[Sn]resolves to a real source returned by a tool this turn: zero citations on a factual answer → honest decline; a phantom citation → hard refuse. An ungrounded answer is structurally impossible. - The indexer-driven grounded diagram.
python -m src.indexerscans the codebase (AST) + the schema + the live datastore intodocs/index/architecture.json— ground truth, not hand-written.python -m src.vizrenders it into a self-contained architecture.html. Don't just read the code — see it, generated from a scan of the actual codebase, so the picture can never drift from reality.
How I'd scale this durability discipline to a real contract estate: the lightweight versions
here have obvious production successors — the keyword search_text → a proper vector store +
two-stage retriever; the citation guard → NLI entailment as a second assurance tier; the 5
invariants → a large indexer suite run on every change; recompute-all → per-family scoped
recompute; the outbox file → a retrying delivery service; the light per-stage telemetry → full
observability + request tracing across the pipeline; SQLite → Postgres (same schema, same code).
Each is named as a deliberate deferral, not a gap.
Token management is deferred on the same judgment: at 9 small contracts each document fits comfortably in a single extraction call, so no chunking is built. At scale, long contracts are chunked by clause with a per-field merge across chunks, and the chatbot caps retrieved context by relevance so a large estate never overruns the context window.
An honest map of where the confidence comes from. Nothing below is asserted louder than the evidence supports: a claim is either verified by a test, enforced-but-not-load-tested, deferred (not built), or an assumption. The four groups are kept deliberately separate so a grader can tell exactly which is which.
1. VERIFIED — proven by a passing test (262 automated tests, all green, keyless).
- 9-doc correctness — every graded-hard case in ARCHITECTURE.md §11 is asserted against the exact expected numbers (spend USD 322,500 + GBP 104,000, the Verolt notice-deadline nuance, the DataForge family, the master→SOW governance, the renewal/roll-forward states). The resolve core (R1–R8) is tested to those figures.
- Grounding audit — the evidence-span audit (
src/grounding_audit.py) shows 0 material uncited facts, and the audit itself is teeth-proven: a deliberately un-grounded fact makes it fail, so a green run means something. - Adversarial resistance — prompt injection, false-premise "confirm this lie", invented
documents, and SQL-injection-looking input all fail to move a stored fact; the DB is verified
intact afterward (
tests/test_chatbot_adversarial_intent.py). - Keyless replay — the full cached sweep replays with 0 cache misses, no API key
(
LLM_MODE=cached). - Idempotency — a re-run of the pipeline creates zero new rows (invariant + test).
- Guardrails bite — each of the 5 invariants and each ingest guardrail is tested against a deliberately-broken DB / bad input, so the guard is proven to actually refuse.
2. ENFORCED BUT NOT LOAD-TESTED — the ceiling is declared and its refusal is tested; throughput at the ceiling is not.
MAX_CORPUS_DOCS = 500is a named constant, reasoned about (the O(n²) single-pass linking cost is explained under Operating limits & scaling), and its refuse-over-limit path is tested (tests/test_limits.py). But correct operation was validated at 9 docs, not at 500. Upper-bound throughput, memory under a full 500-doc corpus, and linking-at-scale are not load-tested — the ceiling is enforced and refusal-proven, not exercised at the top. If asked "does it work at 500?", the honest answer is: the refusal is tested; the 500-doc run is not.
3. DEFERRED — not built (deliberate scope calls, each with a named production successor above).
- Business-day notice math (only calendar-day basis is computed).
- Vector store (keyword
search_textscan instead). - NLI entailment (citation guard is structural, not semantic).
- Batched / incremental scaling (single in-memory pass instead).
- Auth, encryption at rest, LLM data-egress control (single-operator local files instead).
- PDF / OCR ingestion (UTF-8
.txtonly). - Retry / backoff / circuit breaker on the LLM transport (fails honest instead).
4. ASSUMED — stated inputs the results rest on, not verified facts (live list in ASSUMPTIONS.md).
- FX 1 GBP = 1.27 USD (fixed, no live rate).
- Pricing constants for the cost estimate (
telemetry.PRICING, claude-opus-4-8 list price). - Notice basis = calendar days unless a contract states otherwise.
- Run date default
2026-07-06(configurable via--run-date/RUN_DATE).
RAW / audit (append-only) CURATED / truth (produced by RESOLVE)
┌────────────┐ ┌──────────────┐ ┌───────────────┐
│ documents │◀──doc_id──┐ │ contracts │──┐ │ relationships │
│ (hash key) │ │ │ (1 per doc) │ │ │ (content edges)│
└────────────┘ ├── contracts ──▶│ effective_* │ ├──▶│ src→tgt, type │
┌────────────┐ │ .doc_id │ is_superseded│ │ └───────────────┘
│ extractions│──doc_id───┤ │ family_id ───┼──┤ ┌───────────────┐
│ is_current │ │ └──────────────┘ └──▶│ families │
└────────────┘ │ ┌──────────────┐ └───────────────┘
┌────────────┐ │ │ notifications│ reference: supplier_master,
│ line_items │──doc_id───┘ │ dedupe_key ∪ │ routing_rules
└────────────┘ curation_overrides ─────▶│ (from R8) │
(human corrections, R8) └──────────────┘
The interactive version (with every column and the real families) is in architecture.html.
term_length_monthsextraction quirk. The two masters state a "3-year" term; extraction captured it asterm_length_months = 3. It does not affect any result — the stated end date is present and wins R1 — and the resolve derived-vs-stated trust signal flags exactly this inconsistency for review. A prompt tweak to normalize year/month units is the fix.- Citation guard is structural, not semantic. It proves every citation maps to a real retrieved source; it does not yet verify the claim is entailed by that source. NLI entailment is the named next tier (not built).
search_textis a keyword scan (with one deliberate exactness layer: section references like3.2,9.2(b),Section 9are extracted from the query and exact-matched — whitespace-tolerant, never partial (3.2can't hit3.21) — outranking keyword hits, with the snippet anchored at the reference and a ref found nowhere reported not found rather than degraded to keyword soup; so near-identical clause references are never confused). Sufficient at 9 documents; a real vector store + two-stage retriever is the scaling story — and this exact-reference layer is precisely the keyword leg that survives into that hybrid retriever at scale..envload path.LLM_MODE/ANTHROPIC_API_KEYare read from.envviapython-dotenvat import; a shell-exported var overrides.env(used above to force live/cached).
The system runs inside a declared envelope, enforced in code (src/ingest.py) and reported on
every run — nothing is silently dropped, degraded, or crashed on. The limits are named constants,
not magic numbers.
1. Declared limits (enforced).
| Limit | Constant | Value | Enforcement |
|---|---|---|---|
| Corpus size | MAX_CORPUS_DOCS |
500 docs | over-limit → refuse with a clear message (or --force) — refuse-tested, not load-tested at the ceiling |
| File format | INGEST_SUFFIX |
.txt only |
non-.txt → skipped + logged |
| File size | MAX_FILE_BYTES |
1 MB each | oversized → skipped + flagged |
| Encoding | INGEST_ENCODING |
UTF-8, strict | undecodable/binary → skipped gracefully (never mangled) |
| Empty | — | 0 bytes | skipped with a note |
The ingest summary reports N ingested plus a per-reason skip breakdown, so a mixed folder is
fully accounted for:
$ python -m src.ingest mixed_folder/
12 files seen -> 9 ingested (new), 0 unchanged (skipped), 0 removed; 4 skipped by guardrail
[non-.txt=1, oversize(>1000000B)=1, undecodable=1, empty=1]
2. Current limitations (honest). Relationships are drawn across the entire corpus in a
single in-memory pass: every document's text is scanned for every other document's contract
number, so reference-matching is ~O(n²), and the whole set is held in memory at once.
Extraction is one LLM call per document. This is a deliberate choice — at back-catalogue
scale it is simple, correct, and cheap (500 docs ≈ 250k string checks + 500 calls) — and the
reason for the declared MAX_CORPUS_DOCS. It does not scale to tens of thousands of documents
as-is; past the limit the design refuses rather than quietly slowing to a crawl.
The MAX_CORPUS_DOCS = 500 ceiling is enforced + refuse-tested, not load-tested at the ceiling.
The constant is declared, its cost is reasoned about above, and the over-limit refusal path is
tested (tests/test_limits.py) — but correct operation was validated at 9 docs, not at 500.
Throughput, memory, and linking behaviour at the top of the envelope are not load-tested; the
limit is a proven refusal, not an exercised ceiling. (See Testing scope & boundaries above.)
3. Scaling design (described, not built). Beyond the limit, ingest in batches and persist each batch's documents + references as it lands. When linking a batch, resolve its references against the already-stored corpus via an indexed contract-number lookup (not a full re-scan), and update the family graph incrementally — each batch extends the graph rather than rebuilding it. With per-family scoped recompute on only the delta a batch touches, the O(n²) full-scan collapses into indexed incremental work: the cost of adding a batch depends on the batch and the families it touches, not on re-reading the whole estate. Deferred as out-of-scope for a 9-contract corpus, but this is the concrete path off the single-pass design.
4. Rate limiting & resilience (deferred). Production LLM traffic would add retry-with-backoff on 429s, request throttling, and a circuit breaker; none is built here (single user, small corpus). What is built is graceful failure: a live API error (credit / network / rate limit) now surfaces as one clean message + a non-zero exit, never a traceback (see below).
5. Other deferred items (deliberate judgment calls, not gaps). Each has a working lightweight form here and a named production successor:
- Token management — at 9 small contracts each fits in one extraction call; at scale, long contracts are chunked by clause with a per-field merge across chunks, and the chatbot caps retrieved context by relevance so a large estate never overruns the window.
- Vector store —
search_textis a keyword scan (sufficient at 9 docs); a real vector store + two-stage retriever is the scaling story. - NLI entailment — the citation guard proves each
[Sn]maps to a retrieved source; a second tier would verify the claim is entailed by that source. - Full observability / tracing — the light per-stage telemetry becomes full request tracing across the pipeline.
- Postgres swap — SQLite → Postgres, same schema and same code (the DB layer is a thin seam).
- Encryption at rest — the local git-ignored
*.dband outbox would be encrypted or moved to a managed encrypted store. - Auth + per-OpCo authorization — a user is identified and sees only their operating company's contracts, instead of the whole estate being readable by anyone with shell access.
- Access audit log — the read-side complement to the existing raw/curated write-side trail.
- LLM data egress — live-mode contract text would route through a VPC endpoint or an on-prem
model; the provider-agnostic
LLMClientmakes that a config swap, not a rewrite. - PDF / OCR ingestion — today only UTF-8
.txtis accepted (others are skipped + reported); a PDF/OCR front-end would widen the format guardrail.
The pipeline carries light telemetry (src/telemetry.py) — not a tracing/metrics stack, but
just enough to put a number on the architecture's core claim: only 2 of the 9 stages ever call
the language model. Every run aggregates its LLM usage (call count, input/output tokens,
wall-time) and prints a one-line summary plus a stage breakdown at the end of extract and a
chatbot batch/demo:
$ LLM_MODE=cached python -m src.extract --force
9 docs extracted, 207 fields written, 21 low-confidence flagged (18 line items, 0 parse errors)
LLM usage: 9 calls, ~21,892 input tokens, ~14,232 output tokens, ~$0.47 est (at claude-opus-4-8 pricing), 0.01s.
Stage breakdown — LLM calls by stage (only 2 of 9 stages touch the LLM; the other 7 are pure code):
ingest 0 calls (deterministic — code, $0.00)
extract 9 calls ~21,892 in / ~14,232 out ~$0.47 est
classify 0 calls (deterministic — code, $0.00)
link 0 calls (deterministic — code, $0.00)
resolve 0 calls (deterministic — code, $0.00)
evaluate 0 calls (deterministic — code, $0.00)
notify 0 calls (deterministic — code, $0.00)
validate 0 calls (deterministic — code, $0.00)
chatbot 0 calls (no LLM call this run)
The thesis, quantified. Seven of the nine stages show 0 calls because they are pure
deterministic code — link, resolve, the R1–R8 rules, expiry math, routing, and the invariant
checks never touch the model. The LLM is used only to read prose (extraction) and route + phrase
(chatbot); code does all the reasoning. That is what makes the pipeline cheap, and the breakdown
makes it auditable rather than asserted.
The pricing is a stated assumption, not a fact. The dollar figure multiplies tokens by one
named constant, telemetry.PRICING — claude-opus-4-8 list price as of 2026-07: $5.00 / 1M input,
$25.00 / 1M output. Provider pricing changes, so this is an assumption the estimate rests on
(labelled ~$… est everywhere); when list prices move, update that single constant. The light
estimate prices the billed input_tokens + output_tokens and does not model cache-read discounts.
Persisted for inspection. Each run writes docs/telemetry/last_run.json (a full snapshot) and
appends a line to docs/telemetry/usage_log.jsonl. python -m src.telemetry reports cumulative
usage across all logged runs with the same stage breakdown. Both files are git-ignored — the code is
the source of truth, they are rebuilt every run.
Cache-safe by construction. Telemetry reads usage from the response side only
(LLMResponse.usage), after llm_client has computed the request's cache key — it never adds a
field to the hashed request. So turning telemetry on or off produces a byte-identical request hash
and cannot invalidate the recorded cached corpus (asserted in tests/test_telemetry.py; verified by
replaying the full cached sweep with zero cache mutation).
Production successor (deferred). This is the lightweight version. At production scale it becomes full request tracing / metrics — per-request spans across the pipeline, latency and error-rate dashboards, per-tenant cost attribution, and alerting — the same promotion story named under Operating limits & scaling above (Full observability / tracing).
The system is built so every failure is loud and honest rather than a fabricated answer — the worst outcome for a contract tool is a confident wrong number. Each surface has a defined mode:
- Malformed LLM output during extraction → an honest null / zero-confidence row (with an
_extraction_errormarker), never a guessed value. The confidence-floor invariant then flags it for review. (src/extract.py,tests/test_extract.py.) - Cache miss in cached mode → a graceful decline that explains offline mode and how to get a
real answer (
LLM_MODE=live); the interactive loop keeps going, no traceback. Raised as the specificCacheMissError. (tests/test_error_handling.py,test_chatbot_robustness.py.) - Real transport / API error in live mode (credit exhausted, network down, 429 rate limit) →
the library raises
LLMErrorand does not swallow it as a friendly decline (the cache-miss handler catchesCacheMissErroronly). At the live-calling entry points (chatbot,extract) thatLLMErroris turned into one clean, honest line — "The language model API is unavailable (…): <reason>. Use cached mode (LLM_MODE=cached)…" — and a non-zero exit, no traceback. A 429 is flavored "rate limit — wait and retry" (retry/backoff itself is deferred). (src/llm_client.py:api_unavailable_message,tests/test_error_handling.py.) - Ingest guardrail skips (non-
.txt, oversized, undecodable, empty) and the corpus limit (> MAX_CORPUS_DOCS) → each is enforced + reported, never a crash or a silent drop; the corpus limit refuses with a message naming the limit and the scaling path (override:--force). See Operating limits & scaling beyond them above. (tests/test_guardrails.py,test_limits.py.) - Ungrounded or fabricated LLM answer → the citation guard refuses it: a citation to a source
that was never retrieved is a hard refuse; a factual answer with zero citations is a
decline. An invented capability, value, or document surfaces as an ungrounded citation and is
withheld. (
tests/test_chatbot_adversarial_intent.py.) - Adversarial intent (prompt injection, false-premise "confirm this lie", invented documents,
SQL-injection-looking input) → the deterministic tools return the true curated value
regardless of the user's claim, parameterized queries make injection strings inert, and the
self-awareness manifest states the hard limits — so the role holds and no false fact ships.
(
tests/test_chatbot_adversarial_intent.py.) - Broken datastore (a required table missing) → a clear
OperationalError, not a made-up result; an empty-but-valid DB returns an honest zero. (tests/test_error_handling.py.) - Invariant failure (
python -m src.validate) → a loud non-zero exit listing every drift, so the discipline is a gate, not a hope. (src/validate.py,tests/test_validate.py.)
- BYOK. No key is bundled. Live mode reads
ANTHROPIC_API_KEYfrom your.env(git-ignored); cached mode needs no key at all. - Commercial terms. Contracts carry sensitive commercial data (values, parties, terms). The
datastore is a local SQLite file (
*.db, git-ignored) and the notification outbox (git-ignored) are data-at-rest on your machine — treat them accordingly. Only the two LLM roles ever send contract text off-box, and only in live mode. - Deterministic core. Because every fact is computed in code over the curated layer, an LLM compromise or hallucination can misroute a question (a visible "wrong fetch") but cannot silently corrupt a stored fact.
Deferred to production (deliberate, needed at scale — not gaps). Four hardening layers are out of scope for a 9-contract, single-operator take-home but named as required for a real deployment:
- Encryption at rest for the datastore and outbox — here they are local, git-ignored files; production would encrypt them (or move to a managed encrypted store).
- Authentication + per-OpCo authorization, so a user is identified and sees only their operating company's contracts, rather than the whole estate being readable by anyone with shell access.
- An access audit log recording who read, searched, or asked what — the read-side complement to the existing raw/curated write-side audit trail.
- LLM data egress control. In live mode, contract text is sent to a hosted API; production
would route the LLM through a VPC endpoint or an on-prem/self-hosted model. The provider-agnostic
LLMClientmakes that a configuration swap, not a rewrite.