Retrieval for a regulated insurer — 40M docs, per-clause ACLs, 12 weeks

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The problem, in the client's words

"Our claims handlers waste 20 minutes per complex claim searching policy documents,
precedents and regulatory guidance. We want them to ask a question and get a cited answer.
Nothing can leave our VPC, and a handler must never see a document outside their line of
business."

Constraints

• Latency: 3 s p95. Handlers are on the phone to a customer.
• Cost: no hard ceiling given, which means there is one and nobody has said it. I will
  produce a per-query number in week one and make them own it.
• Data residency: everything inside their VPC. No managed API that leaves it.
• ACLs: line of business × jurisdiction × seniority. Roughly 400 distinct scopes.
• Corpus: ~40M chunks. Policy documents change on renewal; regulatory guidance changes
  unpredictably and matters immediately.
• Team: me plus one of their engineers, twelve weeks.

Proposed design

flowchart LR
    subgraph idx["Index path — nightly + CDC"]
        S[Policy docs<br/>Precedents<br/>Reg guidance] --> P[Parse + normalise]
        P --> C[Structural chunking<br/>clause-level, heading path retained]
        C --> A[ACL denormalised onto chunk<br/>+ its own CDC stream]
        A --> E[Self-hosted encoder<br/>GPU backfill budgeted separately]
        E --> V[(OpenSearch in-VPC<br/>BM25 + kNN, one filter language)]
    end
    subgraph qp["Query path"]
        Q[Question] --> F[ACL predicate built from<br/>the handler's session claims]
        F --> H[Hybrid retrieve N=100<br/>ACL pre-filtered in the query]
        H --> R[Cross-encoder rerank →25]
        R --> K[Pack k=6, 5k token cap]
        K --> G[Generate + cite]
        G --> GD{Inline guardrail<br/>grounding + policy}
        GD -->|pass| U[Handler]
        GD -->|fail| RG[Regenerate or abstain]
    end
    V --> H

Loading

Key choices and why:

1. OpenSearch in-VPC, not a managed vector DB — one filter language and one ACL model for
   both legs, and it satisfies residency. Costs real cluster operations, which is a headcount
   conversation I will have in week one rather than week ten.
2. Clause-level structural chunking. For contracts the natural retrievable unit is a
   clause, not a paragraph. Heading path carried into every chunk so a retrieved clause is
   attributable.
3. ACL denormalised onto the chunk, with its own change-capture stream. Resolving 400
   scopes per request against the source system would blow the latency budget. The cost is
   revocation lag, and I will state it in minutes.
4. Inline guardrail from day one. The cost of a wrong answer here is regulatory, not
   embarrassing.

What my own design costs

• Pre-filtering with 400 selective scopes will degrade ANN recall. Graph traversal
  dead-ends when the permitted set is sparse. I plan per-line-of-business namespaces to
  mitigate, which costs cross-LOB recall — real, and I would rather lose that than leak.
• The guardrail is inside the latency budget on every turn. That is ~200 ms I do not get
  back, and it makes the 3 s p95 tighter than it looks once generation is in there.
• Self-hosted encoder means a GPU backfill for 40M chunks before we serve a single query,
  and again on every encoder upgrade. Weeks, not days.
• Twelve weeks is not enough to do all of this well. I would cut agentic search entirely
  and probably cut the precedents corpus from phase one.

What I would measure first

Week one, before any of the above: label 100 real handler questions against the
fault-isolation tree. If most failures turn out to be "the document exists but is out of
date", this is a freshness engagement and the retrieval architecture above is over-built.

What I want challenged

1. Is denormalised ACL with a CDC stream the right call at 400 scopes, or should I be pushing
   for namespaces from the start and accepting the cross-scope recall loss?
2. Am I wrong to cut the precedents corpus from phase one? It is the part the client is most
   excited about.

[akash-coded]

Noor · cohort peer · works in a regulated bank

Reviewing this from the regulated side, because I have shipped something close to it and the design has one assumption that will not survive contact.

"ACL denormalised onto chunk + its own CDC stream." With ~400 scopes expressed as line-of-business × jurisdiction × seniority, you are not storing 400 values — you are storing a set per chunk, and sets do not denormalise cleanly.

The failure is not on the read path, it is on the re-grant. Someone moves team, a jurisdiction's rules change, a whole LOB gets reorganised — and now you are rewriting the ACL field on some large fraction of 40M chunks. Your CDC stream is sized for document edits, which are rare. Permission changes are not rare and they are bursty: a reorganisation touches millions of chunks in one transaction.

What I would want to see argued: why not keep ACLs out of the chunk and evaluate a predicate at query time against a separate, small, fast-changing store? You pay a join; you stop paying a rewrite.

[akash-coded]

the original poster · cohort

This is the strongest objection and I did not have a good answer to it at the time.

Where I would still defend denormalisation: pre-filtering has to happen inside the retrieval query, not after it. If ACLs live elsewhere, either you post-filter — which collapses k unpredictably and is the k_collapse failure notebook 03 measures — or you push a potentially enormous ID list into the query, which OpenSearch will not thank you for.

Where you are right and I would change the design: the ACL should be a small denormalised group key, not the full set. Chunks carry acl_group_id; the mapping from user to permitted group IDs is resolved at query time from the small fast store. A reorganisation then rewrites the mapping, not 40M chunks.

That is a materially better design and it came out of this comment.

[akash-coded]

Noor · cohort peer · works in a regulated bank

Yes — that is the shape. One addition: version the group mapping and put the version in the trace. When someone asks in six months "could this user have seen this document on the 3rd of March", you need the mapping as it was, not as it is. Audit will ask. Ours did.

[akash-coded]

Marcus · cohort peer

On cost — the design has "self-hosted encoder, GPU backfill budgeted separately". Separately from what? That reads like the line item that eats the project. If the corpus is 40M chunks and you re-embed on every encoder change, is that not the dominant cost in the whole system?

[akash-coded]

Rafael · cohort peer · came from search infra

It is usually the second-largest line and the most under-forecast, so pushing on it is right. But "budgeted separately" is defensible if it means amortised differently, which I read as the intent.

The distinction that matters: backfill is capex-shaped — large, infrequent, plannable. Serving is opex-shaped — smaller, continuous, scales with traffic. Blending them into one per-query number hides the fact that one of them is a decision you make twice a year and the other is a bill you pay every day.

Where I would push instead: what is the re-embed trigger policy? "Encoder change" is not a policy, it is an event. Without a written rule — how much quality gain justifies a week of GPU and a shadow evaluation — you get one of two failure modes: never upgrading, or upgrading on every release because someone read a benchmark. The design should name the threshold.

[akash-coded]

Dana · reviewer · owns the eval harness

One thing missing from the design that will be the first question in an actual client review: there is no evaluation plan.

The diagram goes from documents to a handler. Nowhere does it say how you know it works, or how you would know it had stopped working. In a regulated context that is not a nice-to-have — "the model got worse and we noticed in week six" is a reportable event for some of these clients.

Concretely I would want, before any of the retrieval design:

• Who writes the gold set, and when. Twelve weeks with one of their engineers means the eval set competes with the build for the same person. Say which wins.
• The frozen slice, agreed with them up front, so that "better" has a definition neither side can move afterwards.
• What blocks a release. Your CI gate is the model for this and it belongs in the proposal, not in the implementation.
• The abstention target. In insurance, a confident wrong answer to a handler on a call is materially worse than a refusal, and that asymmetry should be a number before you build, not a discovery after.

[akash-coded]

Marking Dana's comment as the answer, because the missing evaluation plan is the defect that would actually lose this engagement, and the other two are refinements to a design that would otherwise be built without a definition of "working".

Synthesis of the three critiques, and what changes:

Critique	Verdict	Change
ACL denormalisation does not survive re-grants (Noor)	Accepted	Chunks carry acl_group_id; user→groups resolved at query time; mapping is versioned and lands in the trace
Encoder backfill is under-forecast (Marcus/Rafael)	Partly	Capex/opex split stands. Missing piece is a written re-embed trigger policy with a quality threshold
No evaluation plan (Dana)	Accepted	Weeks 1–2 are the eval set and the frozen slice, agreed in writing, before any retrieval work

What stays as designed and why: OpenSearch in-VPC, clause-level structural chunking, and hybrid-with-rerank all survive review unchanged. The residency constraint makes the first non-negotiable, and the other two are corpus-shaped decisions that this review did not challenge.

What is still open, and should be argued in its own thread rather than buried here: whether the inline guardrail is a model call or a classifier. That is #35, and it is the same question with a different failure cost.