Multi-tenant retrieval where two tenants want different encoders

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The constraints

• Tenants: 40, from 2k to 900k chunks. Long tail
• Isolation: hard. A tenant must never retrieve another's content, nor infer its existence
• The awkward one: two large tenants have domain vocabulary — one legal, one clinical — and
  want a fine-tuned encoder. The other 38 are happy with the shared one
• Cost: the 38 small tenants must not pay for the 2 large ones' infrastructure

The question

One index with a tenant predicate, or an index per tenant?

	Shared + predicate	Per tenant
Isolation	a filter — one bug from a leak	physical
Small tenants	cheap	38 near-empty indexes with fixed overhead
Different encoders	impossible — one index, one embedder_tag	trivial
Ops	one thing to run	40 things to run

Proposed design: neither, exactly

Tiered. The 38 small tenants share one index with an ACL predicate pushed into the query. The 2
encoder-divergent tenants get their own. A router picks by tenant id, and the Hit interface is
identical either way so nothing above the router knows which shape it hit.

What I am least sure about

Two code paths that must stay behaviourally identical — and the shared one is exercised constantly
while the dedicated one holds the large accounts.

[akash-coded]

Noor · cohort peer · works in a regulated bank

The tiering is right, and the isolation argument for it is stronger than the cost argument you led
with. I would reorder the write-up.

A shared index with a predicate is one query-construction bug from a cross-tenant leak. In a
regulated context that is not a quality incident, it is a reportable breach — the difference
between "we filter" and "they are physically separate" is the difference between a paragraph and a
regulator.

But do not let that argue you into 40 indexes. The right frame is: which tenants have a
contractual requirement for physical separation? That set is usually small, usually known, and
usually not the same set as "wants a custom encoder". You may need three tiers, not two.

[akash-coded]

Three tiers is right and I had collapsed two orthogonal axes into one. Encoder divergence and
isolation requirement are different questions and I was treating them as a single tenant list.

[akash-coded]

Dana · reviewer · owns the eval harness

What will actually hurt is neither of those. It is that you now have two code paths and one eval
set.

The shared path is exercised by 38 tenants continuously. The dedicated path holds your two largest
accounts and is exercised rarely. That is backwards from where you want coverage, and it is the
shape of every "it worked in staging" incident I have seen.

Before this ships:

• The eval set runs against both paths on every PR. A path not in CI will drift, and you will
  find out from the largest customer you have
• A contract test asserting behavioural equivalence. It will fail the first time you run it,
  and that failure is the point
• Per-tenant metric slices — an aggregate across 40 tenants of wildly different sizes is
  dominated by the two large ones and will hide a regression affecting the other 38

The last is the aggregate-hides-minority-failures problem with tenants as the slice.

[akash-coded]

Rafael · cohort peer · came from search infra

Endorsing the contract test with a warning: it cannot assert identical rankings once the tenants
have different encoders, because the embeddings genuinely differ.

It can only assert invariants — ACL isolation holds, k is respected, no cross-tenant id ever
appears. Write it as an invariant test, not a golden-output test, or it is permanently red and gets
deleted within a month.

[akash-coded]

Correct, and that distinction would have cost me a week. Invariants, not golden outputs.

[akash-coded]

Marking Dana's, because two-paths-one-eval-set is the defect that would cause the outage; the other
two are refinements to a design that would otherwise ship untested.

Accepted: three tiers on two orthogonal axes, not two tiers on a conflated one. CI runs the
eval set against both paths. Invariant contract tests. Per-tenant slices.

Stands: the router behind a common Hit interface, and the shared index for the long tail.

Open: what happens when a small tenant grows into the encoder-divergent tier — a migration
between paths with no downtime. Not designed. Probably its own thread.