PII Gate

A privacy boundary that sits between a source document and anything downstream - an embedding model, a vector store, an external LLM. Nothing crosses without passing through it.

Built on Microsoft Presidio. Docker deployable. The policy lives in a YAML file, not in code.

document ──▶ detect ──▶ decide ──▶ rewrite ──▶ safe text ──▶ embedding / LLM
                │          │                        │
                │          │                        └─▶ surrogate mapping
                │          │                             (stored separately)
                │          └─▶ below threshold? flag for human review
                └─▶ confidence score per span

Run it

export PII_GATE_SECRET="$(openssl rand -hex 32)"
./demo.sh

Or one document at a time:

python -m pii_gate.cli sample_docs/support_ticket.txt --rules rules.tuned.yaml
python -m pii_gate.cli sample_docs/support_ticket.txt --json

Docker:

docker build -t pii-gate .
docker run --rm -e PII_GATE_SECRET="$(openssl rand -hex 32)" pii-gate

Tests:

pytest tests/ -q

What it does

Detection with a confidence score per span. Presidio's built-in recognizers plus whatever the rule file declares.

Three verdicts per entity type, set in rules.yaml:

verdict	what happens	when to use it
keep	left alone	dates, URLs, anything not sensitive in your corpus
redact	replaced with [REDACTED:TYPE], unrecoverable	card numbers, salaries, internal codenames
pseudonymise	replaced with a stable surrogate, reversible	names, emails, employee IDs

Configurable rules. Entity thresholds, custom regex patterns with context words, an exact-match deny list, and an allow list for strings that look like PII but are known-safe in this corpus. All of it in rules.yaml. Changing policy does not mean changing code.

A report of what was detected, what was done, what was allow-listed, and what needs a human. Redacting without reporting is not auditable.

The two decisions that matter

Pseudonymisation, not redaction, for anything you still want to ask about

Plain redaction is safe and useless. Turn every name into [REDACTED] and the question "who did Sarah escalate to?" stops having an answer, because every person in the corpus is now the same token.

So names, emails and IDs get a stable surrogate instead: Sarah Whitfield becomes <PERSON_ea4090> in every document she appears in. The corpus stays answerable. The model still cannot name her.

The surrogate is an HMAC of the value under a secret key, not a plain hash - a plain hash of a person's name is reversible by anyone willing to hash a name list. The key never leaves the gate. The mapping is returned separately and never written into the safe text; store it where only the re-identification path can read it, and only render real values back to viewers entitled to see them.

Values that nobody should ever recover - card numbers, salary figures, internal codenames - get redact instead, and never enter the mapping at all.

Fail closed on uncertainty

A span the detector is unsure about is the dangerous case. Below the entity threshold but above review_threshold, the gate does not quietly pass the document through: it flags the span and exits non-zero so the calling pipeline can hold the document out of the index until a human clears it.

demo.sh shows this. Step 1 runs untuned rules and refuses the document because two phone numbers scored 0.40 against a 0.60 threshold. Step 2 runs rules tuned against this corpus and passes cleanly. The rule file is the control surface; until it is tuned, the gate errs toward blocking.

Known limits

Worth stating plainly, because a privacy control you have oversold is worse than one you have not.

• Alias resolution is not solved. Sarah Whitfield and a later bare Sarah are two entities to the NER model, and in one of the sample documents the bare mention is missed entirely. Real deployments need an entity resolution pass before the gate, or a per-corpus name gazetteer feeding the deny list.
• Street addresses are not covered. Presidio's LOCATION catches cities and countries, not 14 Berkeley Row. That needs a custom recognizer.
• Recall is a function of thresholds, and thresholds are corpus-specific. The numbers in rules.yaml are a starting point, not a guarantee. Tune them against a labelled sample of your own documents and re-measure.
• Checksummed types behave differently. Presidio validates card numbers with Luhn rather than matching on shape alone, so a made-up sixteen-digit string is correctly ignored. Useful, and worth knowing before you write a test with an invalid number and conclude the detector is broken.
• English only as configured. Other languages need their own spaCy model and their own thresholds.

Layout

rules.yaml         policy: thresholds, actions, patterns, deny/allow lists
rules.tuned.yaml   the same policy tuned against the sample corpus
pii_gate/config.py loads the policy, builds recognizers - no decisions here
pii_gate/gate.py   detect, resolve overlaps, decide, rewrite
pii_gate/report.py what was detected and what was done about it
pii_gate/cli.py    entry point; non-zero exit when review is required
sample_docs/       two documents sharing a person, to show surrogate stability
tests/             9 tests, including cross-document stability and fail-closed