hidef — hide & define

Portable, auditable, on-device pseudonymization for engineering organizations. hidef strips PII and business-sensitive information from documents, spreadsheets and chat text before any cloud LLM sees them, restores full detail in the model's output, and proves what was sent via signed receipts.

Built to spec v0.3 (see docs/ history in the handoff). Languages in v1: English, Chinese, Japanese, with Malaysia/Australia format packs. Compliance targets: GDPR and Malaysia PDPA.

Honest scope: hidef protects identifiers and listed secrets, not ideas expressed in free text. If a paragraph describes a distinctive project in recognizable detail, redacting names alone may not prevent re-identification — use deny-listed phrases or don't send it.

How it works

draft → detected → reviewed → pending → confirmed(receipt) → released

• redact() returns a pending payload — never releasable text. There is no API path to released without confirm(). This gate lives in the core library, so every wrapper (CLI, web UI, future MCP/proxy) inherits it.
• On confirm, the core emits the safe payload (the only sendable artifact) and a signed receipt: payload hash, source hash, per-type counts, policy id+version, mode, warnings shown, confirmer, timestamp — never values.
• Rehydration runs a hard verification pass: every placeholder sent must appear in the model output, or hydration fails loudly listing what was lost.

Quick start

git clone https://github.com/dandycrypto/hidef.git && cd hidef
./install.sh           # tests + builds + installs `redact` to ~/.local/bin
                       # (Windows: powershell -ExecutionPolicy Bypass -File install.ps1)

cd ~/your-documents
redact init --project my-project           # scaffold .redact/ (policy, glossary, templates)
redact scan  brief.md                      # findings report, nothing stored
redact run   brief.md                      # detect → review → PREVIEW → confirm
                                           #   → brief.safe.md + receipt.json
# …paste brief.safe.md into your LLM, save the answer as out.md…
redact hydrate out.md                      # placeholders → values + verification
redact serve                               # local web UI on 127.0.0.1 (token-gated)
redact mcp --dir ~/your-documents          # MCP server: use hidef as an LLM tool

• Guides: docs/INSTALL.md (technical) · docs/USER-GUIDE.md (plain language, for everyone)
• Sample data to test with immediately: examples/ (fictional tender brief, pipeline sheet, 中文 minutes, 日本語 memo)

Use it inside ChatGPT / Claude / Gemini (browser extension)

The extension/ folder ships an MV3 extension that protects the chat box itself: a live detection pill while you type, send interception with a one-tap review sheet (the core confirm gate, in-page), protected send, and local restoration of replies — real values never leave your machine. Pair it with redact serve (see extension/README.md); an end-to-end suite drives the real extension in real Chromium in CI.

Use it as an LLM tool (MCP)

redact mcp exposes hidef to Claude Desktop / Claude Code / any MCP client as four tools — scan_file, redact_file, fetch_confirmed, hydrate_to_file — designed so nothing sensitive can enter the model's context:

• scans and previews return counts only, never detected values;
• redact_file releases the safe payload only after you confirm in the local browser UI (or via a signed auto_confirm org policy);
• hydrate_to_file writes the restored answer to a local file and returns only the verification report — real names never round-trip through the model.

claude mcp add hidef -- redact mcp --dir /path/to/your/project

Detection pipeline (spec §3)

1. NFKC + offset map — full-width digits (０１２３) are matched, exact original spans are replaced. No \b regex anywhere; boundaries come from Intl.Segmenter.
2. Allow-list glossary claims spans first (engineering terms, standards, acronyms — org-shared via git, PII-free).
3. Deny values / public targets (deny values live only in the local vault).
4. Recognizer packs (deterministic, checksum-validated): MY (MyKad, SSM, phones, addresses, plates), AU (ABN/ACN/TFN/Medicare, phones, postcodes), ZH (身份证, 统一社会信用代码, HK/TW IDs, mobiles), JA (マイナンバー, 法人番号, 〒 postal, addresses, phones), universal (email, IBAN, Luhn cards, GPS, IP/MAC, passports, MONEY/AMOUNT). Low-specificity digit runs require context words — drawing numbers, tender refs and part codes survive.
5. Anchors: titles (Ir./Dato'/Tan Sri/… + name), 百家姓 surname + honorific (张伟先生, 王总), JA name + さん/様/氏/部長, company forms (Sdn Bhd, Pty Ltd, 株式会社, 有限公司, …).
6. NER tier: multilingual ONNX token classification via onnxruntime (CPU), pinned by SHA-256, runs only on unclaimed spans; low-confidence hits go to the review queue per policy. Both WordPiece and SentencePiece unigram tokenizers are supported (unigram uses Viterbi segmentation — the piece CJK text needs, since there are no spaces to split words on). Install the default 10-language bundle with redact models install (~130 MB, hashes pinned at install and verified on every load, integrity cross-checked against the publisher's digests), or point at any org-vetted XLM-R/mDeBERTa-class model for stronger Japanese coverage with --repo <org/model> --tokenizer unigram --expect <org-pins.json>. Without a verified bundle the pipeline runs deterministic-only and shows ⚠ NER disabled on this device — reduced name coverage — never silently. redact models shows status.
7. Triage: unknown all-caps tokens and unlisted NER hits land in one queue; one decision per term (allow / deny / public-target) persists org-wide.

Three-way classification & modes (spec §4)

class	document mode	research mode
allow	keep	keep
deny (your side)	redact	redact — always
public-target (client orgs, announced projects)	redact	keep (model needs them to search)

Research mode shows a linkage warning when deny-class values and public-target names would co-occur in one payload. AMOUNT is a first-class policy entity with a math hint when redacted.

Placeholders & vault (spec §5)

• [[P4LGK:PERSON]], [[CUV66:COMPANY]], [[MLRFK:AMOUNT]] — ASCII, code-like, survives translation. Payloads carry a configurable preamble telling the model to reproduce tokens exactly.
• Deterministic ids: base32(HMAC-SHA256(vault_key, NFKC(value)))[0:4] — same value → same placeholder across documents and sessions. Collisions extend the id.
• Vault: SQLite, one per project per user, strictly local. Every value is AES-256-GCM encrypted before touching disk; lookups go through an HMAC index. Key lives in the OS keystore (Keychain / libsecret / DPAPI), with PIN-derived (HIDEF_PIN) and 0600-file fallbacks (loud, never silent). Fields include created_by, retention_until, legal_basis; the audit table is append-only with a hash chain.
• Org-shared via git: glossaries, deny-list templates, policies, column mappings — rules, never data. Strictly local (~/.hidef): vaults, deny-list values, receipts, audit DB, keys. Documents are rehydrated by whoever redacted them; there is deliberately no cross-user vault merging.

Policy engine (spec §9)

.redact/policy.yaml — strict (queue must be empty before confirm), standard (ambiguous items queue), fast (chat-speed: auto-redact everything, lightweight confirm). auto_confirm: true only takes effect with a valid org signature (redact policy sign); accountability transfers to the signer and the receipt records it.

Spreadsheets (spec §6)

CSV/TSV/XLSX are first-class: per-column policies (redact columns pipeline.xlsx --set "Contact=deny" --set "Value=amount" --set "Client=public-target") are remembered per header fingerprint, so recurring reports need zero re-configuration. Output keeps the sheet shape (placeholders in cells) as csv/markdown/xlsx. docx and pdf extract visible body text (dependency-free: pdf via object scanning + FlateDecode + ToUnicode CMap decoding).

GDPR / PDPA (spec §10)

redact init drops DPIA and Records-of-Processing templates into .redact/templates/. Engineering measures: encrypted vault + OS keystore, confirm-gated release, receipts + hash-chained audit log (redact audit --verify), retention auto-expiry, redact burn (crypto-shredding: destroy key, then file), redact export --subject for DSARs. Legal sign-off remains the organization's responsibility.

Development

bun test          # 74 tests: checksum vectors, golden corpus (EN/ZH/JA) with a
                  # recall-weighted Fβ(β=2) gate ≥ 0.9, adversarial engineering
                  # set, round-trip property tests, state machine, vault, API,
                  # MCP, keystore probes, real ONNX inference on a tiny fixture
bun run bench     # spec §11 performance budget (gated in CI, see docs/PERFORMANCE.md)
bun run sbom      # CycloneDX SBOM
bun run build     # portable single binary via bun build --compile
redact doctor     # live verification of a machine (keystore, vault, NER, …)

CI runs the suite on Linux, macOS and Windows; tagged releases ship cross-compiled binaries with SHA-256 checksums, a CycloneDX SBOM, and GitHub build-provenance attestations.

Dependencies: platform stdlib + SQLite driver (built into Bun/Node) + onnxruntime (optional, loaded only when a verified model bundle exists). Auditability is a feature — the list stays that short.

Residual risks (documented, accepted — spec §12)

1. No detector reaches 100% recall; strict policy + confirm preview is the control, and the receipt records what was approved.
2. Paraphrase/translation can destroy placeholders; verification exposes it.
3. Distinctive projects may be re-identifiable from context; deny-list phrases or don't send.
4. The LLM provider still sees which topics an account researches over time.
5. The vault is the crown jewel; encryption, keystore, burn and retention expiry are the controls.

See SECURITY.md for the full threat model — including defenses proven with real adversarial browser tests, not just reviewed — covering malicious document content, shared/compromised LLM accounts reading tool-call transcripts, and hostile local web pages.