Repository navigation
Glossary
The shared vocabulary. Where a term has a precise meaning in this codebase, that meaning is the one below, and several are deliberately narrower than their everyday use.
A redaction whose status is accepted. The exporter reads these and nothing else (invariant 2). A suggestion nobody accepted appears in the export untouched, and a test asserts that. Rules and manual redactions are accepted when made, because they are decisions.
The per-owner gate that decides when a queued document starts processing (ANONIFY_BATCH_PROCESSING). It runs at the process route, the retry route, the end of every run (admitNext), and as a backstop in the cleanup endpoint (admitStalled). The queue is not a table: status: queued with no workflowRunId means waiting.
How Hush proposes a change: a card showing what the change will do and the evidence for it, which does nothing until Approve. Approvals are HMAC-signed by the server.
Links visible characters back to the byte ranges of the markup that produced them. Used by RTF and HTML email bodies. A literal atom is one source byte, an escape is several bytes for one character, and a structural atom contributes text (now markdown) of its own and is never removed.
Several documents sharing one review pass and one download. A batch owns decisions, not processing: each document keeps its own run, quota, failure and expiry. A message with attachments and a mailbox each become batches.
A decision carried across every document in a batch (BatchRule), including documents still processing, which pick it up in carryDecisions.
batch-report.json, in every batch zip: which upload each file came from (by id and part path, never a name), rebuilt mailboxes' included and left-out messages, skips with reasons, and vaults. Counts, never content.
AI_MODEL_CAPABILITIES: what setup or pnpm ai verify proved about a model (structured output, image input), bound to the provider, the model and the destination URL. Change any of them and the model must be verified again.
An attachment in a format Anonify cannot read: kept in the message with nothing inside it redacted, and said so in the report, per attachment.
What a value is: person, address, phone, email, government-id, bank-account, financial, date-of-birth, customer-id, confidential, api-key, url, face, other. Presets select categories, methods are gated by them, and reports count by them.
How every object has been sealed since 1.6.0: a 16-byte header, then 1 MiB AES-256-GCM chunks, each with its own tag, a counter nonce, and an AAD binding the header, the logical key and a final-chunk flag. v0 is the older single-pass envelope. Which one a document uses is recorded in encryptionFormat, never guessed from the bytes.
A hint, shown as a band and a percentage for triage. Never a probability, never a verdict.
A format that is never extracted itself and expands into documents. mbox is the only pure container (extractable: false, exportable: false), and it ends at status expanded.
The model's part of detection: classify, detect in ~6 KB chunks (told what patterns already found), verify shaky candidates. Optional, and paced through the throttle.
The corpus scorer's primary number: every character of a labelled value is under a redaction. Partly covered counts as missed, because partial coverage is a leak. In Hush, a place is "covered" when a redaction there is accepted or still suggested.
A random 256-bit key, one per document, wrapped by the master key and stored on the row. Never derived from anything an attacker controls (invariant 8).
A run where the model pass fell short (no model, unsupported, rate limit, empty balance, bad key, spend cap). It writes document.ai.degraded, and the usage panel says so. Pattern-only results must never look like a model's.
A pattern plus a validator: Luhn for cards, structural validity for SSNs, a required label for accounts and birth dates. Free, reproducible, no tokens, runs first.
What happened to an attachment on export: redacted, carried-through or removed.
The terminal status of a container: a mailbox that became a batch of messages. Neither ready nor failed.
Step 5 of detection: a value judged once is found at its other occurrences by local search. It costs a scan, not a request, and cannot give a different answer the second time.
The second artifact beside every export: counts by category, methods and styles, what was declined, the preset, attachment dispositions, and both checksums. Counts, never content, enforced by assertReportOmitsValues.
One of 16 stable codes on a failed document, with a fixed sentence and a retryable flag. The message never comes from the thrown error. See Failure Codes.
Two meanings, historically. In the API and schema, GlobalRule is any rule applied to one document (its own, or a copy of a batch or owner rule). In the UI, a Global-scope rule is an OwnerRule: applied to everything you upload from now on, pattern sealed, expiring 30 days after last use.
The review assistant: a tool-using agent over the document, with every change waiting for approval. See Hush Assistant.
A rule offer Hush makes in the browser, with no model, after three hand-made redactions of the same anchored shape (e.g. EMP-00\d{3}).
The rule governing every model answer: the model returns the text it saw, the application finds that text in the source itself, and anything it cannot locate is discarded (invariant 5).
The path an object was written under (documents/:id/source.bin, …). Bound into every chunk's AAD, so one of a document's objects cannot be swapped for another.
ENCRYPTION_KEY. Wraps every data key and seals owner-rule patterns and a ChatGPT sign-in. Checked at boot. Changing it makes everything stored permanently unreadable.
What an accepted redaction does to the bytes: mask (default), pseudonymize (a stable surrogate, no mapping kept), tokenize (a surrogate plus a vault), encrypt (ciphertext plus a key in the vault). Gated by category and by whether the target can hold text, applied per value, strongest wins. Distinct from image style (solid, blur, pixelate).
Layer B: pages (flat text + spans with offsets and geometry), sheets, regions. One vocabulary for every format, sealed outside the database, with a page index.
One value and all its occurrences (category|normalizedText), so a decision is taken once instead of seventeen times.
Everything in the normalized model except the pages: kind, metadata, sheets, regions, page numbers. The editor opens it first.
Pacing holds a request until the bucket can afford it, so known limits are never hit. Retrying handles unknown ones (429, 5xx, timeouts), with jitter and honouring Retry-After. 401, 402 and 403 are never retried. Step-level retries (1 s → 30 s backoff) are separate and cover storage and database blips.
Document.normalizedIndex: byte offsets of each page in the sealed model, so one page is one ranged read. Offsets only, never content.
Per document: 5 s of matching and 10,000 matches (50,000 for search). Over budget, a rule fails whole and writes nothing.
A named set of detectors and categories, chosen at upload. It changes what is looked for and nothing else, applies to model proposals too, and is never named for a regulation (enforced at import).
The batch download layout with every document's own output in folders by upload, message number and MIME part path. Never by subject or filename.
ANONIFY_PROFILE: self-hosted (no quotas, permissive) or demo (small allowances, strict).
How Hush's header classifies where answers come from: hosted API, local model ("Stays on this machine"), ChatGPT plan, or gateway.
A daily allowance per identity per UTC day (pages, filled cells, slides, KiB of email text, images, uploads, Hush tokens), charged once and idempotently. Self-hosted has none.
A token bucket rationing callers arriving here (upload, processing, export, read, search). Rate is not concurrency: a rate limit refuses, while concurrency makes work wait.
The linear-time regular-expression engine (RE2JS) used for search and rules. Backreferences and lookaround do not compile, so a pattern cannot hang the server.
The original-format download of an MBOX: its messages' verified exports in order, separator lines written fresh (MAILER-DAEMON, the message's own date), mboxrd quoting, then verified by re-splitting. Messages left out are named.
A redaction the reviewer declined ("kept in file"). A test asserts that a rejected value is still present in the export.
A pattern, a category, a scope (document, batch, global) and a kind (literal, regex). Applied on the server, never through a model, writing accepted redactions in one transaction. See Search and Rules.
HMAC, 5-minute TTL, domain-separated (document vs batch), compared timing-safely, and necessary but not sufficient.
A run of text with an id (its source address, e.g. word/header1.xml#p3r1) and offsets, plus geometry where the format has it. What boxes and edits are placed from.
A "use step" function in a durable run: retried on its own (up to 4 times, paced), result persisted, resumed on replay.
A redaction with status: "suggested": a proposal, not a decision. No code path makes detection produce an accepted redaction.
The stand-in string for a pseudonymised or tokenized value (PERSON_014), written where the value was, or drawn on the strip for PDFs and images. Excised before verification searches, because it is authored, not derived.
Alongside precise edits, every export removes accepted strings wherever else they appear in the container (headers, hidden sheets, notes, quoted replies).
One of up to four outputs from a single review, each a full pass with its own verification and report. Named from its methods, never typed.
The artifact that reverses tokenize / encrypt. For a single export it is returned inline and stored nowhere. A batch export seals it beside the file. It is never inside the file and never in the multi-copy zip.
Every export is re-opened and read as an adversary would, and refused if an accepted value survived: not saved, no link (invariant 4).
The batch-export unit: ANONIFY_BATCH_EXPORTING documents at once, with nothing in the next wave starting until the current one finishes, so a stop takes effect within one wave.
Where durable runs live: the platform on Vercel, or Postgres through @workflow/world-postgres with a worker started by instrumentation.ts.
Anonify: AI proposes, the application applies, and only what a person accepted is removed.
Repository · Report a bug · Request a feature · Security policy · Apache-2.0
This wiki describes v1.10.0 and summarises and links the in-repo documentation. Where the two disagree, the files under docs/, CHANGELOG.md and SECURITY.md are canonical.
Getting going
Using it
- Supported Formats
- Review Workspace
- Search and Rules
- Hush Assistant
- Redaction Presets
- Export and Anonymisation
- Batches and Mailboxes
How it works
- Architecture
- Processing Workflow
- Redaction Engine
- Storage and Encryption
- Streaming IO
- Data Model
- API Reference
- Security Model
When it breaks
Contributing
Repository · Issues · Releases · Changelog