Export and Anonymisation

Export and Anonymisation

What happens when you press Export: how a value is replaced, how the file is verified before anyone can download it, what the report says, what a vault is, and how to get values back.

Canonical sources: docs/pipelines.md ("What happens to a value") · docs/api.md (export, download, restore) · code: lib/redaction/export.ts, deliver.ts, methods.ts, surrogates.ts, vault.ts, variants.ts, restore.ts, report.ts, validation.ts.

---

1. The export pipeline

flowchart TD
    A["Sealed source (layer A)<br/>read whole"] --> PLAN
    C["Accepted redactions only (layer C)<br/>+ only the pages they touch"] --> PLAN["Plan<br/>per-format instructions<br/>method per value (strongest wins)"]
    PLAN --> GEN["Generate<br/>precise edits + value sweep<br/>+ metadata sanitisation"]
    GEN --> VER{"Verify<br/>re-open adversarially"}
    VER -- "an accepted value survived" --> REF["Refused: ExportVerificationError<br/>nothing stored · no link"]
    VER -- clean --> REP["Build the report<br/>assertReportOmitsValues"]
    REP -- "report quotes a value" --> REF2["Refused: ReportLeakError"]
    REP -- clean --> SEAL["SHA-256 → seal under the document key<br/>→ ExportArtifact row"]
    SEAL --> OUT["Signed 5-minute links: file + report<br/>vault returned inline, stored nowhere"]

Loading

One function, deliver, does this for a single export and for every document in a batch export, so there is one verification gate, never a second copy of it to forget. A message is the one document that is not exported on its own: exporting a message exports every attachment first through the same function, then substitutes the results back into the message, checked by checksum.

---

2. The export dialog

Control	Default	Notes
Method per category or per suggestion	mask	see §3
Metadata sanitisation	on	strips author, tooling, EXIF, GPS
[REDACTED] labels	off	writes a placeholder where content was removed
Image style	solid	solid / pixelate / blur, images only. The dialog says it plainly: heavy blur can in principle be attacked. Text is always filled solid.
More than one copy (variants)	one	up to 4 outputs from one review, e.g. an internal copy with names masked and a shareable copy with them tokenized

After generation the dialog shows each copy's download, report and checksum (folding away with more than one copy), an inline report summary (removed by category, "3 suggested items were not accepted and remain in the file", the preset caveat), the cost of the analysis, and the vault when one exists. The dialog scrolls instead of growing past the screen (fixed in 1.5.0).

Download all (since 1.5.0) zips every copy and its report in the browser, from the same signed links the individual buttons use, so nothing new is stored or authorised. The vault is deliberately left out of that zip: a zip that packed the reversal key next to the copy it reverses would make the convenient thing and the unsafe thing the same. Entries are named by variant (letter-redacted-tokenized.pdf).

---

3. What happens to a value: methods

An accepted redaction carries a method from a closed set:

Method	What happens to the bytes	Who can reverse it
mask (default)	replaced by a label, or by nothing	nobody
pseudonymize	replaced by a stable surrogate (PERSON_014): the same value maps to the same surrogate throughout the document	nobody, including Anonify; no mapping is kept anywhere
tokenize	the same surrogate, with the mapping written into a vault you download	whoever holds the vault
encrypt	replaced by AES-256-GCM ciphertext under a key in the vault	whoever holds the vault

None of these is anonymisation in any regulatory sense, and the interface does not say it is. pseudonymize in particular preserves equality, and equality is exactly what someone joins on to re-identify. tokenize and encrypt are reversible by design.

Two gates, and both have to open

The category gate is a table in lib/redaction/methods.ts:

Categories	Allowed
face, government-id, bank-account, financial, api-key	mask only. A pseudonymised government ID joins the same records the number did, and a face has no textual form.
address, date-of-birth	mask or encrypt; not the surrogate methods, because a stable surrogate for a street or a birth date is a quasi-identifier
everything else	all four

The capability gate is about the bytes. A method other than mask has to put a surrogate somewhere. A face, a whole column whose header the export keeps, or a region OCR found nothing behind has nowhere to put one, so masking is not just the default there: it is the only option. The method is stored, not trusted. Both gates are asked again at export time, so a method that was valid when chosen and is not valid now resolves to a mask rather than a substitution that cannot be made.

One method per value, not per redaction

A document contains occurrences of a string, and the value sweep finds the ones in a hidden sheet, a speaker note or a quoted reply that nobody reviewed. If a name were pseudonymised where it was reviewed and masked where it was swept, the export would silently break the join the pseudonym existed to preserve. So the method applies per value, and where redactions of one value disagree the strongest wins: mask > pseudonymize > encrypt > tokenize.

Where a surrogate goes, per format

• DOCX, PPTX, XLSX, CSV, TSV, TXT, RTF, EML: exactly where the value was, as real text the next program can read and search. In EML this includes header values and filenames: pseudonymising a sender is the same operation as redacting one.
• PDF and images have no text stream left to write into, because a redacted PDF page is rasterised. The strip covering the value is drawn with the surrogate on it, so a scanned contract gets the same vocabulary as a DOCX. The text is shrunk to fit, and dropped if it cannot be drawn legibly (an illegible smear reads as a rendering fault; a plain strip reads as what it is). A raster encrypt paints a short reference (ENC_007), and the vault carries the ciphertext.

---

4. Variants

ExportOptions carries per-category method overrides, and an export can ask for up to four variants. Each is a full pass: its own plan, artifact, verification and report. Nothing is shared between them, because the only way to know an artifact is clean is to re-open that artifact, and sharing work would mean verifying one file and delivering another.

• Variant names are derived from the methods applied, never typed, because the name reaches the report and the report must never carry a free string.
• The export rate limit is charged once per variant, all or nothing. If today's allowance does not stretch to every variant, the whole request is refused, rather than silently delivering only the masked copy of a request for a tokenized one.
• A batch gets one method per file, not variants per file (see Batches and Mailboxes).

---

5. The vault

The vault is a separate, labelled artifact, deliberately not the report:

	Report	Vault
Audience	someone who was not allowed to see the original	the one person allowed to reverse the substitution
Contains	counts, styles, checksums, never content (enforced)	the original values, and for encrypt the key
Stored	sealed beside the artifact	single export: returned inline in the response and stored nowhere (not in the database, not in storage)

Anonify cannot reverse an encrypt export. That is the design, not a gap. There is no URL for the vault and no way to ask for it again: save it or it is gone. A message and the attachments substituted into it share one key, so one vault opens the whole message.

The one exception is a batch export. It finishes minutes after the request that started it, so there is no response to hand the vault back in. The vault is therefore sealed under the same per-document key (vaultBlobKey), shipped in the archive beside the file (<name>-vault.json, never inside it), and purged with the document. Within the retention window the source document is already in that bucket under that key, so storing the vault grants no capability that was not already there. The batch report and the export dialog both say, before the run, that the archive contains both the reversible file and the thing that reverses it. A vault that fails its checksum is left out, because half a mapping restores half a document.

---

6. Restore

/restore → POST /api/restore: upload the exported file and its vault, and get the original back. It is export with the direction reversed: every format's value sweep already replaces a value wherever it occurs, which is exactly what a restore is, so nothing new had to be written per format.

• It takes no document id and stores nothing: not the upload, not the vault, not the result. A version that remembered the export could reverse it without you.
• Response headers: X-Restored-Values, X-Unresolved-Values, and X-Vault-Matches (whether the vault names this exact artifact; reported rather than enforced, because the wrong vault produces plausible nonsense rather than an error).
• Not possible: PDFs and images (their surrogates are pixels, 422); pseudonymised values (no mapping exists anywhere). A message's attachments are restored on their own, with the same vault.
• Filenames outside Latin-1 restore under their real name (fixed in 1.10.0).

---

7. Verification

The artifact is re-opened and read the way an adversary would, per format (Redaction Engine §7, Supported Formats). Any accepted value still present throws ExportVerificationError: the artifact is not saved, no link is issued, and the reviewer is told "did not pass verification and was not saved". The operator log records how many values leaked. The client is told nothing that would help locate them.

What verification had to learn with methods: strings the pipeline authored (surrogates, ciphertexts) are excised from the haystack first. A long random ciphertext can contain a short accepted value by coincidence, and refusing a correct export over that would be the worst kind of false alarm, one that looks exactly like a leak. Excising them cannot hide a real leak, because a surrogate is a counter and a ciphertext is indistinguishable from random without the key.

---

8. The export report

A second artifact per export (?part=report on the same signed link):

• the document id, kind and source checksum; the artifact's checksum, MIME type and extension;
• counts removed by category, with the style and method each was applied with;
• how many suggestions were rejected or never decided, and therefore remain in the file;
• lookedFor: the preset, its categories, and whether it narrowed the search (with a note that anything outside it was never proposed);
• for a message, each attachment's disposition: redacted, carried-through (a format Anonify cannot read, nothing inside it redacted, and said so) or removed, keyed by MIME part path, never by filename.

It carries counts, never content, and that is a check, not a promise: assertReportOmitsValues runs before the report is stored and throws ReportLeakError if any free string contains an accepted value. pnpm smoke also reads the delivered report bytes. See Security Model §6.

---

9. Download

GET /api/documents/:id/download?token=…[&part=report]

• The token is HMAC-signed, lives 5 minutes, and names the document, the artifact and the owner. It is necessary but not sufficient: the route also re-derives your session and re-checks ownership.
• The artifact is decrypted a chunk at a time and hashed as it streams, and its last piece is held back until the SHA-256 matches the checksum recorded at export. A mismatch breaks the download off short, so you never hold a complete file that failed its check.
• content-disposition: attachment with <base>-redacted.<ext> or <base>-redaction-report.json, correct for names outside Latin-1 (1.10.0).

---

10. PDF specifics

• A page with accepted redactions is rendered to pixels at 2×, the boxes are burned in, and the page is rebuilt from that image. The characters are gone because the text objects are gone.
• A page without redactions is copied through and keeps its vector text.
• Boxes come from the font's own character widths and reach below the baseline (padded 1.5 pt). This is the 1.5.0 fix for GHSA-h3hv-rpqw-7p84; PDFs analysed before 1.5.0 are covered a whole text run at a time, so re-upload them for tight boxes.
• The trade: a redacted page loses its text layer. Preserving it through content-stream surgery is #40.

---

See also

• Batches and Mailboxes · Redaction Engine · Supported Formats · Security Model · API Reference