Batches and Mailboxes

Batches and Mailboxes

Several files at once become a batch: one upload, one review pass, decisions carried between documents, and one download at the end. A message with attachments and a mailbox become batches on their own.

The part that matters is not the upload. It is that a decision made once is not made again on the next file. "This recurring name is a colleague, not a subject" is answered where it came up and carried to the others, including the ones still being analysed when it was answered.

Canonical sources: docs/pipelines.md (EML, MBOX) · docs/workflow.md §5 · docs/data-model.md §3–7 · code: lib/documents/batches.ts, admission.ts, batch-exports.ts, mbox/, eml/attachments.ts, lib/redaction/batch-download.ts, batch-layout.ts, lib/workflows/export-batch.ts.

---

1. A batch owns decisions, not processing

Each document keeps its own run, its own quota accounting, its own failure and its own expiry. One document failing leaves the rest exactly where they were. The batch is only what makes a decision taken in one document apply to the others.

• A batch is not a discount. Every file is charged exactly what it would cost on its own. When the allowance runs out partway, each file gets its own verdict: accepted files proceed, and refused ones are named with a retry time (a batch that silently starts six of eight documents is one the reviewer will finish believing they reviewed eight).
• Size: ANONIFY_BATCH_MAX_FILES, 20 on the demo and 50 self-hosted. The upload panel asks /api/limits for the real number.
• Processing is admitted a few documents at a time per owner (ANONIFY_BATCH_PROCESSING: 3 / 6). The rest show as queued and start on their own. See Processing Workflow §2.
• In the browser, files transfer one at a time, and a file that fails to upload is reported and skipped while the rest continue. A batch is a convenience over separate uploads, not a transaction.

---

2. Carried decisions

A decision taken with Whole batch in the inspector, or a rule at batch scope, becomes a BatchRule. It is materialised as a per-document GlobalRule in every document of the batch, and documents that finish processing later pick it up in carryDecisions at the end of their run.

• It is planned in every document within the pattern budget before anything is written, then written in one transaction. It is never partly applied.
• Switched off, it stays listed and reaches nothing: its copies lose their redactions, and later documents skip it.
• Removing it undoes every redaction it produced, everywhere. A rule that quietly redacts in files you have not opened has to be visible and removable, which is why the batch page lists carried decisions with their reach.
• Too broad for a newly processed document → that document fails as rule-too-broad and still opens for review.

Global rules (every upload, not just this batch) are carried the same way. See Search and Rules.

---

3. A message with attachments

Since 1.0.0 (#74), before which attachments were carried through unredacted.

• Each attachment in a supported format becomes a first-class child document: sealed under its own key, with its own run, quota, review and export. Provenance is parentDocumentId plus the MIME part path (0.3), not the filename. Filenames collide, get redacted, and only the part path is unique and stable. The database enforces one child per part, so a retried expansion converges instead of duplicating.
• A message uploaded alone gets a batch created for it. A message already in a batch adds its children to that batch, so decisions span the whole thing.
• Expansion has its own fail-closed limits, separate from the MIME parser's: ANONIFY_EML_EXPANSION_MAX_CHILDREN (20), …_MAX_EXPANDED_BYTES (64MB), …_MAX_ATTACHMENT_BYTES (25MB), …_MAX_DEPTH (3, a message in a message). Going over refuses the whole message as too-complex.
• On export, each redacted attachment is substituted back into the message and checked by checksum against the child's verified artifact, because a replacement that landed one part over would produce a message containing none of the accepted values and the wrong file.
• Each attachment's disposition is in the report: redacted, carried-through (a format Anonify cannot read: carried with nothing inside it redacted, stated per attachment), or removed.

---

4. Mailboxes (MBOX)

Since 1.1.0 (#88). An .mbox is not a document, it is hundreds of them. It expands into one document per message, each indistinguishable from the same .eml uploaded from a desktop.

Splitting, which is the whole difficulty

A From line is treated as a seam only when the line before it is blank, the line has the shape of a real separator, and the bytes behind it parse as headers. Failing any of those merges rather than splits: an over-merged mailbox is visibly wrong, while a fractured one is two plausible documents each missing half of the other. A From line inside a message body does not fracture the message. Splitting is streamed (a 16 KiB lookahead), so a large mailbox never has to fit in memory (since 1.6.0).

The container's own ending

The mailbox is the only pure container in the format register (extractable: false, exportable: false). After expansion it ends at the terminal status expanded. It did not fail, and it is not ready either, because there is no model to open and no artifact to export. The workspace shows an ExpandedNotice offering Download redacted mailbox.

Limits and quota

Limit	Demo	Self-hosted	Bounds
ANONIFY_MBOX_MAX_MESSAGES	200	1000	documents one mailbox may produce (floored at the batch cap)
ANONIFY_MBOX_MAX_TOTAL_BYTES	32MB	64MB	content across every message
ANONIFY_MBOX_MAX_MESSAGE_BYTES	12MB	25MB	the largest single message
ANONIFY_MBOX_MAX_DEPTH	2	2	a mailbox reached through a mailbox

A mailbox is charged nothing of its own, because it is never extracted. Each message costs exactly what the same .eml would have, charged idempotently, and a message the allowance does not cover is present, named and explicitly skipped.

Verified by conservation

Every other pipeline asks "is the value gone from the artifact?". A mailbox's failure mode is arithmetic: a splitter that quietly drops the messages it does not recognise produces a batch that looks normal and is missing exactly the unusual ones, which are the ones worth reviewing. So splitting is checked three ways: every byte is inside a message or a separator line, every message still parses under both MIME parsers, and no message kept a From line it should have lost.

What is not claimed

• mboxo vs mboxrd cannot be told apart from the bytes. >From is unquoted as mboxrd, and a wrong guess costs one > in quoted text, never a missed value.
• A whole message pasted into a body after a blank line, headers and all, is indistinguishable from a real seam and will split.
• Content-Length framing (mboxcl) is a stranger's claim and is not trusted.

---

5. Batch export

Exporting a batch is a durable run (lib/workflows/export-batch.ts), because it takes minutes: longer than a request may live and longer than anyone will sit in front of a modal.

Before it starts

BatchExportSetup asks two things:

1. What happens to the values in each file. One method per file, not variants per file: the contract tokenized, the invoices masked, the spreadsheet encrypted, in one run. batchMethodFor() resolves each document's method (its own entry, else the run's default, else mask), and the category table still wins, so a file marked "tokenize" still gets its government IDs, card and bank numbers, API keys and faces masked. (Variants per file would be 48 exports for 12 files × 4 variants, with colliding archive names and a report that double-counts.)
2. What to download (§6).

While it runs

• One step per document, in waves of ANONIFY_BATCH_EXPORTING (2 demo / 4 self-hosted), each through the same verification gate as a single export, charged the export allowance a single export would be. Paced retries, up to 4 per document.
• The progress row is written under SELECT … FOR UPDATE, with totals recomputed from per-document states. Before that existed, the export had to run one document at a time, because two documents finishing together could erase each other's result.
• BatchDownloadButton, on the documents page, the batch page and the workspace header, carries the progress itself: "Exporting 3 of 8" with a fill bar, then "Archive ready". Closing the modal, reloading, or opening the batch on another device reads the same row. The run lives on the server.
• Progress arrives as SSE snapshots (/export/stream, indexed resume), falling back to reading the row every 5 s. The stream carries filenames, states and counts, never a category or a pattern.
• Stop: the document in progress finishes and is kept, everything exported before the stop stays downloadable, and the rest is marked "not reached".

Per-document outcomes

exported (with a count removed), or skipped with a named reason: had not finished processing · failed verification and was withheld · hit the export allowance · did not fit in the archive · could not be exported · not reached.

---

6. The download (since 1.10.0)

GET /api/batches/:id/download?token=…&output=original|processed|both[&document=<id>][&part=report]

Choice	What comes back
Original format (default)	One file per upload, in its own format. A mailbox is rebuilt as a mailbox. A message comes back carrying its redacted enclosures (its own export already is that). A file uploaded alone is its redacted self. One upload is served bare (e.g. inbox-redacted.mbox, application/mbox); several are zipped.
Processed files	Every document's own output, in folders that mirror where it came from: inbox-redacted/0001/message-redacted.eml, inbox-redacted/0001/attachments/0.2-redacted.pdf, with each report under reports/ at the same path.
Both	The two, under original/ and processed/, in one zip.

Folders are named by message number and MIME part path, never by subject or filename. Both are regularly the personal data being removed, and a folder name ends up on a disk and in a zip listing. The upload's own name is the only name in the tree, and you chose that one. The tree is a pure function of parentDocumentId and sourcePartPath, so the pass that decides what goes in and the pass that writes it cannot disagree.

The rebuilt mailbox

Nothing is redacted during download. A rebuilt mailbox is its messages' verified exports, in mailbox order, each re-hashed as it streams. lib/documents/mbox/rebuild.ts writes only what a mailbox has and a message does not:

• The From separator line, which is personal data in the source (the envelope sender and arrival time), is never copied. The rebuild is never even given it. The sender is always MAILER-DAEMON, and the date is the redacted message's own Date: header, or the placeholder Thu Jan 1 00:00:00 1970 when it is missing or would carry an accepted value.
• >From quoting, reapplied as mboxrd.
• The blank line ending each message, in the message's own line ending.

Then the mailbox is verified as its own export: split again by the same scanner that split the upload, and held to what went in. It must have the same number of messages, each equal byte for byte to its export, and no accepted value (from anywhere in the batch) on any separator line outside the fixed text. A mailbox that fails is withheld whole, and the report's containers[].failure names the check (artifact-mismatch, count-mismatch, message-mismatch, separator-leak) and the message it failed at.

A message left out is named, never dropped. One that failed, was refused at expansion, was never exported or failed its own verification appears in the batch report ("3 of 4 messages; 1 left out: message 3 (had not finished processing)") and in containers[].messages.leftOut. Nobody counts nine hundred messages by hand.

The batch report

Every zip carries batch-report.json: which upload each file went into (documents[].container, by id and part-path chain, never a name), each rebuilt mailbox's included and left-out messages, skipped documents and why, and which documents have a vault. A download that is a single bare file offers the report beside it (?part=report). That request runs the same verification, so it costs what the download does, minus sending the files.

Streaming, and the ceiling

Two passes, holding nothing: pass 1 reads every artifact through a hash (and builds each mailbox once through its verifier) and decides what goes in; pass 2 sends, checking again on the way. A 150 MiB ceiling bounds one request's work, and what does not fit is named as skipped. The browser fetches the archive with a streaming reader and saves it automatically, and the button stays in case the auto-download is swallowed.

Vaults in a batch download

A tokenized or encrypted document's vault goes beside the file, never inside it: <name>-vault.json, or inbox-vaults/0001-vault.json for a message of a rebuilt mailbox. A download carrying a vault is therefore always a zip, even for one upload, and the report warns in plain words to separate the vault from the file before sharing either. See Export and Anonymisation §5.

Fixed in 1.10.0

• Batches of more than 50 documents (any mailbox of more than 50 messages) were exported, downloaded and listed as only their first 50, and the workspace lost its place in the batch from document 51 on.
• Downloads of files named outside Latin-1 (收件箱.mbox) failed; they now save under their real names.

---

See also

• Processing Workflow · Export and Anonymisation · Search and Rules · Supported Formats · API Reference