Streaming IO

Streaming IO

How Anonify reads a document without holding it, and why that decides what hardware it needs. This shipped in two parts: 1.6.0 (#102) built the primitives (the chunked envelope, ranged and streamed driver reads, a memory budget), and 1.7.0 (#129) used them everywhere a whole object was still being held.

Canonical source, with every measured figure: docs/streaming.md. The figures can be reproduced with pnpm tsx scripts/streaming-figures.ts.

---

1. Before and after

Before, every stage fetched the sealed object, decrypted it whole, parsed it whole and wrote the result whole. Each step was a full copy, and several copies were alive at once. For a 50 MiB file that is a few hundred megabytes per document, multiplied by how many documents process at once.

After, each stage reads the part it needs, when it needs it:

flowchart LR
    S[("Sealed object<br/>1 MiB AEAD chunks")]
    S -->|"streamed, chunk by chunk"| X1["txt · csv · tsv · eml<br/>read front to back"]
    S -->|"ranged reads"| X2["xlsx · docx · pptx · pdf<br/>read by their own index"]
    X1 --> M[("Sealed model<br/>+ page index")]
    X2 --> M
    M -->|"one page = one range"| B["Browser<br/>outline + ≤ 24 pages"]
    M -->|"pages streamed, text only"| A["Analysis · rules · search"]
    M -->|"only pages with an accepted redaction"| E["Export"]
    S -->|"streamed through SHA-256"| D["Downloads"]

Loading

---

2. The budget

chunkBytes × maxInFlightChunks × processingConcurrency() ≤ memoryBudget

processingConcurrency() is the existing per-owner admission limit, so this composes with it instead of adding a second limiter. ANONIFY_ENCRYPTION_CHUNK_SIZE (default 1MB) and ANONIFY_STREAM_MEMORY_BUDGET (default 8 chunks per processing document on the demo profile, 16 self-hosted) configure it. A budget that cannot give every concurrent document two chunks is refused at startup.

Modelled: six concurrent 50 MiB decks need about 900 MiB on the whole-file path and about 102 MiB on the ranged path. On a 2 GB serverless function, that is the difference between room to spare and an out-of-memory kill that the workflow retries straight into the same wall.

---

3. The normalized model, a page at a time

The model is stored exactly as before, as one JSON document (byte-for-byte JSON.stringify(model)), so anything that reads it whole still works. Beside it, Document.normalizedIndex records where each page's JSON sits: { version, size, open, close, pages: [number, start, end][] }. It holds offsets only, never content.

• A page is one ranged read of the chunks covering it, each authenticated before use. A page that parses but is not the page asked for is refused.
• The outline is everything before [ and after ]: kind, metadata, sheets, regions, page numbers.
• Every page in order is one streamed read, cut at the index's offsets as it passes.
• The index is written in the same update as the model's key, so a retried extraction cannot pair one model's key with another model's offsets.
• A model written before 1.7.0 has no index and is read whole, exactly as before.

What reads it, and what each holds

Reader	Reads	Holds
Editor (GET /content?view=outline, then ?page=N)	outline, the visible page, its neighbours, rail thumbnails within 400 px of view	the outline and ≤ 24 pages (least recently used let go)
Analysis (loadTextModel)	every page's text, never spans, blocks or sections	one page and a chunk
Rules and search (findOccurrencesIn)	every page, a page at a time	the matches
Export (pagesReadByExport)	only pages carrying an accepted redaction (plus page 1 for images)	those pages

Measured: a 100,000-line text file becomes 1,992 pages and a 17.4 MB model. Showing the first page used to transfer 17,382,639 bytes. It now transfers 132 + 9,014 bytes.

---

4. Archives read from their index

Word documents, decks and workbooks are zip archives, and a zip's table of contents (the central directory) is at the end. The old parsers inflated every entry before anyone looked at one.

lib/documents/ooxml/zip.ts reads the end record and the central directory with two ranged reads, then inflates only the entries it is asked for, 256 KiB at a time, and stops reading when the reader stops.

It refuses to guess

Zip is a format where parsers famously disagree, and the ranged reader exists to produce exactly what the whole-file parser would. Where two whole-file parsers could differ, it throws ZipFallback and the workflow extracts the file whole, as it always did. Triggers: bytes after the end record, zip64, multiple disks, bytes in front of the first entry, a non-ASCII, duplicate, encrypted or oddly compressed entry, or a local name or inflated size that disagrees with the central directory. An archive a real office suite wrote passes every check.

A chunk cache for small parts

Fifty small parts inside one megabyte would otherwise fetch and authenticate that megabyte fifty times. cachedRangeSource widens reads to whole chunks and keeps the last four.

Word and PowerPoint: XML parts only

Extraction reads XML and relationship parts and nothing else. Pictures, fonts and embeddings are inflated and let go (so a corrupt one still fails where it always did), and are left as names that throw if anything reads them. An extractor that starts depending on one fails loudly instead of finding it empty. Measured: a 16.8 MB deck with two 8 MiB photos used to hold 16,861,305 bytes of parts. It now holds 84,089.

---

5. Workbooks, one worksheet at a time

exceljs ships a streaming reader, but it was unusable here:

exceljs WorkbookReader	What Anonify needs
spills worksheets to a plaintext temp file when shared strings come last	decrypted content never on disk
does not read hidden rows and columns	hidden data reported to the reviewer, because that is where redactions get missed
drops merged cells and shared-formula text; orders sheets by zip position	the same model the whole-file path produces

lib/documents/xlsx/stream.ts drives exceljs's own whole-file machinery over the same parts in the same order, with one difference: a worksheet is parsed just before it is written and released right after. The output is JSON.stringify(extractXlsx(...).document) character for character (tests/xlsx-stream.test.ts). The peak is the largest sheet plus the shared strings, roughly 1/n of the old peak for n equal sheets.

---

6. Email, scanned forward

The old MIME parser needed the whole message as one string, and attachments are nearly always most of a message. Measured: a 34.4 MB message carrying one PDF was held twice.

lib/documents/eml/scan.ts reads the message once, a line at a time, with the delimiter of every open multipart on a stack. It keeps header blocks and text parts, and lets attachment bodies go past, recording their byte ranges so whatever needs one reads it back from storage. It now keeps 10 bytes between pieces of that 34.4 MB message.

Byte for byte, quirks included. The exporter edits the original message by these offsets, so "close enough" would mean a leak or a corrupted message. The scanner reproduces the old recursive parser exactly, including its idiosyncrasies (an enclosing multipart's delimiter always wins, where a part ends relative to CRLF, when headers end, which empty parts exist). The old parser is kept verbatim as a test oracle. tests/eml-scan.test.ts compares the two over 3,000 generated and mutated messages, fed in pieces from 1 byte to 1 MiB, under two sets of limits, and checks that the corpus reaches every quirk more than five times. Targeted mutants of the scanner all fail it. parseEml now runs on the scanner too, so the exporter and the extractor cannot drift apart.

---

7. PDFs over ranged reads

A PDF's cross-reference table is at the end. pdf.js is handed a PDFDataRangeTransport backed by the chunk cache, with auto-fetch and streaming off, so it asks only for the ranges it parses. Two things the transport handles that pdf.js does not:

• A failed read. pdf.js's transport cannot report one, so the work is raced against the failure (guard) and rejects with the read that failed, instead of hanging.
• Transferred buffers. pdf.js may detach what it is handed, and those bytes are a chunk the cache still holds, so every range is copied first.

Measured: rendering page 500 of a 1,000-page PDF for the vision pass used to read 1,029 KB. It now reads 439 KB, and the saving grows with the document.

---

8. Downloads, verified as they stream

A download used to be decrypted whole, hashed, compared, and only then served. Now:

sequenceDiagram
    participant S as Sealed artifact
    participant O as Chunk opener
    participant V as ChecksumVerifier
    participant B as Browser
    S->>O: chunk 1
    O->>V: plaintext 1 (tag verified)
    Note over V: hash; hold 1
    S->>O: chunk 2
    O->>V: plaintext 2
    V->>B: 1
    S->>O: final chunk
    O->>V: plaintext 3
    V->>B: 2
    Note over V: compare SHA-256
    alt matches
        V->>B: 3, response completes
    else mismatch
        V--xB: stream fails; the download breaks off short
    end

Loading

What a recipient holds after a mismatch is a download that broke off, never a complete file that failed its check.

The batch archive needs its verdict before the first byte is sent (a failing artifact is left out and named, not allowed to break the archive), so it reads twice and holds nothing: pass 1 streams every artifact through SHA-256 and decides; pass 2 streams again into an fflate zip, checked again on the way. A rebuilt mailbox is read three times per message (hashed, verified by re-splitting, delivered). A 150 MiB ceiling bounds one request's work, and anything that does not fit is named as skipped. An abandoned download releases its file handle or socket (fixed in 1.7.0).

---

9. What still reads whole, and why

Path	Why
Export reads each source whole	pdf-lib, the OOXML writer and the redactors rebuild a whole artifact, and the verification gate re-opens that artifact. Streaming it would move invariant 4 onto a streamed artifact and belongs in its own reviewed change.
Images	memory is the decoded raster; streaming the file changes nothing
RTF	the parser is a state machine over the whole string; low value
A workbook's sheets in the editor	the grid renders every cell of a sheet, so the outline carries the sheets
A message's text parts	they are the text being reviewed
GET /content with no query	kept for compatibility; the editor no longer calls it

The invariants all still hold: no plaintext leaves a chunk before its tag verifies, objects are write-once, the chunk counter fails closed, MAX_UPLOAD_BYTES has not moved, and every streamed reader produces exactly what its whole-file counterpart would. Where it cannot be sure, it hands the work back to that counterpart.

---

10. Where each claim is checked

Claim	Test
The model's JSON is unchanged, and every page is where the index says	tests/normalized-pages.test.ts
Rules and export reach the same answer from pages as from the whole model	tests/normalized-pages.test.ts
Every zip entry inflates exactly as unzipSync does; ambiguous archives are refused	tests/zip-reader.test.ts
A workbook streamed a sheet at a time serializes identically	tests/xlsx-stream.test.ts
Word and PowerPoint over XML parts only are identical; skipped parts throw	tests/ooxml-stream.test.ts
PDF extraction and rendering by ranges are identical; a failed read fails, not hangs	tests/pdf-ranged.test.ts
The MIME scanner matches the recursive parser across 3,000 messages	tests/eml-scan.test.ts
A mismatched download never completes	tests/streamed-downloads.test.ts
An abandoned read releases its handle	tests/chunked.test.ts
Vercel Blob / S3 honour ranges	pnpm smoke:blob · pnpm smoke:storage (CI)

---

See also

• Storage and Encryption · Processing Workflow · Supported Formats · Configuration