Bulk Document Download

ADR: Bulk Document Download

Status: Proposed

Context

A public user filters a project's documents on eagle-public and wants one archive of exactly what the filter matched. Three constraints shape the answer.

location /api in eao-nginx carries proxy_cache globalcache. A synchronous zip response would be a cache candidate, and a status-polling endpoint under that block would serve a stale queued forever. nginx also buffers proxied responses by default and spills them to disk past proxy_max_temp_file_size (1 GB), so a sync design needs proxy_buffering off, proxy_cache off, and a location block of its own.

A static object in the store gets HTTP Range and resume for free. A generated stream does not. At the sizes below, a download that dies at 90 % and cannot resume is the normal case.

A sync stream pins an eagle-api worker for the client's whole download. Prod runs 3–6 pods at 1 CPU / 4 Gi. Fifty slow clients on a 10 GB archive is the entire API tier.

Corpus, measured 2026-08-18 against prod:

Public documents	61,605
Total bytes, extrapolated from 1,978 sampled	~624 GB
Mean / median document	10.1 MB / 0.8 MB
p90 / p99 / max document	19.6 MB / 200 MB / 726 MB
Projects	358
Largest project by count	Jumbo Glacier Resort, 2,488 docs
Two projects measured whole	833 docs = 3.48 GB; 401 docs = 1.34 GB

The worst realistic single request, "all documents in the biggest project", lands near 10 GB across 2,500 files. Nothing approaches ZIP64's 65,534-entry threshold, but the 4 GB size threshold is crossed routinely.

Dataverse caps its sync path at 100 MB and hands the rest to a separate zipper. Google Takeout builds asynchronously, emails a link, and expires it after about seven days.

Decision

Every Download All is an async job. There is no synchronous zip stream.

An Async Jobs worker builds the archive, writes it to object storage as a real object, and hands the user a time-limited presigned URL. Small requests take the same path as large ones — one code path, about 15 seconds end to end for a handful of files. Both precedents above cap their sync tier; we skip it rather than maintain two.

Angular (documents tab / search)
  │  POST /api/public/jobs  { type, projectId, filters }
  ▼
eagle-api  ── pre-flight: run the same public query, count files, sum internalSize
  │          ├─ over caps? 400 with the numbers, nothing queued
  │          ├─ archive already built and still fresh? 200 { url }
  │          └─ else agenda.now('batch-document-download', …) → 202 { jobId }
  ▼
Agenda worker (one pod claims atomically)
  │  per document: GET from object store → yazl entry (STORE) → multipart PUT
  │  updates job.attrs.data.progress
  ▼
exports/<filterHash>.zip in the documents bucket
  │
  ▼
GET /api/public/jobs/:jobId   (polled every 5 s, Cache-Control: no-store)
  → 200 { status: 'completed', url: <presigned GET, 24 h> }

api/helpers/jobQueue.js and api/controllers/job.js already provide the queue, the status shape, the progress field, and multi-pod safety. The JOB_PERMISSIONS map in job.js names batch-document-download as its worked example of adding a type.

Filters, not document IDs

The client posts filter criteria and the worker re-runs the query server-side. /api/public/search caps pageSize, so a browser cannot enumerate a 2,488-document result set to send back. Re-running through Utils.runDataQuery('Document', ['public'], …) applies the publication gate through the same code path as every other public read; a client-supplied ID list would have to be re-checked against exactly that anyway.

Caps

This is an unauthenticated fan-out endpoint. OWASP API4:2023 is explicit that per-request rate limiting is structurally insufficient when one request becomes N internal operations, and that unbounded egress is a cost impact on its own. Dynamically generated archives are uncacheable, so each one is 100 % origin egress.

Caps are enforced before the job is queued, from the pre-flight count and internalSize sum:

Cap	Default	Env
Files per archive	5,000	BULK_MAX_FILES
Bytes per archive	5 GB	BULK_MAX_BYTES
Concurrent jobs per IP	1	BULK_MAX_PER_IP
Worker concurrency	2	—

Over a cap returns 400 naming the actual count and size, with advice to narrow the filter. Never a silent truncation. Rate limiting comes from the existing limit_req and limit_conn zones in eao-nginx. No CAPTCHA: the caps plus the per-IP job limit bound the damage, and the content is public.

Dedup

The object key is exports/<sha256 of normalized filter + projectId>.zip. Before queueing, stat the key: if it exists and is newer than the newest _updatedDate among matching documents, return its presigned URL without queueing. This turns the dominant query, "everything in project X", from repeated origin egress into one object served many times.

Zip settings

STORE, not DEFLATE (compress: false on every entry). EPIC is nearly all PDFs, already compressed internally; a second deflate pass buys 1–5 % for real CPU per byte.

yazl rather than archiver. It selects ZIP64 automatically past 4 GB or 65,534 entries, and it is the only Node option that can report an exact total size before any bytes are written — which needs STORE everywhere plus a known size on every added stream. Both hold here: internalSize is already on every Document.

Entry names are display names, deduplicated with a -1, -2 suffix, because EPIC display names repeat within a project. The first entry is a manifest CSV from the existing api/helpers/export-docs-helper.js.

Memory stays flat: one object streamed in, one entry streamed out, multipart PUT to the store.

Retention

A nightly Agenda job deletes exports/ objects older than seven days. Chosen over an S3 lifecycle rule because the queue already exists and a bucket-policy change on the shared object store needs another team.

Consequences

Work required:

File	Change
eagle-api/api/helpers/jobQueue.js	agenda.define('batch-document-download'); nightly exports/ sweep. Also delete the dead pdf-lib / fs / path / MinioController imports.
eagle-api/api/helpers/minio.js	Presigned GET expiry as a parameter, hardcoded to 5 minutes today; add a putObject stream helper
eagle-api/api/controllers/job.js	Public create and public status, capped to this one type; pre-flight caps; dedup check
eagle-api/api/swagger/swagger.yaml	/public/jobs, /public/jobs/{jobId}
eagle-public documents tab and search components	Button, poll, progress, link
eao-nginx	Nothing, provided the status endpoint sends Cache-Control: no-store

Accepted limits:

• The archive reflects the corpus at build time, not at click time. For a document set that changes a few times a week this is not worth solving.
• No resume mid-build. A worker crash restarts that archive from zero after Agenda's 5-minute lockLifetime. Fine at 10 GB, not at 100 GB.
• No splitting. Over the byte cap is a rejection, not a split archive.
• No email delivery. The user keeps the tab open and polls.
• The presigned URL is a bearer capability for 24 hours. The content is public documents, so this is not a leak.

Unresolved:

• Object store CORS is unverified. It only matters if the browser ever fetches objects directly; this design never does.
• The 5 GB byte cap is borrowed, not measured, and the largest project exceeds it. Business needs to say whether "all of Jumbo Glacier", around 10 GB, must succeed.

References

• Async Jobs — the queue this rides on
• Reverse Proxy Configuration — the /api location block
• yazl — ZIP64 and size precomputation
• Azure async request-reply