Bulk Download

Bulk document download

POST /api/bulk-downloads zips many documents into one download. One id is a direct presigned download, never a job. Two or more start a queue-worked job the caller polls until it is ready, or stops with DELETE /api/bulk-downloads/{id}.

Request

POST /api/bulk-downloads

{ "documentIds": ["<id>", "<id>", "..."] }

documentIds must be a non-empty array of non-empty strings. Any other body key is a 400. Ids the caller cannot see are dropped by the read ACL, so the job's documentCount may be lower than the number asked for.

Response: exactly one id

200, same shape as GET /api/documents/{id}/download, tagged single: true. No job, no queue.

{ "url": "...", "expiresIn": 300, "fileName": "...", "displayName": "...", "single": true }

Response: two or more ids

202

{
  "id": "6f1c4b9e-0d2a-4d9d-9c3e-1f5f9a2b7c40",
  "status": "queued",
  "documentCount": 42,
  "estimatedPartCount": 1,
  "statusUrl": "/api/bulk-downloads/6f1c4b9e-0d2a-4d9d-9c3e-1f5f9a2b7c40"
}

estimatedPartCount is a guess made from the sizes on the document rows. The count of parts the job produces is partCount on the status body — see Part splitting below.

Error responses

Status	When
400	Unknown body key, malformed documentIds, more documents than the caller's cap, or the selection is over BULK_MAX_TOTAL_BYTES
404	None of the requested documents are visible to this caller
429	This requester already has BULK_MAX_PENDING jobs queued or running, or has started BULK_MAX_PER_DAY jobs in the last 24 hours
503	Bulk download is off in this environment (BULK_DOWNLOADS_QUEUE unset), or the job was recorded but could not be queued (marked failed immediately rather than left queued forever)

Polling

GET /api/bulk-downloads/{id}

{
  "id": "6f1c4b9e-0d2a-4d9d-9c3e-1f5f9a2b7c40",
  "status": "ready",
  "documentCount": 42,
  "partCount": 1,
  "partsReady": 1,
  "includedCount": 41,
  "errorCount": 1,
  "errors": ["<id>\t<name>\tnot available"],
  "bytes": 104857600,
  "parts": [
    { "n": 1, "url": "https://...", "bytes": 104857600, "count": 41, "expiresIn": 300, "fileName": "EPIC documents - Site C Clean Energy.zip" }
  ]
}

parts is present only when status is ready, and partCount is 0 until then — it counts the parts the worker built. errors on the status body is capped at 100 entries; the full list ships as errors.txt inside the last zip part.

The job id is the capability. Anyone holding it can poll an anonymous job. A job made by an authenticated caller answers only that caller — a mismatch is 404, never 403, so a foreign or unknown id looks the same either way.

Cancelling

DELETE /api/bulk-downloads/{id} → 204, no body. Same capability rule as polling: whoever may poll a job may cancel it, and anything else is 404.

A job still queued, running or failed becomes cancelled, and the worker deletes the zip parts it had already written. failed is included because the queue redelivers such a job and the worker re-runs it on purpose — otherwise the build would resume up to a visibility timeout after the cancel. The in-flight slot goes back if the job was still holding it. A job that already reached ready, expired or cancelled is left as it is and answered 204 all the same, so a client that cancels twice, or cancels a job that finished a second earlier, needs no special case.

The status flip is a conditional patch on the job row: whichever writer takes the row out of a live status owns the outcome. That is what stops a cancel arriving as the worker finishes from overwriting ready, or from freeing the requester's slot a second time — the loser writes nothing and releases nothing.

The worker checks the row for a cancellation between parts and every 20 documents inside a part, so a cancel takes effect within a few files rather than at the end of a 2 GiB part. When it sees one it closes the archive it had open, finishes the upload rather than abandoning the stream, deletes every part it wrote for the job, logs [bulk] job cancelled <id> and stops. It does not retry and does not poison the message. A worker that died before it got that far leaves its parts to the cleanup timer, which sweeps cancelled jobs on the same schedule as ready ones.

Job lifecycle

queued → running → ready, failed or cancelled. A ready or failed job becomes expired when the cleanup timer deletes its zip parts.

• queued: row saved, message on the queue.
• running: worker dequeued the message and is streaming files. A running job whose worker has not reported in over the queue's visibility timeout (host.json extensions.queues.visibilityTimeout, 1 hour) reads as failed on the next poll, even though the row itself is not patched by the poll. A worker that later dequeues the redelivered message reclaims that same stale row (a conditional replace keyed on the row's etag) and runs it again from scratch; whichever instance loses the race gets a conditional-write conflict and walks away.
• ready: every part is built and uploaded. parts[] holds one presigned URL per part.
• failed: the worker threw, or the job sat on the queue past BULK_MAX_JOB_AGE_MS (default 2 hours) and was refused without running — its access snapshot is that stale too. A thrown error is retried up to maxDequeueCount (3) before it lands on the poison queue; a retry re-runs a failed job from scratch on purpose, so a transient object-store outage is self-healing. An expired job is not retried.
• cancelled: the caller stopped the job with DELETE. Terminal — nothing writes over it, the cleanup timer included: a swept cancelled row keeps its status and only loses the parts it named, which is also what takes it out of the sweep's query. The worker deletes those parts when it stops, and the sweep deletes whatever a worker that died first left behind. The row itself goes when its TTL runs out.
• expired: the cleanup timer deleted the zip parts past retention. The job row survives past this point so a poll answers expired rather than a 404 that reads as "never existed".

A single missing or unpublished document never fails the job: it is recorded in errors[] and errors.txt, and the rest of the zip still builds. The job itself fails if the object store streams more bytes than BULK_MAX_TOTAL_BYTES allows — the upfront check against fileSize only estimates; the running byte count from the archive is what the worker enforces while it builds.

Cosmos TTL (defaultTtl: -1, per-item ttl) deletes the job row itself after BULK_JOB_TTL_DAYS (default 30), independent of the cleanup timer that removes the zip objects after BULK_ZIP_RETENTION_DAYS (default 7).

Caps

Setting	Default	Meaning
BULK_MAX_DOCUMENTS	2500	Most documents one authenticated job may ask for. Over it, 400.
BULK_ANON_MAX_DOCUMENTS	100	Same cap for an anonymous caller.
BULK_MAX_BYTES	2147483648 (2 GiB)	Ceiling per zip PART. A job over it is split into numbered parts, not refused. A single file over the cap becomes its own part.
BULK_MAX_TOTAL_BYTES	21474836480 (20 GiB)	Ceiling for the whole job. The one limit that refuses (400) rather than splits.
BULK_FETCH_AHEAD	3	Objects the worker opens ahead of the entry it is writing. Higher overlaps more round trips and holds more sockets and buffers open on one instance at a time.
BULK_FETCH_CONCURRENCY	1	Parked objects fetched in parallel. 1 leaves the feature off.
BULK_FETCH_BUFFER_BYTES	268435456 (256 MiB)	Total bytes held in memory for parked objects.
BULK_MAX_PENDING	3	Concurrent jobs (queued or running) per requester. No app rate limiter exists and APIM Consumption cannot key-limit an anonymous caller, so this is the abuse boundary for the feature.
BULK_MAX_PER_DAY	20	Jobs one requester may start in a rolling 24 hours. BULK_MAX_PENDING alone bounds concurrency only — a caller who waits for each job to finish before starting the next could otherwise start an unlimited number.
BULK_MAX_JOB_AGE_MS	7200000 (2 hours)	How long a job may sit on the queue before the worker refuses it outright, marking it failed without running. The job's access snapshot is a copy of the caller's roles at submit time, and this is the ceiling on how stale that copy may be when it is acted on.
BULK_ZIP_RETENTION_DAYS	7	Days a ready, failed or cancelled job's zip parts stay in object storage before the cleanup timer deletes them.
BULK_JOB_TTL_DAYS	30	Days the Cosmos job row lives (ttl), always longer than the zip retention.
BULK_DOWNLOADS_QUEUE	unset	Storage queue name. Empty means the feature is off: no worker registers, POST answers 503.
BULK_CLEANUP_SCHEDULE	unset	NCRONTAB for the cleanup timer. Empty AND BULK_DOWNLOADS_QUEUE set defaults the schedule to 0 30 3 * * *; empty with no queue set registers no timer.

All thirteen are src/config.js app settings, plumbed through azure/modules/api-function-flex.bicep. Each is parsed with a helper that throws at process start on a value that is not a whole non-negative number, so a typo in an app setting fails the deploy rather than silently removing the limit.

A request takes an in-flight slot when it clears the quota check and gives it back exactly once: on a refusal, on a server error, when the job is ready, when it is cancelled, or when its last delivery fails. A failure with a retry left keeps the slot, because that job is still going to run.

slotReleasedAt on the job row is what stops a job freeing a slot twice: the worker, a cancel and the cleanup sweep each write that stamp under a condition only one of them can satisfy, and only the writer that lands it touches the counter.

Zip layout

• Zip file name: EPIC documents - <project name>.zip when every document in the job comes from one project, EPIC documents - <N> projects.zip when it spans several. A job built as more than one part adds (part <n> of <partCount>) before the extension. The name is baked into the presigned URL as its content-disposition; the storage key is unrelated to it. A job whose single project has no name this caller may see is named EPIC documents.zip.

• One folder per project inside the zip, named from the project's name. A document whose project cannot be resolved goes in unknown-project/.

• File name: the document row goes through the same field redactor as any other document read first, so a name withheld from this caller cannot leak into the zip as a file name. From what survives that: documentFileName, falling back to displayName, falling back to the document id if both were empty or fully redacted. Cleaned of path separators and control characters, capped at 150 characters, and given an extension from fileExt if it has none.

• A name collision inside the same folder (case-insensitive — Windows and macOS cannot extract two entries differing only in case into one folder) gets (2), (3), and so on before the extension.

• Zips are streamed with zip-stream (STORE, uncompressed) straight into the multipart upload — nothing buffers a whole zip in memory.

• errors.txt is written into the LAST part only, one line per skipped or damaged document: <documentId>\t<name>\t<reason>. Reasons are a fixed vocabulary, never the object store's own message:

  Reason	Meaning
  not available	Document row not found for this manifest read: unpublished since submit, deleted, or never visible to this caller — one reason, so the message does not itself disclose which.
  no object key	Document row found but has no stored file (s3Key empty).
  unavailable	The object could not be opened at all.
  truncated	The object opened but the stream failed partway through. The entry is still written to the zip with whatever bytes arrived — it is not dropped, only excluded from includedCount.

Part splitting

The worker packs parts while it streams, in the order the caller submitted the ids. It opens up to BULK_FETCH_AHEAD objects ahead of the entry it is writing, so the next documents' round trips to object storage overlap the current one, while the entries themselves are still written one at a time in the submitted order. A stream that resets while it waits its turn is opened once more before it counts as that document's error. Before adding the next document it starts a new part if the current one has already written BULK_MAX_BYTES, or if that document's recorded fileSize would carry it over. A document bigger than the cap gets a part of its own rather than a refusal. Documents with no recorded size pack together like any other: the bytes the archive has written, not the sizes on the rows, are what close a part. errors.txt goes into the last part built.

estimatedPartCount on the 202 comes from a size-only pack (packParts in src/jobs/pack-parts.js), which counts a missing fileSize as nothing, so it is an estimate and named as one. partCount on the status body is what the worker actually built.

Access rules

• The controller snapshots the caller's resolved access (resolveAccess(req)) onto the job row at submit time. The worker re-runs the document read with that same snapshot rather than trusting the submitted id list, so a document unpublished between submit and run is dropped and listed in errors.txt. Sealed-compartment documents are excluded the same way a normal read excludes them — the worker never passes a compartment and never uses systemAccess().
• Roles in the snapshot are frozen at submit time on purpose — that is the selection the caller asked for. Credentials are different: before running, the worker re-checks each credential in the snapshot against the live registry and drops any that have since been revoked or expired. The re-check covers every party the request loaded grants for — the caller's subject, their API key id, and each realm group they belong to, stored on the job row as parties — so a grant held by a group survives it. This only narrows the snapshot, never widens it.
• An anonymous job is pure bearer: whoever holds the job id can poll it. An authenticated job binds to the caller's key id, subject, or username at submit time; anyone else polling it gets 404.
• The per-requester quota (BULK_MAX_PENDING, BULK_MAX_PER_DAY) keys on the same identity: an API key id or signed-in subject when present, otherwise the caller's IP address (src/utils/caller-ip.js). Behind APIM the last X-Forwarded-For hop is APIM itself, which would make every public caller share one quota bucket, so the app trusts one of two hops instead, each proven by a header only the platform that appended it could have set.
  • Front Door first. eagle-edge (profile eagle-edge-<env>) overwrites X-Edge-Secret on every request it forwards, before APIM ever sees it. When that header matches EDGE_SECRET, the key is X-Azure-SocketIP — the peer Front Door itself accepted the connection from — not X-Azure-ClientIP, which Front Door derives from the caller's own X-Forwarded-For and so any browser can pick. See Environment-Reality-and-Operational-Gotchas § Front Door edge secret for where the value lives.
  • APIM otherwise. APIM asserts the address it saw as X-Client-Ip on every request, trusted only when the request also carries the gateway secret APIM sets alongside it (azure/modules/apim.bicep).
  • Either way, once an asserted address (X-Azure-SocketIP or X-Client-Ip) is itself a proxy WE run (TRUSTED_PROXY_IPS, comma-separated addresses or CIDRs — the OpenShift rproxy that fronts eagle-public), every visitor behind it would still share one bucket. The key is then the first X-Forwarded-For hop to the left of that address that is not a trusted proxy — anchored at the asserted hop's own position in the chain, not the rightmost entry, because Front Door and APIM both append to the same header and the browser's hop sits to the left of both. Everything further left is caller-supplied and never trusted.
  • When the asserted address is not a trusted proxy, that address is the key directly.
  • A request that reaches the app directly, bypassing both Front Door and APIM, cannot present either secret and falls back to the last X-Forwarded-For hop the platform itself appended.
  • To measure the address to add to TRUSTED_PROXY_IPS in an environment, read the egress address of the OpenShift rproxy pod for that environment (eagle-api's outbound IP), or the pod network CIDR if the egress address is not stable. In prod this is still unmeasured — TRUSTED_PROXY_IPS is empty there, so the Front Door path changes nothing until it is.

Infrastructure

Resource	Name
Queue	bulk-downloads (test), set by the bulkDownloadsQueue bicep param — empty in prod today
Poison queue	bulk-downloads-poison, created automatically alongside the main queue
Cosmos container	bulkDownloads, partition key /id. defaultTtl: -1; indexed paths /status, /requesterKey, /finishedAt only — the id lists, access snapshot, errors and parts arrays are excluded from the index
Poison alert	demi-bulk-download-failed-<env>, deployed when deployBulkDownloadPoisonAlert is true. Severity 2 (warning), fires on any AppTraces row containing [bulk] job failed in a 15-minute window

host.json queue settings for this trigger: messageEncoding: "none", batchSize: 1, newBatchThreshold: 0, maxDequeueCount: 3, visibilityTimeout: "01:00:00". batchSize: 1 means one zip builds per Function App instance at a time; Flex scales horizontally (up to 20 instances) instead of running several zips per instance, which bounds the multipart upload buffer each instance holds in memory.

The worker logs [bulk] job failed <id>: ... only on the delivery that will actually poison the message (the last of maxDequeueCount attempts) — an earlier attempt logs [bulk] attempt failed instead, so the alert above does not fire for a job the queue is still going to retry.

The job id is masked out of the per-request access log (GET|POST /api/bulk-downloads/<id>) because it is a bearer capability: logging it would let anyone who can read the logs poll or, once ready, download the job.

Client guidance

• Poll GET /api/bulk-downloads/{id} every few seconds until status is ready, failed, expired or cancelled.
• Trigger each part's download with same-tab navigation (location.assign(url), or a synthetic <a> click without target=_blank) — not window.open, which needs a user gesture Safari blocks on an async response. The presigned URL carries Content-Disposition: attachment already, so this downloads rather than navigates.
• More than one part fires the browser's "allow multiple downloads" prompt once; fire the URLs sequentially.
• Keep the job id in localStorage and resume polling on page reload — closing the tab does not cancel the job, only the caller's ability to notice it finished. DELETE is the only cancel.

See also

• Bulk-Download-Operations — enable in an environment, poison-queue runbook
• Environment-Reality-and-Operational-Gotchas — queue trigger encoding, s3Key visibility
• Attribute-Level-Access — how resolveAccess and the read ACL work
• Architecture — where the object store and Cosmos containers fit overall