Bulk Download Operations

Bulk download: operations

How to turn the feature on in an environment and how to work the poison queue. Field reference and behaviour: Bulk-Download.

Enable in an environment

1. Set bulkDownloadsQueue in the environment's .bicepparam file (azure/main.test.bicepparam or azure/main.prod.bicepparam). Empty means the feature is off and no worker registers. Leaving bulkCleanupSchedule empty while bulkDownloadsQueue is set still deploys a cleanup timer, on a default schedule of 0 30 3 * * *; set bulkCleanupSchedule explicitly only to move that hour.
2. Optionally set deployBulkDownloadPoisonAlert = true in the same file to deploy the poison-queue alert.
3. Run scripts/deploy-infra.sh <env> --live by hand (see Operating-DEMI-in-Azure for the what-if/live convention). Infrastructure has no CI path.
4. Confirm BULK_DOWNLOADS_QUEUE and BULK_CLEANUP_SCHEDULE app settings landed on demi-api-fc-<env> and that POST /api/bulk-downloads with two ids answers 202 rather than 503.

Set TRUSTED_PROXY_IPS per environment

The anonymous bulk-download quota keys on caller IP (see Bulk-Download Access rules). Without this setting, every eagle-public visitor in an environment behind the OpenShift rproxy shares one quota bucket.

1. Find the egress address (or pod network CIDR) of the OpenShift rproxy that fronts eagle-public in the target environment — the address APIM will see asserted as X-Client-Ip for every eagle-public request.
2. Set TRUSTED_PROXY_IPS in the environment's .bicepparam file (azure/main.test.bicepparam or azure/main.prod.bicepparam) to that address. A comma-separated list accepts several addresses; an entry with a / is read as a CIDR (10.0.0.0/8 for an in-cluster router hop), anything else as an exact match.
3. Run scripts/deploy-infra.sh <env> --live by hand.
4. Confirm TRUSTED_PROXY_IPS landed on demi-api-fc-<env> app settings.
5. From an eagle-public browser session, start two anonymous bulk-download jobs and confirm they count against the same quota bucket, not two — the earlier one should count toward BULK_MAX_PENDING.

Leaving TRUSTED_PROXY_IPS empty is not a security hole: the quota key falls back to the rproxy's own address for every caller, which only makes the shared bucket wider, never removes the check.

Natural sort rollout

sortBy=displayName on GET /api/documents orders by displayNameSort, a zero-padded copy of displayName computed by src/helpers/natural-sort.js (see Search-Index-Reference). Rolling the field into an environment that predates it takes four steps, in order:

1. Index PUT. Apply the documents index definition carrying the displayNameSort field (azure/search/indexes/documents.json) with src/scripts/apply-search-definitions.js. The script widens a live index in place; it never refuses this change because it only adds a field.

2. Data source PUT. Confirm the demi-documents-ds data source SELECT already returns displayNameSort — src/scripts/put-search-datasources.js if not. Skipping this step leaves the new index field permanently null: the indexer can only forward what the data source selects.

3. Backfill. Run node src/scripts/backfill-display-name-sort.js (dry run by default, --live to write) to stamp displayNameSort onto every document row written before the four write sites started computing it. Cosmos is private-endpoint-only and keyless, so a live run executes on the devbox (demi-devbox-<env>) via demi-run — see the root README.md "Running anything against the database".

   The walk covers every partition the document rows name, plus one bucket for the rows whose projectId is JSON null or absent. Those address no partition key and cannot be enumerated with the rest, so they are read cross-partition and patched on their own null key. They are counted in expected too: scanned short of expected prints INCOMPLETE and exits 1, so a run that stops reading the bucket cannot pass by measuring itself against a smaller corpus.

4. Indexer catch-up. No manual indexer run is needed: a Patch operation moves a document row's _ts, so the documents-indexer's PT5M schedule re-pulls every patched row on its own.

Doing this out of order leaves sortBy=displayName ordering on a null field for whichever rows have not yet passed through the step already done — the app never errors, it just orders wrong.

Chunk parent-field rollout

typeId, milestoneId, projectPhaseId and documentAuthorTypeId on the chunks index are a copy of the same-named columns on each chunk's parent document, alongside a fifth field, parentFieldsVersion, that is not a copy but a stamp of the writer's CHUNK_PARENT_FIELDS_VERSION constant (CHUNK_PARENT_FIELDS and CHUNK_PARENT_FIELDS_VERSION in src/repositories/chunks.js — see Search-Index-Reference "Chunk filters carry a copy of their parent document's fields"). Rolling them into an environment that predates them takes the same four steps as the natural-sort rollout above, in the same order, and the order matters more here:

0. src/scripts/backfill-document-list-ids.js --live must already have run to completion. The chunk backfill copies what the document row holds. A row seeded before that script kept only the List label and its four ids are null, so a chunk backfill started first stamps nulls, reports success, and checkpoints the partitions it did it to. Check that the List-id backfill's summary reads planned: 0 before starting step 1. That script also re-stamps the chunks of every document whose ids it moves, so on a fresh corpus most of step 3 is already done by the time it finishes; step 3 is still the sweep that proves it.

Steps 1 and 2 are one command, scripts/demi-devbox.sh apply --env <env> --only chunks --datasources demi-chunks-ds --no-wait, run before the app that writes the fields is deployed. apply pulls and writes whatever the devbox checkout holds. It PUTs demi-chunks-ds first, behind its own prompt, then dry-runs and asks again; answering N there leaves the data source rewritten with no index PUT and no reset, so run apply again to finish. After the deploy, scripts/demi-devbox.sh watch --env <env> --datasources demi-chunks-ds waits for the indexer run; it gives up after INDEXER_TIMEOUT_LONG (80 minutes) with a warning, so run it again until the run ends.

1. Index PUT (apply-search-definitions.js --live --only chunks). Applies the chunks index definition carrying the four copied fields plus parentFieldsVersion (azure/search/indexes/chunks.json). Widens the live index in place; existing chunks keep their other values and read null on these five fields until backfilled.

2. Data source PUT and indexer reset (put-search-datasources.js, then reset-and-run-indexers.js chunks-indexer; the data-source script resets nothing). Writes demi-chunks-ds so its SELECT projects the four Cosmos columns and parentFieldsVersion. --datasources forces the reset; without it apply resets only a data source its dry run reports as DIFFERS. The data-source PUT must land before the backfill, not after — skipping it does not error anywhere. The indexer still re-pulls every row the backfill patches, under the old SELECT with no such column, and its _ts high-water mark advances past them anyway. The values are then stranded in Cosmos, reachable only by resetting the indexer over the full ~1.1M chunk rows, and every PUT and indexer run in between answered 200/green.

   Both PUTs land before the app deploy. On test the merge to main is the staging deploy, and the PR gate ("Search Schema Change") probes the test indexes with the branch's definitions and wants Search-Schema: applied test in the PR body. So on test, before the merge, put the devbox checkout on the PR branch (git remote set-branches --add origin <branch>, fetch it, git checkout -B <branch> --track origin/<branch> through demi-devbox.sh run), run apply, and put the checkout back on main after the merge; azure/search/README.md has the commands. On prod the deploy's schema probe checks the live index against the tag's definitions, so apply --env prod runs after the release merge, while the devbox's main has nothing past the tag under azure/search/, and before the azure-deploy-prod.yaml dispatch.

   Chunks written between the PUT and the deploy come from the previous app and miss the fields in Cosmos; step 3 stamps them, because their parentFieldsVersion is below the current one. If the app went out first, the chunks it wrote before the PUT are right in Cosmos and null in the index; the reset re-pulls every row and repairs them. A data-source PUT without the reset leaves them null, and the ordinary backfill never repairs them: it skips a chunk whose Cosmos copy already matches its document, so no _ts moves. Then run step 3 once with --force, which re-patches every chunk regardless of whether it already agrees.

3. Backfill. Run npm run db:backfill-chunk-parent-fields (dry run by default; add --live to write) to stamp the four fields and parentFieldsVersion onto every chunk whose stamped version is below the current CHUNK_PARENT_FIELDS_VERSION, including chunks that were stamped before this field existed. src/scripts/backfill-chunk-parent-fields.js's header carries the full flag set (--project, --pending, --force, --concurrency, --state).

   The walk covers every partition the document rows name, plus one bucket for the documents whose projectId is JSON null. Those address no partition key, and left out of the walk their chunks stay unstamped for good — which holds the unstamped count above zero and keeps the facets withheld on every unscoped search. The bucket is read cross-partition, counted in expected and checkpointed like any other partition.

   A live run also deletes index rows whose Cosmos chunk is gone. Removing a chunk from Cosmos leaves its index row behind — the indexer detects no deletions — and no stamp can ever reach that row, which is the same permanent withhold one project at a time. Every candidate is confirmed by a point read against Cosmos before it is deleted, and the scan is capped per partition: immediately after a patch the whole partition still reads as unstamped in the index, so a partition reporting more unstamped rows than the run examined is finished by the next pass once chunks-indexer has caught up. A dry run counts the orphans and deletes none. Because of this step a live run needs the search grant as well as the Cosmos one.

   The run is resumable: each finished partition is checkpointed to the state file, and the first run also records the corpus total as it stood then. A resume is measured against that recorded total, not against a fresh count, so documents ingested while the backfill was running — which are stamped at ingest anyway — do not make a complete run report INCOMPLETE. When the two totals differ the run logs both. A state file carrying no such total is compared against a fresh count.

   The summary line counts nullProject documents in the null bucket; pendingCleared, pendingConflicts and pendingMissed pending flags; and orphansFound, orphansPurged, orphansFailed and orphansSkipped index rows. orphansFailed above zero means those rows are still searchable and still unstamped. orphansSkipped counts partitions whose orphan scan could not run at all — search unconfigured, or an index with no parentFieldsVersion column — and the line carries the reason; those rows are unknown, not absent.

   Every stamp is guarded on an instant — the document's own parentFieldsPendingAt where it carries one, the walk's start instant otherwise — so skippedNewer counts chunk operations a newer walk had already stamped and this run deliberately left alone, while pendingLeftRaised counts flagged documents a --pending run never reached because their row moved partition between the flag listing and the walk; the run names those ids and exits 1.

   A live run exits 1 on any of: a rejected chunk write, a partition the walk never reached, a pending flag whose row the clear could not find (pendingMissed), a flagged document the run never reached (pendingLeftRaised), or an orphan purge that did not finish (orphansSkipped or orphansFailed above zero). A dry run is not held to the purge, since it was never going to delete anything. pendingConflicts is not a failure: the write that moved the row owns that flag now.

   Cosmos is private-endpoint-only and keyless, so a live run executes on the devbox (demi-devbox-<env>) via demi-run, as in the object-store backfill below.

4. Indexer catch-up. No manual run needed: the backfill's Cosmos PATCH moves each chunk's _ts, so chunks-indexer's PT5M schedule re-pulls every patched row within one cycle.

5. Queue the ongoing re-stamp. Set chunkRestampQueue in the environment's .bicepparam and deploy the infrastructure, which creates the queue, its -poison pair and the CHUNK_RESTAMP_QUEUE app setting. Infrastructure before code: with the setting present but no queue behind it, the send fails and the re-stamp is lost with only a log line. Code before the setting does not patch inline. The write logs one warning, flags the document row parentFieldsPending: true, skips the patch and returns, so a misconfigured environment cannot hold up an eagle-api push; the repair for that window is npm run db:backfill-chunk-parent-fields -- --live --pending, which walks exactly the flagged documents. A seed-nosql --live or backfill-document-list-ids --live run can raise the same flag when its own re-stamp fails; it prints the affected document ids, and the same backfill-chunk-parent-fields.js --live --pending run clears them. EVERY live run clears the flag, not only that one: a --live --project <id> repair and a full-corpus run compare the same chunks against the same document row, so they hold the same proof, and each clears the flag on the documents it verified — patched or already agreeing. A document whose chunks the run could not write keeps its flag; that is decided per document, so one lost chunk no longer holds up the clear for the rest of its partition. The clear is conditional on parentFieldsPendingAt, the token the flag was raised with, rather than on the row's _etag: an unrelated write — an extraction patch, a display-name push — moves the etag and would make the clear lose forever, while the token only moves when the flag is raised again. A document flagged again while the repair walked it keeps its flag for that newer write to clear and is counted as pendingConflicts; a flag whose row the clear cannot find at all is counted as pendingMissed and exits 1. The same flag is set when the enqueue itself fails, and the nightly reconcile reports the count as parentFieldsPending=N on the line demi-reconcile-drift-<env> alerts on — so a lost re-stamp is a number somebody sees rather than a log line nobody reads. Set CHUNK_RESTAMP_INLINE=1 to get the inline walk back — that is the func start path on a box with no storage account, not something to set on a deployed app. az deployment group create does both halves of the setting in one template, so the order that needs watching is deploy-infra then deploy-api.

Shipping the app out of order is survivable in both directions, and the response says which one you are in. Before step 1, the API reads the live index rather than the packaged definition, finds no typeId to filter on, and names the key in meta[0].dropped.filter — it never emits a clause the service would 400. Between step 1 and the end of step 3, the filter IS emitted and matches only the chunks stamped so far, so the response carries meta[0].degraded.reasons: ['chunk-parent-fields-unstamped'], which lets a UI say "filter partially applied" instead of showing a short answer as a complete one. That count is taken under the caller's own ACL clause and the request's project scope, so running the backfill project by project marks the projects still behind rather than every page in the corpus. The completeness count behind that signal reads parentFieldsVersion, not whether the copied List refs are null: a document can legitimately carry four null List refs and still be fully stamped, so counting nulls can never reach zero and would leave the degraded banner permanently on. Counting chunks whose parentFieldsVersion is below CHUNK_PARENT_FIELDS_VERSION or absent is what lets the count reach zero once the backfill finishes. Both signals clear on their own; the live-schema answer is cached for ten minutes, so nothing needs restarting after the index PUT.

Poison-queue runbook

Two queues run on this storage account — bulk-downloads below, and chunk-restamp, whose messages are JSON ({"documentId","projectId"}) rather than a bare id. A message in chunk-restamp-poison means one document's chunks still carry their old typeId/milestoneId/projectPhaseId/documentAuthorTypeId; the repair is npm run db:backfill-chunk-parent-fields -- --live --project <projectId>, which re-stamps that project, clears the pending flag on the documents it verified, and needs no queue. Re-sending the message works too, since the handler re-reads the document row.

Both get three deliveries, but they wait for them differently. visibilityTimeout in host.json is one hour, sized for a zip that can run for half an hour, and it applies to every queue in the app. A chunk re-stamp that hits a transient Cosmos error does not wait that long: the handler re-sends the message itself with a 30-second delay, doubling per attempt, carries the attempt number in the body, and returns so the original message completes. Only the last delivery throws, which is what moves it to chunk-restamp-poison, and only that delivery logs [chunk restamp] job failed with the document id — the line demi-chunk-restamp-failed-<env> matches. Earlier attempts log at warn and raise nothing.

A job's worker retries up to maxDequeueCount (3) times before the storage queue moves the message to bulk-downloads-poison. Each retry re-runs the job from queued state, so most transient failures (an object-store blip) self-heal before reaching the poison queue.

1. The demi-bulk-download-failed-<env> alert fires when it does reach it — the job row is already failed, the requester already sees that status, and the job's in-flight quota slot is back.
2. Read the poison message body: it is the bare job id (messageEncoding: "none" — do not base64 decode it).
3. Read the job row (GET /api/bulk-downloads/{id} or a Cosmos point read on bulkDownloads) for error, the message the worker logged before it last threw.
4. To retry: either copy the message text back onto the bulk-downloads queue (same job id, same row, resumes from queued), or have the requester POST /api/bulk-downloads again with the same documentIds to get a fresh job id. A requeued job holds no quota slot, and the slotReleasedAt on its row stops the retry releasing one a second time.
5. Delete the poison message once retried, so it does not re-fire the alert on the next poison-queue sweep.

Stage 0 measurements

Measured 2026-09-01 against the test corpus, ahead of shipping the caps in Bulk-Download:

• s3Key missing on 0 of 61,587 documents — no backfill needed before turning the feature on.
• Largest single document: 1,258,271,336 bytes (1.17 GiB), under the 2 GiB part cap, so no document in the corpus forces a single-file part on its own.
• Largest projects by total size: project 91 (589 docs, 8.18 GiB), 126 (592 docs, 8.17 GiB), 45 (624 docs, 6.74 GiB), 159 (1,261 docs, 5.62 GiB), 272 (2,158 docs, 5.14 GiB) — each packs into 5 parts or fewer and stays under the 20 GiB total-job cap, so the caps needed no adjustment.

Measured 2026-09-08 on test: raising BULK_FETCH_CONCURRENCY from 1 to 4 made no difference to worker throughput, about 6 MB/s either way. The serial multipart zip upload sets the ceiling, not the object fetches. Details in the repo file docs/bulk-download-performance.md.

Backfill missing objects on test

The test bucket (zdspnb, no key prefix) is missing objects for rows a backfill job copied from prod: a --since 1970-01-01 dry run on 2026-09-24 found 413 of 61,683 test documents with no object in zdspnb, so the download 404s and a bulk-download part fails on them. --since skips any row with no datePosted, so those rows are not in the 413 count. The gap is accepted and not backfilled: zdspnb is eagle-api TEST's own bucket, not a copy target to fill on a schedule. Every missing s3Key exists in the prod bucket ozwdez on the same object-store host under the identical key with no prefix, so src/scripts/backfill-objects.js can copy them across if that is decided.

The script needs the MinIO settings, which are not in the devbox's app settings. Put them in /home/demi/.minio-env (MINIO_HOST, MINIO_PORT, MINIO_USE_SSL, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, MINIO_BUCKET_NAME, MINIO_KEY_PREFIX, MINIO_REGION) and source it into the run.

Add SOURCE_MINIO_ACCESS_KEY and SOURCE_MINIO_SECRET_KEY to the same file — plus SOURCE_MINIO_HOST, SOURCE_MINIO_PORT and SOURCE_MINIO_USE_SSL when the source bucket sits on another endpoint — when the target credentials are not authorized on the source bucket: with a source key set the script reads each object under that second credential and streams it into the target bucket, instead of asking the object store for a server-side copy. The probe and the progress line report which of the two runs, transfer=stream or transfer=copy.

1. Probe first. One document's key is stat'd in both buckets. Exit 0 means the credentials can read the prod bucket; exit 1 means they cannot, and nothing else in this section will work.

   sudo -u demi /usr/local/bin/demi-run 'cd /opt/eagle-demi && set -a && . /home/demi/.minio-env && set +a && node src/scripts/backfill-objects.js --probe'
   

2. Dry run. Without --live the script stats every key and counts the gap, writing nothing. planned is how many objects a live run would copy. Add --limit 200 for a quick trial.

   sudo -u demi /usr/local/bin/demi-run 'cd /opt/eagle-demi && set -a && . /home/demi/.minio-env && set +a && node src/scripts/backfill-objects.js --since 1970-01-01'
   

3. Live run, detached with a log file, since a full scan outlasts a single run-command invocation. This writes into zdspnb, eagle-api TEST's own bucket, so get a decision to fill the gap before running it live:

   sudo -u demi /usr/local/bin/demi-run 'cd /opt/eagle-demi && set -a && . /home/demi/.minio-env && set +a && nohup node src/scripts/backfill-objects.js --since 1970-01-01 --live > /home/demi/backfill-objects.log 2>&1 &'
   

4. Read the summary line at the end of the log: scanned, noKey, present, planned, copied, missingInSource, failed. missingInSource is a key absent from prod too, which no re-run fixes. failed is anything else, with the first 20 keys logged; the exit code is non-zero when it is not 0.

Options: --source-bucket (default ozwdez), --since (default: every document), --concurrency (default 4), --limit. Re-running is safe: each key is stat'd in the target before it is copied, so a killed run picks up where it stopped.

See also

• Bulk-Download — request/response shapes, caps, job lifecycle, access rules
• Operating-DEMI-in-Azure — what-if/live deploy convention