Chunk Parent Field Backfill

Chunk parent-field backfill

This page rolls the chunk parent fields into an environment that predates them, and repairs chunks whose copy drifted. What the fields are: Chunk-Parent-Fields.

When to run it

• Roll out: the chunks index lacks typeId, milestoneId, projectPhaseId, documentAuthorTypeId or parentFieldsVersion. Follow steps 0 to 5.
• Field added: CHUNK_PARENT_FIELDS_VERSION was bumped. Follow steps 1 to 4.
• Repair: the nightly reconcile reports parentFieldsPending=N, or a message sits in chunk-restamp-poison. Run the repairs at the end.

Cosmos is private-endpoint-only and keyless, so a live run executes on the devbox (demi-devbox-<env>) through demi-run; see the root README.md "Running anything against the database". A live run also deletes index rows, so it needs the search grant as well as the Cosmos one.

Roll out the fields

Steps 1 and 2 are one command, run before the app that writes the fields is deployed:

scripts/demi-devbox.sh apply --env <env> --only chunks --datasources demi-chunks-ds --no-wait

apply pulls and writes whatever the devbox checkout holds. It PUTs demi-chunks-ds first behind its own prompt, then dry-runs and asks again. Answering N at the second prompt leaves the data source rewritten with no index PUT and no reset, so run apply again to finish.

0. Finish the List-id backfill. Run src/scripts/backfill-document-list-ids.js --live to completion and check that its summary reads planned: 0.
   • The chunk backfill copies what the document row holds. A row seeded before that script holds only the List label, with its four ids null.
   • A chunk backfill started first stamps nulls, reports success and checkpoints those partitions.
   • The List-id script re-stamps the chunks of every document whose ids it moves, so step 3 then mostly confirms.
1. Index PUT. apply-search-definitions.js --live --only chunks adds the four copied fields and parentFieldsVersion from azure/search/indexes/chunks.json. It widens the live index in place; existing chunks read null on the five fields until backfilled.
2. Data source PUT and indexer reset. put-search-datasources.js writes demi-chunks-ds so its SELECT projects the four columns and parentFieldsVersion. Then reset-and-run-indexers.js chunks-indexer resets the indexer; the data-source script resets nothing. --datasources forces the reset. Without it, apply resets only a data source its dry run reports as DIFFERS.
3. Backfill. Run npm run db:backfill-chunk-parent-fields, a dry run by default; add --live to write. It stamps the four fields and parentFieldsVersion onto every chunk whose version is below CHUNK_PARENT_FIELDS_VERSION.
4. Indexer catch-up. No manual run is needed. The backfill's Cosmos PATCH moves each chunk's _ts, so the PT5M schedule of chunks-indexer re-pulls every patched row within one cycle.
5. Queue the ongoing re-stamp. Set chunkRestampQueue in the environment's .bicepparam (azure/main.test.bicepparam, azure/main.prod.bicepparam) and deploy the infrastructure.
   • The deploy creates the queue, its -poison pair and the CHUNK_RESTAMP_QUEUE app setting.
   • Deploy infrastructure before the code. With the setting present and no queue behind it, the send fails and the re-stamp is lost with only a log line.

Order of the PUTs and the deploy

• The data-source PUT must land before the backfill. Otherwise the indexer re-pulls the patched rows under the old SELECT, which has no such column, and its _ts high-water mark advances past them. The values then sit in Cosmos with no way into the index except a reset over the full chunk corpus, and every step in between answered 200.
• Both PUTs land before the app deploy. On test, the merge to main is the staging deploy, and the PR gate "Search Schema Change" wants Search-Schema: applied test in the PR body.
• So on test, put the devbox checkout on the PR branch before the merge, run apply, and put the checkout back on main after it. The commands are in azure/search/README.md.
• On prod, the deploy's schema probe compares the live index with the tag's definitions. Run apply --env prod after the release merge and before the azure-deploy-prod.yaml dispatch, while the devbox main has nothing past the tag under azure/search/.
• After the app deploy, scripts/demi-devbox.sh watch --env <env> --datasources demi-chunks-ds waits for the indexer run. It gives up with a warning after INDEXER_TIMEOUT_LONG (80 minutes), so rerun it until the run ends.
• Code deployed before the chunkRestampQueue setting does not patch inline. The write logs one warning, flags the document row parentFieldsPending: true and skips the patch, so a misconfigured environment cannot hold up an eagle-api push. The --pending repair below clears that window.
• Do not set CHUNK_RESTAMP_INLINE=1 on a deployed app. It restores the inline walk for func start on a box with no storage account.

Chunks written during the rollout

• Chunks written between the PUT and the app deploy come from the old app and miss the fields in Cosmos. Step 3 stamps them, because their version is below the current one.
• If the app went out first, the chunks it wrote before the PUT are right in Cosmos and null in the index. The reset re-pulls every row and repairs them.
• A data-source PUT without the reset leaves those chunks null. The ordinary backfill skips a chunk whose Cosmos copy already matches its document, so no _ts moves. Run step 3 once with --force, which re-patches every chunk.

Flags

node src/scripts/backfill-chunk-parent-fields.js [--live | --dry-run]
       [--project <id> | --pending] [--force] [--concurrency N] [--max-attempts N]
       [--state ./backfill-chunk-parent-fields.state.json]

Flag	Effect
--live	Writes. Without it, or with --dry-run, the run is a dry run that counts and writes nothing. --live with --dry-run is an error.
--project <id>	Walks one project: the manual repair the poison-queue alert names. It ignores the state file.
--pending	Walks only documents flagged parentFieldsPending. It ignores the state file and cannot combine with --project.
--force	Re-patches every chunk, even one that already agrees with its document.
--concurrency N	Parallel partitions, 1 to 8, default 2.
--max-attempts N	Chunk-patch retry budget, 1 to 20, default 12. A corpus walk on a serverless account stays throttled for minutes, and a batch that runs out of attempts leaves the document part-stamped.
--state <path>	Checkpoint file, default ./backfill-chunk-parent-fields.state.json. A corpus run resumes from it.

What a run covers

• The walk covers every partition the document rows name, plus one bucket for documents whose projectId is JSON null. That bucket is read cross-partition, counted in expected and checkpointed like any other. Left out, its chunks stay unstamped, the unstamped count stays above zero and the facets stay withheld on every unscoped search.
• A live run deletes index rows whose Cosmos chunk is gone, because the indexer detects no deletions and no stamp can reach such a row. Each candidate is confirmed by a Cosmos point read before deletion. A dry run counts orphans and deletes none.
• The orphan scan is capped per partition. Right after a patch the whole partition still reads as unstamped in the index, so the next pass finishes any partition that reported more unstamped rows than the run examined.
• Each finished partition is checkpointed, and the first run records the corpus total. A resume is measured against that total, not a fresh count, so documents ingested during the run cannot make a complete run report INCOMPLETE.
• A state file with no recorded total is compared against a fresh count.
• Every stamp is guarded on an instant: the document's parentFieldsPendingAt where it has one, the walk's start instant otherwise.
• Every live run clears the pending flag on the documents it verified, patched or already agreeing, under any of the three scopes. The clear is conditional on parentFieldsPendingAt, so an unrelated write to the row cannot defeat it.
• A document whose chunks the run could not write keeps its flag. The decision is per document.

Read the summary

The summary line carries these counters.

Counter	Meaning
nullProject	Documents walked in the null-project bucket.
skippedNewer	Chunk operations a newer walk had already stamped; the run left them alone.
pendingCleared	Pending flags cleared.
pendingConflicts	Flags left raised because the document was flagged again during the walk. The newer write owns the flag and clears it. Not a failure.
pendingMissed	Flags whose row the clear could not find.
pendingLeftRaised	Flagged documents a --pending run never reached, because the row moved partition between the flag listing and the walk. The run names the ids.
orphansFound, orphansPurged	Index rows without a Cosmos chunk, and how many were deleted.
orphansFailed	Orphans the service refused to delete. They stay searchable and unstamped.
orphansSkipped	Partitions whose orphan scan could not run, because search is unconfigured or the index has no parentFieldsVersion column. The line carries the reason. Those rows are unknown, not absent.

A live run exits 1 on any of these:

• a rejected chunk write;
• a partition the walk never reached (INCOMPLETE);
• pendingMissed or pendingLeftRaised above zero;
• orphansSkipped or orphansFailed above zero.

A dry run is not held to the orphan purge.

Repair

The --pending repair covers a lost or skipped re-stamp. A seed-nosql --live or backfill-document-list-ids --live run can raise the same flag when its own re-stamp fails, and prints the affected document ids.

npm run db:backfill-chunk-parent-fields -- --live --pending

The project repair covers a message in chunk-restamp-poison. It re-stamps one project, clears the flag on the documents it verified and needs no queue. Re-sending the message works too, because the handler re-reads the document row.

npm run db:backfill-chunk-parent-fields -- --live --project <projectId>

What callers see during the rollout

• Before step 1, the API reads the live index, finds no typeId to filter on and names the key in meta[0].dropped.filter. It never emits a clause the service would answer with a 400.
• From step 1 to the end of step 3, the filter is emitted and matches only the chunks stamped so far. The response carries meta[0].degraded.reasons: ['chunk-parent-fields-unstamped'], so a UI can say "filter partially applied".
• That count uses the caller's own ACL clause and the request's project scope. Backfilling project by project marks only the projects still behind.
• The completeness count reads parentFieldsVersion (below the constant or absent), not null List refs, because a fully stamped document can hold four null refs.
• Both signals clear on their own. The live-schema answer is cached for ten minutes, so nothing needs a restart after the index PUT.