Chunk Parent Fields

Chunk parent fields

The chunks index repeats five fields of each chunk's parent document row. A chunk filter then runs on the chunk alone, with no join across indexes.

Rolling the fields into an environment: Chunk-Parent-Field-Backfill. Index definitions: Search-Index-Reference. How queries use the copies: Search-Query-Construction.

Fields

Field	Type	Flags	Wire key
typeId	Edm.String	filterable, facetable	and[type]
milestoneId	Edm.String	filterable, facetable	and[milestone]
projectPhaseId	Edm.String	filterable, facetable	and[projectPhase]
documentAuthorTypeId	Edm.String	filterable, facetable	and[documentAuthorType]
projectId	Edm.String	filterable, not facetable	none
parentFieldsVersion	Edm.Int32	filterable, retrievable, not facetable, not sortable	none

• Each copy is a straight copy of the same-named column on the parent document row.
• CHUNK_PARENT_FIELDS in src/repositories/chunks.js is the one list every writer reads; CHUNK_PARENT_LIST_REFS is its four List-ref subset.
• DOCUMENT_FACETS in src/search/eagle-query.js maps the four wire keys onto the List refs for both dataset=Document and dataset=DocumentChunk. The same key means the same field on either dataset.
• projectId scopes access on every chunk read. Chunks are partitioned by documentId, so a document pushed under a new project leaves its chunks on the old scope until they are re-stamped.
• parentFieldsVersion is a stamp, not a copy. parentFieldsOf() writes it on every stamp from CHUNK_PARENT_FIELDS_VERSION.
• parentStampedAt orders writers and answers no query. chunks.json does not declare it and demi-chunks-ds does not project it, so it costs no index PUT.

The version exists because four null List refs cannot tell "stamped, parent has no List refs" from "never stamped". Many prod documents have no List refs, so a null count never reaches zero.

• chunkMatchesParent() treats a chunk whose version is absent or below the constant as a mismatch, whatever its values hold.
• The search-side completeness probe counts parentFieldsVersion lt CHUNK_PARENT_FIELDS_VERSION or parentFieldsVersion eq null.
• projectId is not versioned: every ingest path writes it, so a stale value fails the plain value comparison.

Keeping the copy current

The copy is a snapshot, so a writer must re-stamp it when the parent changes.

Event	What stamps the chunks
Ingest	The chunk build copies the values off the document it is chunking.
Ingest end	The document is re-read and compared with the snapshot; any change enqueues a re-stamp.
Parent write that moves a value (eagle-api push, staff PUT /documents/:id)	A re-stamp job on the CHUNK_RESTAMP_QUEUE storage queue.
Missed or lost re-stamp	backfill-chunk-parent-fields.js --pending --live.

• The ingest-end check catches a re-type that lands halfway through a long stream. The NDJSON path refreshes its values every 25 batches (5,000 chunks), which narrows the window but does not close it.
• The stream latches every change it sees, so a value that moves and moves back is raised too. A failed ingest-end check logs a line and never returns a 500, because the chunks have already landed.
• The re-stamp is one bulk PATCH per 100 chunks with 429 backoff, riding the indexer's _ts high-water mark. A 6,000-chunk document takes minutes, longer than eagle-api's 10-second push timeout, so it runs on the queue (src/jobs/restamp-chunks.js).
• The read[] propagation stays on the request, because it is a visibility boundary and a queue would leave a window where the index disagrees with Cosmos.
• The message carries the document and project ids only. The handler re-reads the row, so a redelivered message stamps the later state and the handler can throw on a failed patch.
• With no CHUNK_RESTAMP_QUEUE, the write logs one warning and skips the re-stamp. CHUNK_RESTAMP_INLINE=1 restores the inline walk for func start on a box with no storage account.
• Failure handling for the queue: Bulk-Download-Operations "Poison-queue runbook".

The pending flag

A document row owes a re-stamp from the moment the change is durable. The flag is an outbox.

• Every write that moves a value sets parentFieldsPending: true and parentFieldsPendingAt on the same upsert that changes the fields, before anything is enqueued.
• The re-stamp is dispatched after that write lands, and a landed walk is the only thing that clears the flag. A failed enqueue only logs, because the row already carries the flag.
• Without the flag, a lost re-stamp is invisible: the request answers 200 and a chunk filter matches fewer rows than it should.
• The nightly reconcile counts flagged rows as parentFieldsPending=N on the line demi-reconcile-drift-<env> alerts on.
• backfill-chunk-parent-fields.js --pending --live walks exactly those documents, re-stamps them and clears the flag.
• An eagle-api push replaces the whole row and carries the flag across.
• The documents container's indexing policy in azure/modules/cosmos-nosql.bicep includes /parentFieldsPending/?. CI never deploys infra; an index-only change can use scripts/apply-cosmos-index.sh, see Operating-DEMI-in-Azure.

Clearing the flag

• clearPending() in src/jobs/restamp-chunks.js clears it after the queue handler's walk and after the inline walk.
• The clear is conditional on the parentFieldsPendingAt token, not on the row's _etag. An unrelated write, such as an extraction patch, moves the etag and would refuse the clear for a document whose chunks are already correct.
• documents.freshPendingAt() mints each token strictly later than the one on the row, so two raises in the same millisecond cannot share one.
• A refused clear returns conflict: a newer parent-field change owns the flag and its own re-stamp clears it. The handler logs one info line and completes the message.
• A vanished row returns missing, logged at warn, and also completes. A row raised before tokens existed carries none and clears unconditionally.
• setParentFieldsPending() in src/repositories/documents.js patches the row directly, so it needs the row's partition key, projectId. A document with no project holds a JSON-null projectId. The writer reads such a row cross-partition and passes its own projectId through; an empty string is a real partition and is used as is.
• A row purged between that read and the write is left unpatched. restamp-chunks.js reads body.projectId ?? undefined, so only null or undefined falls back to a cross-partition read.

Guarded document writes

A write that moves a value rebuilds the whole row from the row it read, and a Cosmos upsert replaces the item. Both paths in src/controllers/nosql/document.js therefore carry IfMatch on the _etag they read.

• A 412 means another writer landed first. The write re-reads the row, rebuilds the change on top of it, mints the token off the stored one and retries, up to three times.
• The guard keeps a flag another walk cleared from being written back. It also keeps a value another writer moved from being reverted with no flag raised.
• After three losses the staff route answers 409 and the push answers 503, the only answer eagle-api's push client retries.

Chunk ordering token

• An ingest stamps parentStampedAt on every chunk it writes: the row's parentFieldsPendingAt where the row owes a re-stamp, the ingest's start instant otherwise.
• A walk patches a chunk only if the chunk's token is older than the walk's. An unstamped chunk is older than every walk.
• A chunk stamped by a newer walk keeps its values and counts as skippedNewer, an outcome and not a shortfall. The token stops a re-stamp that began before an ingest from overwriting the chunks that ingest wrote.

Add a field to the copy

Adding a field touches five places:

1. CHUNK_PARENT_FIELDS in src/repositories/chunks.js.
2. The field definition in azure/search/indexes/chunks.json.
3. The SELECT in azure/search/datasources/demi-chunks-ds.json.
4. The vis catalog entry in src/vis/catalog/chunks.js.
5. CHUNK_PARENT_FIELDS_VERSION, bumped.

Tests pin the list against chunks.json, demi-chunks-ds.json and the vis catalog (test/azure/search-datasource-columns.test.js), so leaving out one of the first four fails a test.

• The bump is what puts every stamped chunk back in scope. backfill-chunk-parent-fields.js re-stamps a chunk only when its stamped version is below the constant.
• Skip the bump and the backfill still runs, exits 0 and reports success, but re-stamps nothing, so the new field stays null on every row.
• A bump with no change to what parentFieldsOf() stamps has nothing to backfill.
• Skip the bump only for a field every ingest path already writes, as with projectId: the value comparison catches a wrong value, and a bump would force a corpus-wide re-stamp.