Search Index Reference

Search index reference

The three Azure AI Search indexes DEMI queries. Definitions are committed under azure/search/ and applied by src/scripts/apply-search-definitions.js. The JSON in the tree is the authority on which fields are searchable, retrievable and filterable — read it, not the live service. Overview: Search-and-Retrieval.

Indexes, indexers and data sources

Index	Indexer	Data source	Cosmos container
chunks	chunks-indexer	demi-chunks-ds	chunks
documents	documents-indexer	demi-documents-ds	documents
projects	projects-indexer	demi-projects-ds	projects

Routes that read the projects index: GET /search, GET /search/counts, native GET /projects and native GET /counts?kind=projects. Native GET /documents, GET /documents/passages and GET /counts?kind=documents read documents and chunks; see Search-Query-Construction.

Index names come from SEARCH_INDEX, SEARCH_INDEX_DOCUMENTS and SEARCH_INDEX_PROJECTS and default to the names above. Each indexer runs on a PT5M schedule with maxFailedItems: 0 and an _ts high-water mark.

The data source's SELECT decides which Cosmos columns the indexer can see at all, so a field added to an index but not to the live data source stays null with no error anywhere. apply-search-definitions.js never writes a data source; it prints a warning when the live one differs from the committed copy. Data sources are written by src/scripts/put-search-datasources.js.

No data source declares a dataDeletionDetectionPolicy, so deletes are application work — see Search-Query-Construction.

Api-versions

Object	Api-version
indexes, indexers, every query call	2024-07-01
data sources	2024-05-01-preview

A data source carrying a user-assigned identity is rejected on 2024-07-01, which is why put-search-datasources.js pins its own version.

Similarity, analyzers and semantic configuration

All three indexes use BM25 similarity with Azure's default k1 and b, and declare no scoring profile. Searchable text fields use the en.microsoft analyzer.

documents and projects each add one custom analyzer, filename — a pattern tokenizer splitting on [^A-Za-z0-9]+ plus a lowercase filter. On documents it analyzes fileNameTokens, a non-retrievable copy of documentFileName; on projects it analyzes nameTokens, a non-retrievable copy of name. These fields are what make keywords=mine match 2019-mine-plan.pdf and a project named "Red Mountain Mine" — en.microsoft strips stopword terms like mine at both index and query time, so on its own it matches neither.

Only chunks declares a semantic configuration. Its name is derived as <index>-semantic rather than hard-coded, so a renamed index cannot point at a configuration it does not declare. content is retrievable: true because a field named in a semantic configuration must be both searchable and retrievable.

Searchable fields

Index	Searchable fields	Notes
projects	name, displayName, description, proponent, nameTokens	nameTokens uses the filename analyzer and is retrievable: false; the rest en.microsoft. proponentId is filterable, not searchable — see Proponent filter below
documents	displayName, documentFileName, description, fileNameTokens	fileNameTokens uses the filename analyzer and is retrievable: false
documents sort key	displayNameSort	not searchable, sortable: true. Zero-padded copy of displayName (src/helpers/natural-sort.js); sortBy=displayName on GET /api/documents redirects to it (SORT_KEYS in src/search/eagle-query.js) because $orderby on a plain string field is codepoint order and puts "Item 10" ahead of "Item 2"
chunks	content	twelve fields in total; every field is sortable: false

Every name in a searchFields or select list must exist in the index. A stray one is a 400 on every query, not a missing field on one row.

Field metadata comes from the committed index definitions

filterable, sortable and the Edm type are read out of azure/search/indexes/*.json at require time, so a schema change cannot leave a hand-written copy behind. Two consequences: the index PUT must land before (or immediately after) the app deploy, or the app emits $orderby on a field the service does not have; and the deploy package must include azure/search/indexes, which scripts/package-api.py excludes wholesale unless re-included.

Type gates, not filterable and sortable flags

Whether a key can be filtered or sorted on is decided by the field's Edm type (TERM_TYPES, ORDER_TYPES), never by its filterable/sortable flag. projects.centroid is both filterable and sortable and is still a 400 either way, because OData reaches a geography only through geo.distance(...). ORDER_TYPES is one type wider: Edm.DateTimeOffset has no eq case but sorts, and documents.datePosted is eagle-public's default document sort.

Cosmos and index field names differ

The indexer's SELECT aliases stored columns (c.projectState AS status, c.eagleId AS legacyEagleId, c.projectType AS type), so an AI Search hit and a row read straight out of Cosmos carry different names for the same value. Reading index names on the Cosmos branch is worse than a missing value: p.status is undefined there, so a || 'Active' default fires on every row and the registry asserts every project is Active. address is the one column the index cannot emit at all, so the Cosmos branch alone returns location.

searchVis on documents

documents.searchVis is Edm.Int32, filterable only: not retrievable, searchable, sortable or facetable. A native GET /documents with q filters searchVis ge <caller level>, so a keyword cannot count a row whose searched field the caller may not see.

Value: the highest level from 4 to 0 at which all three dials (displayName, documentFileName, description) allow the row, else -1. A dial allows a level when it is absent, or is an integer at or above that level. A dial that is not an integer allows nothing, so the row gets -1. The integer guard keeps one odd value from stopping documents-indexer, which has maxFailedItems: 0.

The data source SELECT in azure/search/datasources/demi-documents-ds.json computes it as nested ternaries AS searchVis. Nothing writes it to Cosmos. A dial change bumps _ts, so the next PT5M indexer pass carries it. Data check: the count of rows with searchVis eq null is 0 once the indexer has caught up.

Proponent filter

projects carries two proponent columns: proponent, the NAME the list renders and sorts on (aliased from Cosmos proponentName), and proponentId, the Organization ObjectId, filterable but not sortable. and[proponent]=<ObjectId> maps onto proponentId (ALIASES.Project in src/search/eagle-query.js); a bare sort on proponent falls back to the name column, since proponentId cannot be sorted.

pcp is a ProjectNotification filter, not a projects index field, and that dataset is Cosmos-served, not index-served — see below.

Chunk filters carry a copy of their parent document's fields

chunks holds typeId, milestoneId, projectPhaseId and documentAuthorTypeId, filterable and facetable, Edm.String, plus projectId, which is filterable but not facetable. Each is a straight copy of the same-named column on the field's parent document row — the Azure AI Search parent-child pattern, where a child index repeats the parent's filter columns instead of the query joining across indexes. CHUNK_PARENT_FIELDS in src/repositories/chunks.js is the one list every writer reads; DOCUMENT_FACETS in src/search/eagle-query.js maps the wire keys and[type], and[milestone], and[projectPhase] and and[documentAuthorType] onto them for both dataset=Document and dataset=DocumentChunk, so the same filter key means the same field on either dataset.

projectId is on that list for a second reason. It is the field every chunk read scopes access on, and chunks are partitioned by documentId, so they do not move when a document is pushed under a new project: without the re-stamp the chunks of a moved document keep answering the old project's scope. It needed no index or data-source change, and no version bump — every ingest path has always written it, so a chunk holding an old project fails the ordinary value comparison and is re-stamped without putting the whole corpus back in scope. DOCUMENT_FACETS maps no wire key onto it; CHUNK_PARENT_LIST_REFS is the facet subset.

parentFieldsVersion (Edm.Int32, filterable, retrievable, not facetable, not sortable) rides alongside the copies but is not itself a copy. chunks.parentFieldsOf() stamps it on every write — the ingest chunk build, the propagation patch (setParentFieldsForDocument), the seed re-stamp, and both backfill scripts — set from the exported constant CHUNK_PARENT_FIELDS_VERSION in src/repositories/chunks.js. It exists because the four copied List refs cannot tell "stamped, and the parent legitimately has no List refs" apart from "never stamped": a document with no List refs at all stamps four honest nulls, and prod holds many such documents, so a completeness check reading "no chunk has all four fields null" can never reach zero even on a fully rolled-out corpus. chunkMatchesParent() treats a chunk whose parentFieldsVersion is absent or below CHUNK_PARENT_FIELDS_VERSION as a mismatch regardless of what the copied values hold, and the search-side completeness probe counts parentFieldsVersion lt CHUNK_PARENT_FIELDS_VERSION or parentFieldsVersion eq null for the same reason.

The copy is a snapshot, not a live join, so it goes stale unless something re-stamps it:

• New chunks get it at ingest, read straight off the document being chunked.

• An ingest re-reads the document at the end and compares it against the snapshot the chunks were stamped from. A long stream is minutes of batches — the NDJSON path refreshes the values it stamps every 25 batches (5,000 chunks), which narrows the window but does not close it — and a re-type landing halfway through is a write whose own re-stamp walks only the chunks that exist when it runs, and then clears the flag. So whatever moved while the stream ran is raised again at ingest end, with a token later than the mid-stream write's, and a re-stamp is enqueued that walks the complete set. The stream latches every change its refresh sees, so a value that moves and moves back is raised too: the two ends agree by then, while the batches in between carry the middle value. It is best-effort: the chunks are written by then, so a failure is a log line, not a 500 that would have the extraction host re-ingest a document that landed.

• An existing document's chunks get re-stamped whenever a write moves one of the four values — an eagle-api push or a staff edit in the DEMI admin panel — the same bulk PATCH setAclForDocument uses for read[], riding the indexer's ordinary _ts high-water mark rather than a search-service write.

• That re-stamp runs on a storage queue (CHUNK_RESTAMP_QUEUE, producer and handler in src/jobs/restamp-chunks.js), not on the write that caused it. The walk is one bulk PATCH per 100 chunks plus 429 backoff, so a 6,000-chunk document takes minutes, while eagle-api's push client aborts at 10 seconds and pushes again — the inline version bought a duplicate push doing the same work twice. The read[] propagation beside it stays on the request, because that one is a visibility boundary and a queue would leave a window where the index disagrees with Cosmos.

• The message carries the document and project ids only, never the field values: the handler re-reads the document row, so a message redelivered after a later edit stamps the later state. That makes redelivery safe, which is why the handler throws on a failed patch and lets the queue retry it rather than swallowing it as the inline version had to.

• With no CHUNK_RESTAMP_QUEUE set the write logs one warning and skips the re-stamp, so a misconfigured environment slows nothing down. CHUNK_RESTAMP_INLINE=1 asks for the inline walk back, for func start on a box with no storage account.

• The flag that makes a lost re-stamp visible is written as an outbox, in this order. Every write that moves one of the values — a staff PUT /documents/:id, an eagle-api push, any other caller of propagateParentFields — puts parentFieldsPending: true and parentFieldsPendingAt on the same upsert that changes the fields, before anything is enqueued. The re-stamp is dispatched after that write has landed, and a landed patch is the only thing that clears the flag. So a document owes a re-stamp from the moment the change is durable, and stays on the reconcile line until the walk actually happens.

  The order is what matters. Raising the flag only when the enqueue failed or was skipped covered the loss anyone would notice and missed the quiet one: a message that was accepted and then failed every attempt into the poison queue never came back to raise anything, so the reconcile and --pending were blind to exactly the documents that needed them. An enqueue that fails now only logs, because the row already carries the flag.

  Without the flag a lost re-stamp is invisible: the request answers 200, the document row is right, and the only symptom is a chunk filter matching fewer rows than it should. The nightly reconcile counts the flagged rows and puts parentFieldsPending=N on the line demi-reconcile-drift-<env> alerts on, and backfill-chunk-parent-fields.js --pending --live walks exactly those documents, re-stamps them and clears the flag. The flag is carried across an eagle-api push, which otherwise replaces the whole row.

  Deploy note: the documents container's indexing policy in azure/modules/cosmos-nosql.bicep includes /parentFieldsPending/? (PR #458, 2026-09-29). CI never deploys infra, so after merge run scripts/deploy-infra.sh test, then scripts/deploy-infra.sh prod, by hand. Adding the path starts a background index rebuild on documents; reads are unaffected while it runs. An index-only change can use scripts/apply-cosmos-index.sh instead; see Operating-DEMI-in-Azure.

• setParentFieldsPending() (src/repositories/documents.js) is how the flag is cleared, and how the backfill repairs a row: it patches the document row directly, so it needs the row's partition key, projectId. A document not linked to any project holds a JSON-null projectId, and String(null) is the string 'null' — a partition no document lives in, so a patch addressed that way 404s, which left those rows flagged forever. The writer now resolves the partition key rather than stringifying it: given a project id it addresses that partition directly (an empty string is a real partition and is used as-is); given none, it reads the row cross-partition first and hands Cosmos the row's own projectId, null included. A row purged between that read and the write is left unpatched. The queue side had the mirror fault — restamp-chunks.js read body.projectId || undefined, which turned a genuine empty-string partition into "no project" and forced a cross-partition read; it reads body.projectId ?? undefined now, so only a real null or undefined falls back.

• Clearing is one function, clearPending() in src/jobs/restamp-chunks.js, called by the queue handler after its walk and by the inline path after its own. It is guarded on the token, not on the row revision. parentFieldsPendingAt is that token: the flag and the timestamp go on the same upsert, the timestamp is minted strictly later than whatever the row already carried (documents.freshPendingAt(), so two raises in the same millisecond cannot share one), and the clear patches under the condition that the row still holds the value the walk read. Unconditional clearing could lose a write — a newer edit lands after the handler's read and raises its own flag, and the handler then clears a flag it never satisfied, leaving that edit's chunks stale with no reconcile signal.

  The token replaced an IfMatch on _etag, which asked the wrong question. An etag answers "has anything written this row at all", so an extraction patch or a display-name push landing during the walk refused the clear, and the flag then stood until an operator ran --pending — a repair run for a document whose chunks were already correct. The token asks only whether the raised flag is still the one this walk was for. A refused clear comes back as conflict rather than as a thrown 412: a newer parent-field change owns the flag, its own re-stamp will clear it, and the handler logs one info line and completes the message. A row that vanished comes back as missing, logged at warn, and also completes. A row raised before the token existed carries none and clears unconditionally.

• A write that moves one of these values is an etag-guarded upsert. Both paths — the staff PUT /documents/:id and the eagle-api push, in src/controllers/nosql/document.js — rebuild the whole row from the row they read, and a Cosmos upsert replaces the item, so the write carries IfMatch on the _etag that read returned. A 412 means another writer landed first: the write re-reads the row, rebuilds the change on top of what is stored, mints the pending token off the stored one and sends it again, up to three times. That is what keeps a pending flag another walk cleared from being written back, and what keeps a value another writer moved from being reverted with no flag raised — a revert to the snapshot's own value looks like no change at all to the comparison that decides whether to raise. Past three tries the staff route answers 409, and the push answers 503, the only class of answer eagle-api's push client sends again.

• Chunks carry an ordering token of their own, parentStampedAt. An ingest stamps it on every chunk it writes — the row's parentFieldsPendingAt where the row already owes a re-stamp, the ingest's start instant otherwise — and a walk patches under the condition that the chunk's token is older than the walk's. A chunk stamped by a newer walk keeps what it has and is counted as skippedNewer, an outcome rather than a shortfall. An unstamped chunk is older than every walk, so the token is what stops a re-stamp that set out before an ingest from overwriting the chunks that ingest just wrote with the values it started with. The field orders writers and answers no query: azure/search/indexes/chunks.json does not declare it and demi-chunks-ds does not project it, so it costs neither an index PUT nor an indexer pass.

Rolling these fields into an environment that predates them, and the deploy order that matters: Bulk-Download-Operations "Chunk parent-field rollout".

Adding another field to the copy later touches five places: CHUNK_PARENT_FIELDS in src/repositories/chunks.js, the field definition in azure/search/indexes/chunks.json, the SELECT in azure/search/datasources/demi-chunks-ds.json, the vis catalog entry in src/vis/catalog/chunks.js, and bumping CHUNK_PARENT_FIELDS_VERSION — unless, like projectId, the field is one every ingest path already writes, in which case a wrong value is caught by the value comparison and the bump would only buy a corpus-wide re-stamp. test/azure/search-datasource-columns.test.js pins the list against the two committed JSON files, so leaving one of the first four out fails the test rather than shipping a field that reads null. The version bump is what makes the backfill do anything: backfill-chunk-parent-fields.js re-stamps a chunk only when its stamped version is below the current one, so bumping the constant is what puts every already-stamped chunk back in scope for the next backfill run. Skip the bump and the backfill still runs, still exits, and still reports success — it just walks the whole corpus finding that the old four fields already agree with the parent and re-stamping nothing, a green run that silently skips the new field on every row. parentFieldsOf() is the only place stamping happens, so a version bump with no accompanying code change to what gets stamped has nothing to backfill.

What DEMI's indexes cannot express

No wire key today resolves to a real, wrong-value-space field the way proponent used to (the gate for that case, UNMAPPED_KEYS in src/search/eagle-query.js, is empty). A filter key that names no field, or an unfilterable one, is dropped and reported under meta.dropped rather than sent to the service — see Search-Query-Construction.

dateUpdated and documentUrl

projects carries dateUpdated (Edm.DateTimeOffset, filterable, sortable, retrievable) — the search page's "Last updated" column and default project sort. activities carries documentUrl (Edm.String, filterable, not searchable, not retrievable) — the "has an attachment" filter on the activities list; the row itself is still read back from Cosmos.

Both are widenings, so they need no rebuild, but each needs its own index PUT and its own data source PUT (demi-projects-ds, demi-updates-ds) plus an indexer reset before the app can use them — see "Widening an index is three separate writes" in azure/search/README.md. Until that lands, sortBy=-dateUpdated and and[documentUrl]=true are dropped and named in meta[0].dropped, same as any other field the live index does not yet carry.

The Updates fields on activities

activities carries shortHeadline and summary (Edm.String, searchable with the English analyzer, like headline), and status (Edm.String) and publishDate (Edm.DateTimeOffset), both filterable, for the publish gate. A keyword search matches headline, shortHeadline, summary, content and notificationName.

The gate filter is status eq null or (status eq 'published' and publishDate le <now>). The app names all four fields in searchFields and $filter, and the service answers an unknown field with a 400, so the rollout order is fixed:

1. Deploy the infra bicep with the new updates container index paths, and wait for the Cosmos index rebuild to finish.
2. PUT the activities index, PUT demi-updates-ds, then reset and run activities-indexer.
3. Deploy the DEMI app.
4. Run the publishDate backfill (src/scripts/backfill-update-publish-date.js --live), and repeat a dry run until it reports undated=0. Then set updatesPublishDateFallback = false and redeploy.
5. Deploy eagle-api (its migration, then demi-repush), then eagle-admin.

The reset in step 2 is needed because a row pushed between the index PUT and the data source PUT reaches the index with status null. Until the reset, that row passes the gate in counts and its summary cannot be found by search. The steps are in azure/search/README.md.

Project type spellings

DEMI stores Track's type_name for a Track-backed project and Eagle's type for an Eagle-only one, while eagle-public's option list is Eagle's. Three spellings differ. The pairs are in PROJECT_TYPE_SPELLINGS in src/helpers/spellings.js; spellingsOf returns a value with its other spelling. A filter matches both spellings rather than swapping one for the other, because both are stored: the index holds more projects than the 382-record Track export accounts for, and the surplus are Eagle-only. GET /search (VALUE_ALIASES in src/search/eagle-query.js) and the native GET /projects and GET /counts?kind=projects (projectsKind in src/controllers/nosql/project.js) share the one table. An unlisted value is matched as sent; Track can add a type without asking us.

Project sub-type and the sector fallback

projects carries projectSubType (Edm.String, filterable, sortable, facetable, retrievable, not searchable), the stored projectSubType the merge takes from Track's sub_type_name, and legislationYear (Edm.Int32, filterable, retrievable, not searchable, sortable or facetable), Eagle's legislationYear. A Project row's displayed sector is Eagle's sector, else the sub-type in eagle-admin's PROJECT_SUBTYPES names (normalizeSubType in src/search/sub-type.js), else Other. Three Track names differ:

Track name	2002 Act or earlier	Otherwise
Transmission Lines	Electric Transmission Lines	Electric Transmission Lines
Solid Waste Management Facilities	Local Government Solid Waste Management Facilities	Solid Waste Management
Marine Port Projects	Marine Port Facilities	Marine Port Projects

The year comes from legislationYear on both paths. The bare Cosmos list also reads the legislation label (2002 Environmental Assessment Act) when the year is missing; the index has no legislation column, so a keyword or filtered search uses the 2018 name for such a row. No year reads as 2018.

The sub-type fills in only when the stored row has no sector. A sector a dial withholds from the caller shows as Other, never as the sub-type.

The fallback changes the displayed value only. and[sector]=X and sortBy=sector read the index sector column, so a row that shows its sub-type does not match a filter on that sub-type.

Cosmos-served datasets

GET /search also answers six datasets straight out of Cosmos NoSQL, with no index, indexer or data source behind them. Redaction runs the same way as the indexed datasets (redactAllForAccess), but paging, filtering and sorting are Cosmos SQL, not OData, and none of the rules above (Edm type gates, analyzers, UNMAPPED_KEYS) apply to them.

Dataset	Container	Partition key	Params
List	lists (kind: 'List')	/kind	no filter; caller pages the whole set and resolves ids client-side
Organization	lists (kind: 'Organization')	/kind	companyType
CommentPeriod	commentPeriods	/projectId	_id; or and[project]=<Eagle project id> (translated to the DEMI project id first), paged and sorted within that project; or, with no project, every period this caller may see — keywords/q (substring match on the period label, or on its project's name for keywords of 2+ characters), and[status]=open|upcoming|closed (one value, never repeated or empty), sortBy=dateStarted|dateCompleted|dateAdded (+/-, default -dateStarted), paged by pageNum/pageSize across the whole filtered set. Deleted periods are excluded from the list, its count and the rail. Rows whose parent project or notification the caller cannot see are withheld — after a failed publish cascade the count can briefly exceed the rows returned. and[status]=open with no sortBy, no keywords and no pageNum answers the capped home-rail read instead (soonest to close first, plus closedCount)
Comment	comments	/periodId	_id, or and[period]=<id> — required; a bare list with neither is refused, not answered empty
RecentActivity	updates	/id	top=true (the home strip: at most 4 rows, pinned first, no paging), or and[project]=<Eagle project id> plus keywords/q. Below staff level, drafts and not-yet-due Updates are withheld here and on the activities index; see Eagle-Mirror-Reference § Updates
ProjectNotification	notifications	/id	_id, or and[type], and[region], and[pcp], and[decision]

List and Organization share one container, told apart by kind; List has no eagle-api push — see Eagle-Mirror-Reference § Migration re-push rule. A CommentPeriod row's project field and a RecentActivity row's projectId are the Eagle project id, resolved from the stored DEMI id under the caller's own access, because eagle-public's models read the Eagle id. A ProjectNotification row's proponent is resolved from an Organization id to its name when one matches; sent through unchanged otherwise.

Every dataset here is subject to the same pageSize ceiling as the indexed ones: a request that carries a filter or a keyword is capped at 500 (aiSearch.MAX_PAGE_ROWS), refused with a 400 above it rather than truncated. The unscoped CommentPeriod list, when that outer gate does not apply, falls back to the same per-request ceiling every other Cosmos list read uses (pageSizeFor): 10 rows when pageSize is absent, an unauthenticated request over 100 refused with a 400, an authenticated one clamped to 1000. A sortBy key CommentPeriod cannot use is named in meta[0].dropped.sort, same shape as a dropped filter key.

and[status]=closed treats a period whose dateCompleted falls exactly at Pacific midnight (America/Vancouver) as still open for the whole of that Pacific day, matching how eagle-public renders it — the boundary is computed in Pacific wall time, not UTC.

See also

• Search-Query-Construction — what the query layer does with these fields
• Architecture — containers and the ACL model
• Environment-Reality-and-Operational-Gotchas — indexer and data source gotchas in the estates
• Bulk-Download-Operations — rolling a new field into an environment: index PUT, data source PUT, backfill, indexer catch-up, for both displayNameSort and the chunk parent fields