Deep Search Queries

Deep Search queries

Deep Search is keyword search inside documents. It runs on dataset=DocumentChunk of GET /search and on GET /documents/passages, and the chunks index answers it. This page owns how those queries filter, page, degrade and gate.

Routing and the rest of GET /search: Search-Query-Construction. Native list endpoints: Native-List-Reads. Index fields: Search-Index-Reference.

Native GET /documents/passages

The endpoint returns one row per chunk hit. q is required, and pageSize is at most 100. count=true and page work as on Native-List-Reads, and ids= and sort are not accepted. Filters: project, documentId and the four id filters of GET /documents.

searchChunks answers the query. The parent gate then drops chunks whose document the caller cannot read, with the count net of the drop (src/search/chunk-query.js, see "The parent document is the chunk gate").

A parent-field filter on a scope the backfill has not fully stamped answers 503 with the reason and logs a warning. A 200 with zero rows would read as "no such documents".

Chunk filters answer on the chunk's own copy

and[type], and[milestone], and[projectPhase] and and[documentAuthorType] filter dataset=DocumentChunk directly. buildFilter maps each wire key onto a field the chunks index already carries (typeId, milestoneId, projectPhaseId, documentAuthorTypeId; DOCUMENT_FACETS in src/search/eagle-query.js) and emits one clause, as for any other chunk filter. No document-index lookup runs first. Field reference and how the copy stays current: Chunk-Parent-Fields.

Three cases are not answered on the chunk row, and all three take the same path out.

The first is a facet the live index does not carry. The app reads expressibility from the service, not from the chunks.json in the deployed package, which nothing on the deploy path applies.

Between an app release and the index PUT the two disagree. A clause on a missing field is a 400, which the route answers as a 502 over the whole tab.

• The answer is cached for ten minutes, so widening the index works without a restart.
• A probe that could not run is held for 30 seconds, because that is a hiccup and not a fact about the index.
• Only a field the probe vouched for is sent. An unreachable field is treated as inexpressible. Guessing it exists answers the whole tab as 502, while dropping it costs one filter the documents index usually answers.
• The probe asks one question per field, all at once, so one field's hiccup drops one filter, not all four.

The second is a column the live index carries but this request's rows do not all hold a value for yet. The schema probe answers only whether the column exists. A chunk that backfill-chunk-parent-fields.js has not reached holds null in all four parent columns. typeId eq '<id>' over it is a non-match, and the 200 reads as "this project has no documents of that type".

liveChunkFacets in src/search/chunk-query.js counts stale chunks under the caller's own ACL and project scope. It withholds all four facets together when the count is nonzero, because one facet answering zero poisons the whole conjunction. This is the common case during a backfill run.

The third is a document-only key: datePostedStart, datePostedEnd, isFeatured, legislation or documentSource. A chunk carries no copy of those.

All three are resolved through the documents index: ask which documents match, then scope the chunk query to their ids. That query runs under the documents index's own ACL.

A document-scoped credential compares documentId on chunks and id on documents. Reusing the chunk clause is a 400 naming a field the documents index lacks, which the route answers as a 502.

The id list is capped at 249, one service request's worth. Over the cap the key stays in dropped rather than applying to an arbitrary prefix.

Project-scoped, which is how eagle-public sends these, the set is a handful of documents and the filter applies. Corpus-wide, a date range does not.

So during a deploy or a backfill window a facet degrades to a capped but applied filter, not to no filter. A key neither index can express, documentAuthor for one, is dropped and named in meta.dropped.filter.

Deep Search can be degraded three ways

meta[0].degraded on dataset=DocumentChunk carries missing and up to three reasons, in any combination.

• missing: [field]: the live index could not answer a column the app selected, so the search layer dropped it and retried once. The page is narrower than asked for.
• reasons: ['chunk-parent-fields-unstamped']: the facet filter was withheld, not applied. This request's scope is mid-backfill, and the documents-index recovery was over the 249-document cap, so the page is short. When the recovery lands under the cap, the documents index answers the filter in full and nothing is marked, which covers most requests because eagle-public sends them project-scoped.
• reasons: ['chunk-parent-fields-unknown']: the same withheld filter, for a question nobody could answer. A live-index probe that could not run, or a stale-chunk count the service could not give, withholds the facet as a positive count does. It clears when the service answers again, while the unstamped mark clears when the backfill finishes.
• reasons: ['chunk-parent-fields-missing']: the same withheld filter, because the live chunks index does not carry the column. This is a deploy before someone applies azure/search/indexes/chunks.json, and an operator clears it by applying that definition. Without the reason, the wire shape is a key in dropped.filter with nothing beside it, like an ordinary over-cap drop.

All three reasons are raised only when the documents-index recovery was over the cap, since only then is the page short. They can appear together, for example a probe that died while the count that did answer came back positive.

The mark covers the rows the caller asked for, not the corpus. The count behind it uses the same filterFor clause as the page, plus the request's project scope. A backfill in one project therefore does not banner another project's page.

The index-wide count is only a free short-circuit: it is a superset of any scope, so zero there is zero everywhere, and null marks nothing. Counts are cached per access-and-project scope for ten minutes, like the schema answer, in a map bounded at 32 scopes. A count of "cannot say" (the service failed, or the index has no stamp column) is held for 30 seconds.

Unknown is not zero in either direction. A per-scope count of null withholds the facet as a positive count does, but only a measured positive count raises chunk-parent-fields-unstamped. An unanswered count is not evidence the backfill is behind, so it reports chunk-parent-fields-unknown.

The reason carries no count. The route answers anonymous callers, and a backfill statistic is not something a caller can act on.

The app reads the state off the stamp, parentFieldsVersion below CHUNK_PARENT_FIELDS_VERSION or absent, never off the four values. A document with no List refs legitimately stamps four nulls, and prod has many, so a probe counting "all four null" would never reach zero.

A live index with the four fields but no stamp column answers "unknown" and leaves the page unmarked.

Chunk paging is a window

Chunk rows are grouped by parent document, so a page of documents costs a window of chunks. The window is both the fetch and the paging unit, and skip and top must share it or consecutive pages leave gaps.

SERVICE_MAX_TOP caps the window, not MAX_PAGE_ROWS. This query runs on every debounced keystroke, and a larger window costs a second request. pageSize is a fetch knob for this dataset, not a row count.

The parent document is the chunk gate

A chunk is a fragment of its document, so a caller who cannot read the document cannot read its text. The chunk's own read[] is an ingest-time snapshot. DEMI cascades a document's ACL onto chunks only on a publish transition, so a stale chunk ACL cannot leak text on its own.

count is net of what the page withheld and floored at the surviving chunk count, never at the row count, because one row can carry a dozen matches.

The summary endpoint's gates

GET /api/search/summary is on authMiddleware, not passiveAuthMiddleware: a <mark> fragment discloses a phrase, and a synthesised cross-chunk answer discloses substance. Three gates are load-bearing:

• the AI Search ACL filter
• chunksRepo.getById re-applying the ACL at the database
• the parent-document check the chunk search path also applies

The model is never called when retrieval returns nothing; that is the grounding guarantee.