Runbook Search Outage

Runbook: search is down or returns nothing

Neither deploy workflow applies index definitions, so the live AI Search index and the deployed app can disagree at any time. When they do, every query that touches the missing field fails. This page is the response, in the order to do it. Background on why: CI-Deploy-Workflows and azure/search/README.md in the repo.

Symptom

• eagle-public search returns "Document search is unavailable"; the browser console has nothing else. Project search may still work — the two use different indexes.
• Or search returns 200 with zero results, or with a filter that matches nothing. That is the quieter half of the same fault: the field exists in the index but every row holds null, because the data source is still projecting the old column list.

1. Name the cause, one curl

curl -s https://www.projects.eao.gov.bc.ca/demi-search/health/search-schema

{"ok":true} means every committed field is in the live index. A 503 names the index and the missing fields ({"ok":false,"indexes":{"documents":{"ok":false,"missing":["fileSize"]}}}). A 404 means the deployed build predates that endpoint — use the query itself:

curl -s "https://www.projects.eao.gov.bc.ca/demi-search/search?dataset=Document&pageSize=1"

502 is the drift shape. 200 with searchResultsTotal above zero means search is up and the problem is elsewhere.

2. Read the 502 body

A search 502 carries a code, and sometimes a traceId:

code	Meaning	Next step
SEARCH_SCHEMA_DRIFT	The live index cannot answer a field the app asks for; the body's field names it	Go to step 3
SEARCH_UPSTREAM	The service did not answer at all — a role, a timeout, a Cosmos fault	Not this runbook; check the identity's role assignment and Cosmos health first

The upstream error text itself never appears in the body — it can carry the index name and the caller's OData filter. It is in Log Analytics instead (demi-logs-prod, demi-logs-test):

AppTraces
| where Message startswith "[search]" and Message has "failed"
| project TimeGenerated, Message, Properties
| order by TimeGenerated desc

Could not find a property named '<field>' is schema drift and this page fixes it. Anything else — throttling, timeouts, 403 — is not, and the fix here will not help. A traceId in the response body is the request id every log line for that request already carries; use it to find the exact call.

3. Confirm the drift

Read-only. Exits 1 when a committed field is missing from the live index:

scripts/demi-devbox.sh drift --env prod     # or --env test

It runs apply-search-definitions.js --check on the devbox under a temporary Search Service Contributor grant, and revokes the grant afterwards. The comparison is against the definitions in the devbox checkout, which tracks main — if prod is running an older tag, that is still the right comparison, because the index has to satisfy whatever is deployed next as well.

4. Fix it, per environment

scripts/demi-devbox.sh apply --env prod --only documents    # or --env test

--only keeps the run to one index and its indexer. It dry-runs first, prints what it would write, and asks before writing. demi-devbox.sh reads the resource group, subscription, service and indexer identity off the environment it is given, so a prod run cannot inherit the test defaults:

	test	prod
SUBSCRIPTION	7897ceb1-9a86-4639-87d7-7f9ff67142b3	be5924ac-1083-4a1b-be92-7b444882cfd9
RG	c4b0a8-test-rg	rg-demi-prod
SERVICE	demi-search-test	demi-search-prod
IDENTITY	demi-identity-test	demi-identity-prod
devbox VM	demi-devbox-test	demi-devbox-prod

What it does, in this order, because any other order either takes the site down or fills the index with nulls:

1. PUT the index and its indexer. Adding a field is supported in place; a drop, a retype or a changed analyzer is a rebuild and the script refuses it. A refusal here means the change is not a widening — go to azure/search/README.md, "Restoring one", and do it by hand.
2. PUT the data source, if its committed SELECT differs from the live one. Without this the indexer keeps projecting the old columns, the new field stays null on every row, and nothing reports an error. Only the data sources that differ are written.
3. Reset and run that indexer, and wait for the run to finish. A schema-only change re-pulls nothing on its own — the high-water mark is _ts — so the reset is what fills the new column. The script refuses to reset while an execution is in progress: a PT5M tick that finishes after the reset writes its old high-water mark back and silently undoes it. The wait also tolerates a 409 on the run POST (already running from the reset) and follows that run to a terminal status, checking initialTrackingState to confirm the reset actually took.

Then re-run step 1's curl. documents-indexer reports around 61,500 rows processed when it has done its job; projects-indexer around 393. chunks-indexer is hours and about 1.1M rows — do not reset it to fix a metadata index.

Timeout behaviour. The wait is bounded (INDEXER_TIMEOUT, default 1800 s). If the deadline passes and the indexer is still running, the script prints DEMI_WARN … still running and exits 0 — the run was posted and is working, not failed; re-check with watch below. A Failure status on the run itself exits non-zero. chunks-indexer needs the longer INDEXER_TIMEOUT_LONG (default 4800 s) because run-command itself is cut off at 90 minutes, which is why a chunks reset always goes through --no-wait rather than the default wait:

scripts/demi-devbox.sh apply --env prod --only chunks --datasources demi-chunks-ds --no-wait
scripts/demi-devbox.sh watch --env prod --datasources demi-chunks-ds

--no-wait returns as soon as the reset and run are posted; watch picks the wait back up later, against the same indexer, without re-running the reset.

Prerequisites, if the script stops early:

• az login --tenant <tenant id> --scope "https://graph.microsoft.com//.default" when it reports that this login cannot read role assignments. An expired Graph token does not fail the grant loudly; it fails the command on the devbox as a 403 an hour later.
• The devbox VM must exist in the environment. demi-devbox-prod is started by the script and should be deallocated afterwards (az vm deallocate -g rg-demi-prod -n demi-devbox-prod).

5. Or roll back

Faster than a fix when the previous release was healthy, and it is the right move if the index change is a rebuild rather than a widening. Deploy the last known-good tag through "Deploy DEMI to Azure production" (CI-Deploy-Workflows) — deploy source is always a tag, never a branch:

gh release list -R digitalspace/eagle-demi --limit 5
gh workflow run "Deploy DEMI to Azure production" -R digitalspace/eagle-demi -f version=v0.77.1

In production, the rollback job runs automatically when deploy-api fails after publishing—either the post-deploy smoke test fails or the app never comes up. A warm-up curl before the smoke prevents false rollbacks from Flex cold starts. See CI-Deploy-Workflows.

Rolling back the app does not roll back an index that has already been widened, and it does not have to: a wider index serves a narrower $select perfectly well.

6. Break-glass: skip a broken probe

For a different failure than steps 1-5: the deploy gate itself cannot answer, not the schema. Use this only when /health/search-schema or /health/search-data on the app serving production returns 500 — not 404 (that is allow_missing_schema_probe, a first-deploy case) and not 503 (that is real drift, go to step 3).

Precondition: confirm the tag is not actually widening the index before you bypass the check that would have caught it. scripts/demi-devbox.sh drift compares against main on the devbox's own clone, not against the tag being deployed, so this only holds when main already contains that tag:

scripts/demi-devbox.sh drift --env prod

Exit 0 means main's index definitions already match, or are narrower than, the live index. Only then dispatch the deploy with the probe skipped, on the first tag that carries the skip_schema_probe input (v0.114.4 or later):

gh workflow run azure-deploy-prod.yaml -R digitalspace/eagle-demi --ref v0.114.4 \
  -f version=v0.114.4 -f skip_schema_probe=true

The run logs a warning naming the dispatcher and the tag, then deploys normally. skip_schema_probe bypasses both the schema and data probes, not the risk they guard against — treat every use as an incident, not a shortcut. If this input is getting used often, the probe endpoint is the thing to fix, not the flag.

Because the data probe is skipped too, follow up by hand: run the same document and project search queries the deploy's own smoke test runs (see "Deploy DEMI to Azure production" in CI-Deploy-Workflows) and confirm both return results, not just 200.

Example, 2026-09-22: production tag v0.114.1 shipped without azure/search/data-checks.json, so both health routes returned 500 and the fix could not pass the gate.

Afterwards

• Run scripts/demi-devbox.sh drift --env prod as the last step of every prod release (Release-Process), and after any deploy that touches azure/search/ or search code.
• Two alerts exist because of this failure and page on the same outage from different angles: demi-search-failures-<env> (three or more [search] … failed trace lines in five minutes) and demi-search-5xx-ratio-<env> (over 20% of /search requests answering 5xx in five minutes, over at least five requests). demi-search-availability-<env> also fires if the public web test drops under 90% over fifteen minutes. All three mail the same action group — check it is being read.
• Field-by-field background, and every hand recipe this script wraps: azure/search/README.md in the repo.

Example

2026-09-08, 20:56 UTC: production deploy v0.77.2 (from v0.58.3) started selecting fileSize on Document queries. The live prod documents index had no such field. AI Search answered 400, Could not find a property named 'fileSize'; the API turned that into a 502, "Document search is unavailable", on every Document query from 20:59 to 22:05 UTC. The availability alert, demi-search-availability-prod, fired by email at 21:04 UTC and went unread.

Fix: from demi-devbox-prod, under with-search-admin.sh with SUBSCRIPTION set to the prod subscription, RG=rg-demi-prod, SERVICE=demi-search-prod, IDENTITY=demi-identity-prod —

apply-search-definitions.js --only documents --live

The az login that granted the Search Service Contributor role needed a Graph-scoped login first; this is the failure step 4's prerequisites above describes.