Operating DEMI Data Plane

Operating the DEMI Data Plane

How to reach DEMI's private Cosmos and Search data planes from the devbox, probe a change, write code against the Functions dispatcher, read audit, analytics and cost, and flip prod search between backends. Deploying and verifying the Azure estate is Operating-DEMI-in-Azure.

Reach the data plane

Cosmos and AI Search are private-endpoint-only and keyless, so anything touching them has to run inside the network, as the API's own identity. In test that route is the devbox VM demi-devbox-test: scripts/demi-devbox.sh run --env test -- '<command>' starts the VM if needed, waits for its VM agent, and runs sudo -u demi /usr/local/bin/demi-run '<command>' through run-command. demi-run logs in as the UAMI, exports COSMOS_ENDPOINT/SEARCH_ENDPOINT/etc, and execs the command as one process. The script needs az on your machine, and jq unless DEVBOX_RUNNER is set. Deallocate the VM after.

Interactive sessions go through Bastion (bastion-test, Developer SKU — portal browser shell only, no native client or scp). The full recipe, including where the output goes and the storage roles it needs, is in the repo README's "Running anything against the database". This page only points at it, so the two never drift apart.

Flex Consumption has no Kudu and no SSH, so none of Kudu's traps apply to it. Prod has the same devbox route: demi-devbox-prod (Bastion bastion-prod, keypair in demi-kv-prod). Devbox VMs run B2s — the prod landing zone installs Defender/AzureMonitor/ChangeTracking extensions, and on a B1s the install storm wedges the VM agent so run-command dies. A killed run-command wedges the agent the same way; az vm redeploy clears it.

Prefer an API endpoint over out-of-band database access where one exists. GET /admin/index-progress issues no container query at all, so it answers when /db/stats counts are timing out.

After any deploy touching azure/search/ or search code, run the index drift check: scripts/demi-devbox.sh drift --env <env> runs apply-search-definitions.js --check on the devbox under a temporary Search Service Contributor grant and exits 1 when a committed field is missing from a live index — the failure class that is otherwise invisible until a query names the field and 400s. The repo README documents the underlying command. GET /health/search-schema checks the same thing over HTTP, unauthenticated, without touching the devbox: 200 means every live index can answer what the app asks; 503 names the index and the missing field. Outage response, the fix (demi-devbox.sh apply) and rollback: Runbook-Search-Outage.

drift is one run command; apply is three or four, one per phase (dry run; index and data source PUTs in one call; indexer reset, run and wait; plus one pre-create call only when --datasources names a data source that does not exist yet), because each one is a 20-45 s ARM round trip and the per-index writes loop inside one payload instead of one call each. The wait for the indexer reset runs on the devbox itself (src/scripts/reset-and-run-indexers.js), not as a poll per tick from the workstation — that script also waits out any execution that was already running before asking for the reset, so a run started right after the reset does not report items=0.

A run command is one more VM agent goal state, so it queues behind any extension the agent is still working through. Before its first remote call, and again after each apply prompt, demi-devbox.sh reads the instance view and waits until the agent is Ready and no extension or extension handler is in a transitioning, creating, updating or deleting state, up to 20 minutes (DEVBOX_READY_TIMEOUT). Failed and NotReady ones are listed and not waited on: the prod ChangeTracking-Linux handler sits at NotReady/1011 (read 2026-10-01). On timeout it exits 1 and names the agent status and the busy extensions.

Run-command output is capped at 4 KB and a run at 90 minutes in the default mode, az vm run-command invoke. That is why reset-and-run-indexers.js logs one DEMI_POLL line every 4th tick rather than every tick. DEVBOX_RUN_MODE=managed writes output to append blobs in container devbox-run-output on the demifc* storage account instead, through a user-delegation SAS. That SAS carries only the blob roles of the operator's own az login, so each operator needs Storage Blob Data Contributor on the container, and Storage Blob Delegator on the storage account unless they already have Owner or Contributor over the account. Both grants are made by hand. With Reader the VM gets 403 AuthorizationPermissionMismatch (measured 2026-10-01). The container is declared in azure/modules/api-function-flex.bicep and exists on test and prod.

Probe a change

• Take the baseline before the change. "Answers in seconds" is not a result if the old build also answered in two.
• One output file per probe. A timed-out curl does not rewrite its -o file, so a shared file makes the next probe re-read the previous answer.
• Pick probes where the competing explanations predict different answers. A probe that cannot fail proves nothing.
• An ACL probe needs a record the caller must not see. Comparing anonymous and privileged counts proves nothing when every record is public and both pages are capped. Create a throwaway record with read: [sysadmin, staff, demi-admin], expect 0 anonymous and 1 privileged, then delete it.
• A fallback probe needs a nonsense term. The Cosmos fallback ignores keywords and returns the head of the corpus, so a common word returns rows either way. A nonsense term returns nothing when search is healthy and rows when it has fallen back.

Write code against the Functions dispatcher

The Functions HTTP trigger (api/index.js, dispatched by src/http/router.js) always returns a single HttpResponseInit — never a 304, never a streamed response, never platform-level compression() output. Strip conditional headers and buffer static files instead of relying on either.

Read audit, analytics and cost

GET /admin/audit, GET /admin/analytics and GET /admin/cost on demi-api answer under a sysadmin, staff or demi-admin role — no Azure portal access needed. The DEMI admin panel's Audit log, Site analytics and Azure spend screens call these. Each answers 503 while its app setting is empty, and /admin/cost also answers 503 while Azure rate-limits the query and no cached figure exists; see Environment-Reality-and-Operational-Gotchas for the settings and ADR-008-Audit-and-Usage-Analytics-Store for what each table holds. GET /admin/cost caches one hour per API instance, so a spend change can take up to an hour to show.

Bytes per client

Use this to find who pulled how much during a bulk scrape. The Front Door log ends at the 302, so it has no byte counts.

DemiEvents_CL cannot be queried directly. It is on the Auxiliary plan, and a direct query returns no rows. Read the hourly rollup DemiDownloadsHourly_CL instead. The summary rule demi-downloads-hourly writes one row per document per hour, with Downloads and Bytes. Bytes is the sum of Detail.bytes on the document.download events, which is the file size recorded on the document.

1. Open the demi-audit-<env> workspace in the portal, under Logs.
2. Run the query below, with test swapped for the environment. It joins the rollup with the Front Door access log in eagle-logs-<env> on document id and hour.

let perDownload = DemiDownloadsHourly_CL
  | where Downloads > 0 and isnotnull(Bytes)
  | project Hour = _BinStartTime, DocumentId, BytesEach = Bytes / Downloads;
workspace('eagle-logs-test').AzureDiagnostics
| where Category == 'FrontDoorAccessLog' and requestUri_s has '/download'
| where httpStatusCode_s in ('200', '302')
| extend DocumentId = extract(@'/documents/([^/?]+)/download', 1, requestUri_s),
         Hour = bin(TimeGenerated, 1h)
| join kind=inner perDownload on Hour, DocumentId
| summarize Downloads = count(), Bytes = sum(BytesEach) by Day = bin(Hour, 1d), clientIp_s
| top 50 by Bytes

The result is an estimate. It uses the recorded file size, not the bytes sent. A download with no recorded size adds nothing to Bytes. clientIp_s follows X-Forwarded-For when the caller sends one; socketIp_s is the address Front Door accepted the connection from.

Flip prod search between backends

Which backend eagle-public calls in prod is one field in eagle-api's Mongo Config, with its own runbook: docs/prod-flip-runbook.md in the eagle-demi repo.

See also

• Operating-DEMI-in-Azure — deploy, restart, verify
• Environment-Reality-and-Operational-Gotchas — what exists, network reality, app settings, indexer and runtime gotchas
• Runbook-Search-Outage — index/app schema drift, diagnose, fix, roll back
• Edge-Ban-Detector — the hourly timer that bans scraper addresses at Front Door