Operating DEMI in Azure

Operating DEMI in Azure

How to deploy and verify DEMI's Azure estate. Devbox access and probes are Operating-DEMI-Data-Plane. What is deployed and why it is shaped that way is Environment-Reality-and-Operational-Gotchas; subscriptions, Bicep and cost are Azure-Environments.

Deploy the API

The Flex app (demi-api-fc-test, demi-api-fc-prod) has no Kudu or SCM site. Every deploy is az functionapp deployment source config-zip, which uploads the package to a blob deployment container and replaces it whole — there is no merge into an existing wwwroot and nothing to prune.

• Test: CI runs config-zip on every push to main under the API paths.
• Prod: gh workflow run "Deploy DEMI to Azure production" --ref v<tag> -f version=v<tag>.
• By hand:

  BUILD_ID="$(git describe --tags --always)-$(date -u +%H%M%S)" python3 scripts/package-api.py . /tmp/api.zip
  az functionapp deployment source config-zip -g <resource-group> -n <app-name> --src /tmp/api.zip
  

  scripts/deploy-azure.sh is frontend-only; it does not deploy the API.

Verify a deploy landed:

• GET /api/config reports the BUILD_ID package-api.py stamped into the package. This is the only honest signal that the new code is answering.
• az functionapp function list -g <resource-group> -n <app-name> lists api once the Flex host has finished syncing triggers from the new package. That sync is asynchronous — poll it rather than assuming the config-zip call returning success means triggers are live.

Traps:

• Do not prune dist by name at every depth in the package. That also strips node_modules/**/dist, for example @mongodb-js/saslprep, shipping an app that 500s on every request. Prune at the repo root only.
• src/scripts/ is runtime — src/controllers/db.js and wildfire.js require it. Root scripts/ is deploy tooling and is excluded. test/scripts/package-api.test.js pins both halves; conflating them breaks the admin sync endpoints with an ENOENT only visible in Azure.
• Never ship .env in the package.

Deploy infrastructure

scripts/deploy-infra.sh <env> runs a what-if of the application layer. Flags go after <env>, in any order:

• --live deploys instead of what-if.
• --foundation (with or without --live) includes the foundation layer too.
• Prod additionally needs CONFIRM_PROD=yes exported before --live is honoured.

Two layers, two module lists:

• Foundation — deployed only with --foundation: identity, key-vault, cosmos-nosql, observability, audit-logs, foundry, ai-search, search-existing, apim, static-site, document-storage, cost-budget.
• Application — deployed on every run: api-function-flex, devbox, secret-sync, availability, and the cost-reader role assignment.

A change to any foundation module needs a --foundation run. The script compares the commit the newest successful infra-fnd-* deployment ran from against the working tree. When a foundation module changed since then, or no infra-fnd-* deployment is on record, it refuses a --live application run and warns on a what-if instead. A change to azure/main.bicep or a .bicepparam file only warns either way, because those files change for application work too and the script cannot tell which half an edit touched.

Deployment names carry the layer and the commit: infra-fnd-<sha>-<hhmmss> for a foundation run, infra-app-<sha>-<hhmmss> for an application run.

An application-mode what-if lists the foundation's resources under "ignore", not as deletions — that is expected, since the application template does not manage them. If a foundation resource is actually missing, the application run fails instead, with ResourceNotFound on the existing lookup for it.

A new environment needs --foundation --live first, then application runs after.

Measured 2026-09-17/18, ARM deployment duration (az deployment group show --query properties.duration): a foundation run takes about 3 min 54 s on both test and prod; an application run takes about 20 s on both. The script's wall-clock time is dominated by its first step, which starts the devbox VM and lists Key Vault secret names through a run-command — about 2 min on test, and about 33 min on prod on 2026-09-18 (16 min for the VM to start, 16 min for the run-command).

Apply a Cosmos indexing policy alone

scripts/apply-cosmos-index.sh <test|prod> <container> [--live] compares the indexing policy declared for the container in azure/modules/cosmos-nosql.bicep with the live one on demi-cosmos-<env>, and prints the diff. It is a dry run unless --live is given; prod --live also needs CONFIRM_PROD=yes. Use it for an index-only change instead of a --foundation --live run. Cosmos rebuilds the index in the background after the update. Queries that filter on a newly added path may scan until the rebuild finishes.

Deploy the PDF title worker

The worker is the Function app demi-pdf-title-<env>: Python 3.11 on Flex Consumption, defined in azure/modules/pdf-title-worker.bicep. What it does to a PDF is PDF-Titles.

• It has its own user-assigned identity and its own storage account. It does not use demi-identity.
• A timer runs pdf-title/run.py on the schedule in PDF_TITLE_SCHEDULE, default every 10 minutes (0 */10 * * * *).
• It is a dry run unless PDF_TITLE_LIVE is true.
• host.json sets functionTimeout to 20 minutes. PDF_TITLE_MAX_MINUTES stops new rows from starting and must stay under 15; the host refuses a larger value or an unset one.
• maximumInstanceCount is 1.
• It calls the API at DEMI_API_URL with DEMI_API_KEY, a Key Vault reference to the secret pdf-title-worker-api-key. Its identity can read only that secret.
• The module deploys when deployPdfTitleWorker is true and pdf-title-worker-api-key is in optionalSecretNames. Prod's param file sets deployPdfTitleWorker to true, names the secret in optionalSecretNames and sets pdfTitleLive to true.
• It runs on test as demi-pdf-title-test and on prod as demi-pdf-title-prod, both on the 10-minute timer. Prod settings are in PDF-Titles.

The code is shipped by azure-deploy-staging-pdf-title.yaml (CI-Deploy-Workflows).

Stream document downloads

GET /api/documents/<id>/download?inline=1 sends the file from DEMI instead of redirecting to the object store. The mode is described in ADR-011-Public-Read-Surface. Four settings control it.

Setting	Where	Effect
DOWNLOAD_STREAM_MAX_BYTES	Function app setting, default 33554432 (32 MiB)	Largest file DEMI streams. A larger file gets the 302. 0 turns streaming off.
DOWNLOAD_STREAM_MAX_CONCURRENT	Function app setting, default 8 per instance	Streams one instance serves at once. Above it the signed redirect 302 is sent. 0 means no limit.
enableHttpStream	api/index.js, app.setup({ enableHttpStream: true })	The Functions host passes the response body through instead of buffering it. Ships with the API package.
buffer-response="false"	APIM policy on forward-request (apim.bicep)	APIM passes the body through. Needs a --foundation run of deploy-infra.sh (see "Deploy infrastructure").

To turn streaming off without a deploy, set DOWNLOAD_STREAM_MAX_BYTES=0 on the Function app. Every inline link then gets the 302 again. The edge keeps cached copies for up to 5 minutes. The next deploy-infra.sh run puts the setting back, because it writes appSettings as a whole collection. Change the default in the Bicep parameters, not only on the app.

Stream mode runs on test and prod (eagle-demi v0.172.2 and later; eagle-edge v1.4.6 and later, route demi-documents). The APIM policy buffer-response="false" is live on test and prod.

Restart the API

1. az functionapp restart does not recycle the Node worker. Use stop, then start.
2. Never stop or restart while State: Starting. It cancels the incoming container and hands traffic back to the previous one.
3. A warm worker can serve the old build for minutes past the roughly 50-second cold start. Poll a discriminator until it flips rather than trusting the restart.

Deploy the frontend

Test only; there is no prod frontend. scripts/deploy-azure.sh frontend uploads the React bundle with az storage blob upload-batch into $web — no zip, no Kudu, no deployment record to poll, and a blob PUT is live when it returns. FRONTEND_STORAGE_ACCOUNT has no default, because the account name carries a uniqueString suffix, and the script aborts without it.

Before uploading, the script asserts the built dist/env.js keeps API_LOCATION empty (same-origin /api, resolved through the gateway per-environment — see Environment-Reality-and-Operational-Gotchas) and configEndpoint true. A bundle built for the wrong environment fails this check and is never uploaded.

Three consequences:

• Cache-Control is stamped per blob at upload time, because $web cannot add a response header. Pass 1 uploads everything no-store, pass 2 re-uploads content-hashed extensions immutable, pass 3 puts env.js back to no-store — it is the one unhashed .js in the bundle, and a stale copy points the browser at the wrong API and realm.
• There is no pruning, deliberately. Blob upload does not merge into the previous deploy the way zipdeploy does, so a superseded main-<hash>.js is inert rather than a symptom. It costs fractions of a cent of LRS and is linked from nothing.
• Publishing needs a data-plane role. az storage blob upload-batch --auth-mode login requires Storage Blob Data Contributor on the account, assigned by azure/modules/static-site.bicep from frontendUploaderPrincipalId, the object id of the service principal. Website Contributor grants nothing on a storage account and gets a 403 at upload.

Static website hosting is a blob data-plane setting with no ARM property: Bicep rejects it with BCP037, and writing it into a raw ARM template compiles and is then silently dropped, which is worse. Run one idempotent command per account:

az storage blob service-properties update --account-name <acct> --auth-mode login \
  --static-website --index-document index.html --404-document index.html

Skip it and every blob uploads successfully while the site 404s — nothing in the deploy output says so. The eagle-demi-admin deploy workflow catches it: after upload it curls the site through Front Door.

Point Front Door at the frontend, and open CORS

1. Deploy the Front Door profile from eagle-edge (its Deploy edge workflow runs on push to main; demiFrontendWebHostName in azure/main.test.bicepparam there is the DEMI origin). The endpoint hostname is <name>-<hash>.<zone>.azurefd.net and Azure assigns both the hash and the zone, so read it from the profile's edgeEndpointHostNames output; it cannot be composed.
2. Put that hostname in frontendHostName in azure/main.test.bicepparam and redeploy. It feeds both the API's CORS_ORIGIN app setting and the Flex app's platform cors.allowedOrigins. It is committed empty on purpose: empty fails closed — src/http/router.js falls back to an allowlist holding only http://localhost:4200, so the frontend breaks in one visible request rather than the API reflecting every origin.
3. Wait about three minutes. The platform CORS setting and Front Door rule sets keep returning a hard error that looks exactly like a failed deploy for that long after ARM reports Succeeded. Poll rather than concluding from the first check. A cache purge fixes neither, because this is configuration and not cached content.
4. Check the frontend in a browser with the console open. A curl header check passes while every API call is CORS-blocked.

Both CORS layers matter: the platform cors collection answers the preflight first, so setting only CORS_ORIGIN is not enough.

During an origin change, list the old and the incoming host together in frontendHostNames and drop the old one only once the old origin is gone.

The rule set is the only place the SPA fallback and the security headers exist — HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy. DEMI's CSP goes out as Content-Security-Policy-Report-Only until the console is clean.

Verify a deploy

CI runs the post-deploy check.

• The staging API workflow curls https://<function default hostname>/health after the function registers.
• The eagle-demi-admin deploy curls the site through Front Door. It uses the Front Door hostname, not $web. The security headers come from the AFD rule set, so the origin fails them.
• /api/config returning 200 proves nothing about a build — every build answers it. /api/search/summary tells builds apart: 404 before the summariser, 401 once it exists.

Verify document streaming

After a deploy that touches the API, APIM or the edge, check a public document. After an edge deploy, wait 10 minutes before the curl check: Front Door edges converge unevenly, and until they do some answer 404 for every document download. The id 5887d0df3fa55712b70aaf38 is a public PDF on test.

1. Fetch the file:

   curl -s -o /dev/null -D - "https://test.projects.eao.gov.bc.ca/demi-search/documents/5887d0df3fa55712b70aaf38/download?inline=1"
   

   Expect 200, Content-Type: application/pdf, a Content-Length, a Last-Modified and Cache-Control: public, max-age=300. A 302 means the file is over the limit, the setting is 0, or the store could not be read.
2. Fetch a range:

   curl -s -o /dev/null -D - -H "Range: bytes=0-1023" "https://test.projects.eao.gov.bc.ca/demi-search/documents/5887d0df3fa55712b70aaf38/download?inline=1"
   

   Expect 206 and Content-Range: bytes 0-1023/<size>.
3. A second identical request through Front Door shows X-Cache: TCP_HIT once the edge route is deployed.

Count views and egress

In the Front Door log workspace, AzureDiagnostics holds one row per request:

AzureDiagnostics
| where Category == "FrontDoorAccessLog"
| where requestUri_s has "/demi-search/documents/" and requestUri_s has "inline=1"
| summarize requests = count(), bytes = sum(tolong(responseBodyBytes_s)) by bin(TimeGenerated, 1d), cacheStatus_s

cacheStatus_s separates edge hits from requests that reached DEMI. A hit never reaches DEMI, so DemiEventsHourly_CL counts origin fetches only. Once the edge route serves documents, the full view count comes from this Front Door log.

To read document.download events, query DemiEventsHourly_CL in workspace demi-audit-<env>, as GET /admin/analytics does. DemiEvents_CL is an Auxiliary-plan table and summarize over it returns 0 rows. Prod recorded 95,012 document.download events in the 30 days to 2026-10-08.

Verified on test and prod

Checked with curl against https://test.projects.eao.gov.bc.ca/demi-search (2026-10-09):

• A conditional request (If-None-Match) at a cache miss goes to the origin. A 304 answer is not cached.
• A Range request at a miss stores the full object. Later requests without Range hit the cache.
• Front Door ignores Vary: Origin. Requests with and without Origin share one cache entry. DEMI sets no CORS header, so nothing leaks.
• If-Modified-Since or If-None-Match on a hit answers 304 from the edge.
• HEAD requests and attachment downloads are no-store and never cached.

Checked on prod (https://projects.eao.gov.bc.ca, 2026-10-09, after a 25-minute convergence wait):

• /api/public/document/<id>/download answers 302 to /demi-search/documents/<id>/download?inline=1.
• The first fetch of that URL is TCP_MISS, the second TCP_HIT.
• A Range request answers 206.
• redirect=1&inline=1 answers 302 to inline=1.
• Scraper user agents answer 302 to /robots.txt.

Cost of streamed downloads

Front Door Standard charges $0.083/GB to the client and $0.02/GB from the origin to the edge. Measured 2026-10-08 on prod: public documents average 2.78 MB (p95 9.7 MB), and 1.16% are over 32 MiB. Human-like traffic is about 6,000 download requests a day, about 17 GB a day, so about $55 a month before cache hits. Scrapers that get past the edge rules can add 50 GB on a burst day. Rerun the log query above for the actual volume before any prod change. Prices are from the Azure pricing page; the Canada zone is assumed to be Zone 1.

Data plane access and day-two tasks

Reaching the private Cosmos and Search data planes from the devbox, probing a change, writing code against the Functions dispatcher, reading audit, analytics and cost, and flipping prod search between backends are in Operating-DEMI-Data-Plane.

CI/CD

Every workflow, its trigger, identity and Azure roles is CI-Workflows. How a version is cut, tagged and shipped to production is Release-Process.

A merge to main redeploys test and restarts the Flex app container. Any detached job running inside it dies with the restart; a devbox job does not, because it runs on a separate VM. Resume a killed in-container job from its checkpoint; do not rerun from scratch.

Infrastructure has no CI path and is not meant to get one. scripts/deploy-infra.sh is hand-run — see "Deploy infrastructure" above.

See also

• Operating-DEMI-Data-Plane — devbox access, probes, dispatcher rules, audit and cost reads
• Environment-Reality-and-Operational-Gotchas — what exists, network reality, app settings, indexer and runtime gotchas
• CI-Workflows — every workflow, its trigger, identity and roles
• Release-Process — how a version is cut, tagged and shipped
• Azure-Environments — subscriptions, estates, Bicep, cost
• BC-Gov-Azure-Landing-Zone — the platform rules behind most of the above