Azure Environments

Azure environments

Subscriptions, infrastructure-as-code layout, and what DEMI costs. License plate c4b0a8: EPIC.AI.

What is deployed and how it behaves is Environment-Reality-and-Operational-Gotchas; deploying to it is Operating-DEMI-in-Azure.

Subscriptions

• Tenant: Government of BC (bcgov.onmicrosoft.com), 6fdb5200-3d0d-4a8a-b036-d3685e359adc

Environment	Subscription	ID	Role
dev	c4b0a8-dev - EPIC.AI	d2f8d048-2af3-44fd-81cc-858c040001f2	Owner
test	c4b0a8-test - EPIC.AI	7897ceb1-9a86-4639-87d7-7f9ff67142b3	Owner
tools	c4b0a8-tools - EPIC.AI	82efd4f0-7548-4fe6-8741-9e6c3297092f	Owner
prod	c4b0a8-prod - EPIC.AI	be5924ac-1083-4a1b-be92-7b444882cfd9	Owner

Env model: test = staging, dev = empty sandbox, prod = the full DEMI backend. Test holds the full DEMI estate (demi-*-test in c4b0a8-test-rg) alongside eagle-notify and eagle-public's storage and edge. c4b0a8-dev-rg holds one platform action group and nothing else; redeploy azure/main.bicepparam when a sandbox is needed.

demi-cosmos-test is the only copy of the extracted corpus, and the net is thinner than "periodic backups" suggests: Periodic, backupIntervalInMinutes 240, retention 8 hours, Geo redundancy — two restore points. A readable second copy of the text does sit in the demi-search-test chunks index (content is retrievable: true, stored: true), but that restore path has never been exercised. Re-extraction means the whole PDF corpus back through the off-platform GPU box, and a known set of documents already 404s in the object store. Treat any destructive operation on chunks as one-way.

The eagle-search migration and teardown are recorded in the eagle-dev-guides wiki, on Eagle Search Archive.

Prod estate

rg-demi-prod in c4b0a8-prod holds the full DEMI backend. The DEMI demo frontend (the React app deployed from this repo) is test-only; the eagle-demi-admin panel does deploy a static site into this group. eagle-public reaches DEMI through demi-apim-prod, the same shape as test.

Resource	Name	Note
Database	demi-cosmos-prod	Cosmos DB for NoSQL, serverless, keyless (disableLocalAuth), publicNetworkAccess: Disabled. Database demi; containers projects, documents, chunks, apikeys, boundaries, config — no wildfires, no userdata (test only)
API	demi-api-fc-prod	Flex Consumption (FC1), kind: functionapp,linux, functions api, apiRoot, reconcileEagle. VNet-integrated via snet-demi-func-fc1-prod. ENVIRONMENT=prod, Keycloak https://loginproxy.gov.bc.ca/auth realm eao-epic client eagle-admin-console. Timer reconcileEagle runs on RECONCILE_SCHEDULE (0 0 10 * * *), prod only
API gateway	demi-apim-prod	API Management Consumption, gateway https://demi-apim-prod.azure-api.net, fronts demi-api-fc-prod. Same two-API, product and subscription shape as demi-apim-test — see Environment-Reality-and-Operational-Gotchas
Key Vault	demi-kv-prod	Source of truth for the API's secrets; private endpoint pe-demi-kv-prod in c4b0a8-prod-cond-ext-pe-subnet — see Secret-Sync
Search	demi-search-prod	AI Search Basic, one replica, one partition, keyless, publicNetworkAccess: Disabled
Identity	demi-identity-prod	UAMI; holds Search Index Data Contributor on demi-search-prod, Log Analytics Reader on demi-audit-prod and demi-logs-prod, Cost Management Reader on rg-demi-prod, and its client id is the API's AZURE_CLIENT_ID. Same grants on demi-identity-test, scoped to c4b0a8-test-rg
Private endpoints	pe-cosmos-nosql-prod, pe-demi-search-prod, pe-demi-kv-prod	All in c4b0a8-prod-cond-ext-pe-subnet
Logging	demi-logs-prod, demi-insights-prod, demi-audit-prod, demi-audit-dcr-prod	Log Analytics, App Insights, audit workspace and its data collection rule
Alerts / budget	demi-alerts-prod, demi-budget-prod	Action group; monthly budget 400 CAD. Alerts demi-reconcile-drift-prod, demi-cosmos-ru-prod, demi-chunk-ingest-failures-prod and webtest demi-search-availability-prod fire into the action group
Storage	demistgproduvtikwlcqtpga	The Flex app's own storage account. No $web
Admin frontend	demiadmproduvtikwlcqtpga	Static website for eagle-demi-admin; origin demiadmproduvtikwlcqtpga.z9.web.core.windows.net, published behind Front Door endpoint demi-admin-prod-hfgebphjbucqd5bt.a01.azurefd.net. CI, roles and identity are CI-Admin-Frontend-Workflows

Both namespaces carry a demi-app-secrets (ADMIN_API_KEY, DOCLING_API_KEY, TRACK_CLIENT_SECRET, ROLE_SYNC_CLIENT_SECRET, NOTIFY_API_KEY, DEVBOX_SSH_PUBLIC_KEY, EDGE_SECRET), so nothing is exported by hand for a prod infrastructure run any more. Neither environment has a legacy App Service: demi-api-test/demi-plan-test and demi-api-prod/plan-eagle-search-prod are deleted, and access to the private data planes goes through a devbox VM instead — see Environment-Reality-and-Operational-Gotchas and Operating-DEMI-Data-Plane. Prod's devbox is demi-devbox-prod (subnet snet-servers, Bastion bastion-prod). rg-eagle-search-prod holds only eagle-search-identity-prod.

demi-cosmos-prod's projects, documents and chunks containers are a copy of test's, made with src/scripts/copy-to-env.js over the private endpoint pe-demi-cosmos-prod-copy in the test group (test's own corpus comes from prod Eagle). apikeys and boundaries are not copied. Anonymous counts on demi-api-fc-prod: Project 348, Document 61,587 — measured 2026-08-26, the same as test.

Search is shared with eagle-search rather than DEMI-only: demi-search-prod holds eagle-search's indexes (eagle-chunks, eagle-documents, eagle-projects, served by eagle-search-api-prod) alongside DEMI's own set (chunks, projects, documents), fed by indexers chunks-indexer, documents-indexer, projects-indexer on a PT5M schedule and datasources demi-chunks-ds, demi-documents-ds, demi-projects-ds reading demi-cosmos-prod over the shared private link demi-cosmos-prod-link. The indexers run as eagle-search-identity-prod, which holds Cosmos Data Reader and the control-plane Cosmos DB Account Reader Role on demi-cosmos-prod, plus Search Index Data Contributor on demi-search-prod.

Bicep

azure/
├── main.bicep                 # root orchestrator
├── main.test.bicepparam       # staging (test) params
├── main.prod.bicepparam       # prod params
├── ai-search.prod.bicepparam  # params for the standalone demi-search-prod deploy
├── search/                    # committed index, indexer and datasource JSON
└── modules/
    ├── identity.bicep            # user-assigned managed identity
    ├── cosmos-nosql.bicep        # demi-cosmos-<env> + private endpoint
    ├── ai-search.bicep           # demi-search-<env> + private endpoint
    ├── search-existing.bicep     # role grants + shared private link on a search service this template does not own
    ├── api-function-flex.bicep   # demi-api-fc-<env>, Flex Consumption (FC1) — the only API
    ├── devbox.bicep              # demi-devbox-<env>, dev-access VM — data-plane access to Cosmos/AI Search
    ├── key-vault.bicep           # demi-kv-<env> + private endpoint; secret names in [[Secret-Sync]]
    ├── apim.bicep                # demi-apim-<env>, Consumption tier, fronts the Flex app — see Environment-Reality-and-Operational-Gotchas
    ├── static-site.bicep         # demiweb<env><hash> storage account, $web origin for the frontend
    ├── observability.bicep       # Log Analytics, App Insights, alerts
    ├── audit-logs.bicep          # audit workspace + data collection rule
    ├── foundry.bicep             # demi-foundry-<env> + private endpoint
    ├── cost-budget.bicep         # demi-budget-<env> monthly
    ├── document-storage.bicep    # Azure Blob, Phase 3b — written, not deployed
    └── extractor.bicep           # unreferenced by main.bicep

main.bicep always wires identity, cosmos-nosql, observability, audit-logs, key-vault, api-function-flex and cost-budget. Five modules are conditional: ai-search on deploySearch, search-existing on its inverse, document-storage on deployDocumentStorage, foundry on deployFoundry, and static-site on deployStaticSite (prod deploys none of these five except search-existing). Two more: devbox on deployDevbox with a non-empty devboxSubnetId, true in both environments; and apim on deployApim, true in both main.test.bicepparam and main.prod.bicepparam, always targeting the Flex app's host name. extractor is not referenced at all.

The frontend is not an App Service. The bundle is blobs in the $web container of a storage account, published through a shared Azure Front Door Standard profile, eagle-edge-<env> — the CloudFront-to-S3 pattern. Two things follow and both bite:

• The profile lives in eagle-edge, not here. This template owns only the origin. Its frontendStaticSiteHostName output is what eagle-edge's demiFrontendWebHostName parameter wants, and that is what creates DEMI's route. It must be the static-website endpoint (<account>.z9.web.core.windows.net), not the blob endpoint: only the web endpoint serves index.html for a directory path.
• Azure Static Web Apps is not an option. Microsoft.Web/staticSites has no region in common with the enforced Resource-Locations deny on bcgov-managed-lz-live (CanadaCentral, CanadaEast, global only). Separately, an SWA linked backend locks the linked App Service to SWA-proxied traffic with a 45 s request cap, which is the opposite of what demi-api needs.

Containers cannot be created by the app's managed identity — container creation is control-plane, and the data-plane token gets 403 ... cannot be authorized by AAD token in data plane. Containers come from ARM or Bicep only.

main.prod.bicepparam deploys through scripts/deploy-infra.sh prod, which runs what-if by default and needs CONFIRM_PROD=yes with --live to apply. It sets deploySearch=false with existingSearchEndpoint (module search-existing.bicep, pointing at the shared demi-search-prod), deployEnrichment=false, deployFoundry=false, deployStaticSite=false and deployApim=true. App code is not part of this pipeline — see Operating-DEMI-in-Azure.

scripts/deploy-infra.sh splits a run into a foundation layer and an application layer, applied by separate flags; see "Deploy infrastructure" in Operating-DEMI-in-Azure.

demi-search-prod itself is deployed separately from azure/ai-search.prod.bicepparam, which pins semanticSearch: 'disabled' — the free tier latches off after a 402 and silently drops that worker to BM25.

Edge gate

Setting EDGE_GATE on demi-api-fc-<env> (Bicep param edgeGate) refuses requests that did not come through Front Door. The check is in dispatch in src/http/router.js, after the route match.

EDGE_GATE	Behaviour	Set on
empty	Off	Prod, until the rollout
log	Serves the request and writes one warn line	Test
enforce	Answers 403

A request passes when it came through the APIM gateway (the gateway secret is present) and also has one of: the Front Door edge secret, a non-empty X-APIM-Subscription (eagle-api's keyed /machine push), or an X-Api-Key header (the app then authenticates the key). Paths under /health always pass. Function admin endpoints and timers do not go through dispatch.

In log and enforce mode a refused request writes [demi-api] edge gate: request did not come through Front Door with evt: 'edge-gate', path and reason. reason is no-gateway (the request skipped APIM) or no-edge (it came through APIM without an edge secret, subscription or key). APIM Consumption keeps no gateway logs, so this line is the only count of direct traffic:

AppTraces
| where Message has "edge gate: request did not come through Front Door"
| summarize count() by tostring(Properties.path), tostring(Properties.reason)

Run it on demi-logs-<env>. Every row should be a known caller.

Cost

Month-to-date, measured 2026-08-26 over 2026-08-01 onward, in CAD.

c4b0a8-test-rg — about 210 CAD so far this month. The group is shared, so eagle-notify and eagle-public contribute to the App Service, Storage and Front Door lines.

Service	MTD (CAD)
Azure Cognitive Search	90.95
Microsoft Defender for Cloud	54.94
Virtual Network	21.48
Azure Front Door Service	16.15
Azure App Service	15.83
Azure Cosmos DB	5.51
Azure Database for PostgreSQL	2.91
Log Analytics	1.03
Storage	0.75
Everything else	under 0.40

rg-demi-prod — about 20 CAD so far this month: Azure Cognitive Search 18.45, Virtual Network 1.83. The API is Flex Consumption, billed per execution rather than a reserved plan.

Two lines are worth reading carefully:

Defender for Cloud is roughly a quarter of the test-group bill. Fourteen plans sit on Standard (VirtualMachines, SqlServers, AppServices, StorageAccounts, SqlServerVirtualMachines, KeyVaults, Arm, OpenSourceRelationalDatabases, CosmosDbs, Containers, CloudPosture, AI, Discovery, FoundationalCspm). This is a BC Gov landing zone and Defender tiers are set by platform or security policy rather than by this team. Ask the platform team before changing any plan other than the storage override below, and treat it as a question rather than a saving to take.

Defender for Storage is off on all 13 storage accounts in c4b0a8-test-rg, through a per-account override (measured 2026-10-14). An account that inherits the subscription plan costs about 0.63 CAD a day. Every storage module in azure/modules/ declares the override (Microsoft.Security/defenderForStorageSettings, isEnabled: false), so a deploy of a module sets it on that module's account. Defender for Cloud cost about 7 CAD a day across the group; the saving is not measured (read the Defender for Cloud meter in Cost Management after 2026-10-04).

The 5 accounts in rg-demi-prod have no override and inherit the plan (measured 2026-10-14). Defender goes off on an account when its module is next deployed: the foundation deploy covers demifc… and demipt…, the document-backup deploy covers eaglebak…. demistg… and demiadm… stay on until set by hand. The eagle-edge prod accounts, in another resource group, carry the override.

The intake account demiin<env>… is meant to be the only account with Defender on, with a monthly scan cap and a 2 GB size filter. Its cost is in Document-Intake and the decision in ADR-015-Malware-Scanning-on-Intake.

The Virtual Network line is per private endpoint, not a single charge. Test carries four and prod two, all load-bearing, so this does not shrink.

AI Search Basic is the only controllable line, and dropping it means losing fuzzy search — see ADR-005-Cosmos-Full-Text-Search-Ruled-Out for what the cheaper option actually delivers.

Budgets: demi-budget-<env> is a monthly anomaly guard, resource-group scoped, so it sees one group and not the subscription. Test is 550 CAD (budgetAmount in azure/main.test.bicepparam); prod is 400 CAD (azure/main.prod.bicepparam). The annual ceiling is epic-ceiling, 50,000 CAD/yr, scoped to management group c4b0a8 — it covers dev, test, tools and prod together. Notification only, neither budget blocks spend.

The budget catches an overspend after 24-48 hours of billing lag. demi-cosmos-ru-<env> (azure/modules/cosmos-alerts.bicep) checks every 15 minutes instead: it fires when Cosmos used more than cosmosRuPerHourAlert request units in the last hour. Serverless Cosmos in canadacentral lists at 0.3812 CAD per million RU. Test and prod both threshold at 3,000,000 RU/h — about 1.1 CAD/h, 27 CAD/day. demi-chunk-ingest-failures-<env> is the companion alert on the cause — see Environment-Reality-and-Operational-Gotchas and Extraction-Pipeline.

Reading the cost

GET /api/admin/cost on demi-api runs the same Cost Management query on demi-identity-<env>'s own credential, month to date, scoped to the resource group, cached one hour per API instance. This is the path for a caller with a DEMI admin role and no Azure portal access — the admin panel's Azure spend screen. budget in the response is null when BUDGET_NAME is unset.

To query Cost Management directly instead:

az rest --method post \
  ".../providers/Microsoft.CostManagement/query?api-version=2023-03-01"

grouping on ServiceName. az consumption usage list does not work here — it returns pretaxCost as the literal string "None" on these subscriptions.

See also

• Environment-Reality-and-Operational-Gotchas — what is deployed, and how it behaves
• Operating-DEMI-in-Azure — deploying, restarting and verifying
• Operating-DEMI-Data-Plane — devbox access, probes, audit and cost reads
• BC-Gov-Azure-Landing-Zone — platform constraints, private DNS, tagging, AI Services Hub
• Secret-Sync — Key Vault, the secret-sync Function app, required secret names