Repository navigation
API Gateway
demi-apim-<env> is the Azure API Management (APIM) Consumption instance in front of the Flex app demi-api-fc-<env>, in test and prod. Its Bicep module is azure/modules/apim.bicep, gated by deployApim (true in main.test.bicepparam and main.prod.bicepparam). The estate around it is in Environment-Reality-and-Operational-Gotchas.
The gateway buys metering and a rate limit, not a security boundary. Consumption has no VNet, so demi-api-fc-<env>.azurewebsites.net stays publicly reachable and demi-api authenticates every caller itself.
Two APIs share the one backend, in both environments.
| API | Path | Caller | Requires |
|---|---|---|---|
demi-api |
/api |
the browser | nothing at the gateway |
demi-machine |
/machine |
server-to-server |
Ocp-Apim-Subscription-Key, product machine, 300 calls per 60 s per subscription |
Product machine holds the subscriptions eagle-api and engage in both environments. Their keys are read from APIM's control plane, never from this repo.
The global policy deletes any client-supplied X-Gateway-Secret, X-APIM-Subscription and X-Client-Ip, then sets its own before forwarding:
| Header | Value |
|---|---|
X-Gateway-Secret |
the named value gateway-secret, from Key Vault secret apim-gateway-secret in demi-kv-<env>
|
X-APIM-Subscription |
the resolved subscription name |
X-Client-Ip |
the address APIM saw the request arrive from |
demi-api trusts these headers only when the secret matches app setting APIM_GATEWAY_SECRET (a Key Vault reference). The caller's identity then resolves through an apikeys row apim:<subscription-name> with no key material, minted with POST /api/admin/api-keys and id: "apim:<name>".
- The test DEMI frontend reaches it same-origin: its Front Door endpoint routes
/apiand/api/*to the gateway (eagle-edge, gated bydemiApimDeployed), so/api/configreportsAPI_LOCATIONempty. Prod has no DEMI frontend. - eagle-public's
/demi-searchroute on eao-nginx proxies to the gateway in both environments (demiorigin invalues-test.yamlandvalues-prod.yaml). - eagle-api's push into DEMI (
api/helpers/demiPush.js) callsdemi-machinewith theeagle-apisubscription key inOcp-Apim-Subscription-Key(envDEMI_APIM_KEY).DEMI_API_BASEishttps://demi-apim-<env>.azure-api.net/machine(eagle-apihelm/eagle-api/values-*.yaml), so the 300 calls per 60 s limit covers every push. - The extraction host does not use the gateway. It posts chunks straight to the Function App host with its own key, because APIM Consumption caps a request at 30 s and NDJSON ingest can run longer.
Front Door (eagle-edge, profile eagle-edge-<env>) terminates every connection one hop outside APIM. Without a check, the address APIM asserts is Front Door's own egress address, shared by every visitor.
- Rule sets
rulesbackend(the/demi-search/*and/demi-projects/*passthrough) andrulesapi(the DEMI frontend and admin/api/*routes) each carry a rulestampedgesecret. It overwrites the request headerX-Edge-Secretwith a shared value and drops whatever a client sent. -
demi-apitrusts the header only when it matches app settingEDGE_SECRET, with a constant-time compare (same helper as the gateway secret). A match lets it readX-Azure-SocketIP, the peer Front Door accepted the connection from, as the caller's address. -
X-Azure-ClientIPis never read: Front Door derives it from the caller's ownX-Forwarded-For, so any browser can pick it. - An empty
EDGE_SECRETis the off switch:demi-apiignores the header and keys the anonymous bulk quota on the address APIM asserts.
The value is one shared secret per environment.
| Side | Where it lives |
|---|---|
| DEMI | Key Vault secret edge-secret, set by hand from the environment's devbox, read by demi-api-fc-<env> as a reference (see Secret-Sync). Both param files name it in optionalSecretNames. |
| eagle-edge | No Key Vault. scripts/deploy-infra.sh reads EDGE_SECRET from the environment, falling back to OpenShift secret demi-app-secrets in 6cdc9e-test for test only; prod needs it exported. GitHub Actions reads the environment secret EDGE_SECRET. |
Rotate both sides together; a mismatch only drops callers back to the APIM path. Deploying eagle-edge with an empty value leaves an existing stampedgesecret rule in place with its old value, because the deploy is Incremental. Removing the header takes an explicit az afd rule delete.
src/utils/caller-ip.js feeds the bulk-download quota key, the request log and the audit trail. It walks X-Forwarded-For from the rightmost hop and never takes the first entry. A caller can prepend any value to that header, so only the hop each proxy itself appended is trustworthy.
Which hop wins, and the TRUSTED_PROXY_IPS case where the OpenShift rproxy in front of eagle-public is skipped: Bulk-Download "Access rules".
- Environment Reality & Gotchas
- Architecture
- Seeding — filling Cosmos from Track and Eagle
- Frontend — React shell: screens, routes, sign-in gate
- Sources and Status — every source, one state
- Search & Retrieval — overview
- Search Index Reference
- Search Query Construction
- Deep Search Queries — passages, chunk filters, degraded modes
- Native List Reads — native list endpoints and mirror list reads
- Chunk Parent Fields — parent fields copied onto chunks
- Chunk Parent-Field Backfill — roll the fields out, repair
- Search Measurements
- Extraction Pipeline
- Extraction Quality — coverage, word-joining, tiling
- Identity Model — which id means what
- Attribute-Level Access — field-visibility levels, catalog
- Sync Architecture — how records get in
- Eagle Mirror Reference — mirrored kinds, refusals, drift classes
- ENGAGE Rows Reference — ENGAGE-owned comment periods, reconcile
- Public-Read Backfill — filling and repairing the mirror containers
-
Short Links —
/s/{code}redirect and/api/links -
Recent Uploads —
/api/documents/recent-uploads, eagle-public homepage panel - Reports — comment export and BCGW CSV reads
-
User Data — saved map areas, preferences,
/meroutes -
Bulk Download —
/api/bulk-downloads, zip jobs - Bulk Download: Operations — enable, poison-queue runbook
- Document Intake — upload link, malware scan, release
- Connecting an Application to DEMI
- Azure Environments
- API Gateway — APIM, Front Door edge secret, caller IP
- Document Backup — archive copy of originals, run and check
- Document Backup: Revert and Drill — put documents back from the archive
- PDF Titles — public name written into stored PDFs, run, restore
- Secret Sync — Key Vault, secret-sync Function app, required names
-
Operating DEMI in Azure — deploy, verify
- Operating the DEMI Data Plane — devbox access, probes, audit and cost reads
- Edge ban detector — hourly timer, modes, kill switch, soak and promote
-
CI Workflows — every GitHub Actions workflow, trigger, identity, roles
- CI Deploy Workflows — eagle-demi staging and production deploys
- CI Admin Frontend Workflows — eagle-demi-admin checks and deploys
- Release Process — cutting, tagging, shipping and rolling back a version
- Runbook: Search Outage — index/app schema drift, diagnose, fix, roll back
- BC Gov Landing Zone
- Business Rules
- EPIC.search Business Rules — where DEMI differs from EPIC.search
- 001: Typesense for DFL Search — superseded
- 002: Async Processing (MongoDB Queue) — superseded
- 003: Rule-Based Classification
- 004: Read ACL Authorization Model
- 005: Cosmos Full-Text Search Ruled Out — rejected
- 006: AI Summarization over BM25
- 007: Service-to-Service Credentials
- 008: Audit and Usage Analytics Store
- 009: Track Feed
- 010: Attribute-Level Access Control
- 011: Public Read Surface
- 012: Public Config Surface
- 013: Stored Project Summaries — proposed
- 014: Sync-out to Eagle
- 015: Malware scanning on intake
- Developer Brief: Digital File Library
- Developer Brief: AI Classifier Tool
- Developer Brief: EPIC System Integration
- Managing Government Information Policy 2022
- Guidelines on Documenting Government Decisions 2019
- AICT Guidelines
- Per-Document Data Format
- Per-Document Generator
- Document Types Draft
- DFL Reference Metadata
- Act and Legislation Full Text
- ORCS Filing Reference Sheet