Architecture

Architecture

DEMI is EPIC's central data store for projects and documents. It holds a merged model over epic.track, the master project registry, and eagle, which carries richer EA-process fields. demi-api serves that model from Cosmos DB for NoSQL, with Azure AI Search over the same data.

Which id means what is Identity-Model. How records arrive, and what a visibility change cascades, is Sync-Architecture. Which resources exist in which subscription is Azure-Environments.

Stack

Layer	What runs
API	demi-api-fc-test, demi-api-fc-prod. Node 22, Azure Functions Flex Consumption, behind API Management (demi-apim-<env>). See Environment-Reality-and-Operational-Gotchas
Database	demi-cosmos-test, demi-cosmos-prod. Cosmos DB for NoSQL, serverless. Keyless (disableLocalAuth: true), reachable only through a private endpoint
Search	demi-search-test, demi-search-prod. Azure AI Search Basic, lexical BM25. See Search-and-Retrieval
Object store	nrs.objectstore.gov.bc.ca, S3 API through the MinIO client. STORAGE_BACKEND=minio
Identity	Keycloak for users; Entra managed identity for app-to-Azure
Frontend	React demo app, published to a Storage static website behind Azure Front Door. Test only (deployStaticSite = false in prod)

Sources and the merge model

Source	Content
Track (src/data/track_projects_enriched.json, checked in)	382 project records. track_project_id is the master project id. There is no live Track feed — see Sync-Architecture
Eagle (/api/public/search on eagle-api)	Projects, documents, and a 213-item List lookup for type, milestone and projectPhase

buildRegistry (src/merge/project.js) joins the two on epic_guid, the Eagle project _id: 348 matched, 28 Track rows carrying no epic_guid, 6 whose epic_guid points at nothing, and 10 Eagle projects with no Track row — 392 projects. An Eagle project with no Track counterpart is retained, keyed eagle-<eagleId> and flagged sourceSystem: 'eagle'. Nothing is dropped.

The anonymous corpus is 348 projects and 61,587 documents (measured 2026-08-26, identical in test and prod).

Merge rules are pure functions with no I/O, so every rule is tested as data. Identity-Model is the authority on the field meanings.

• Track wins and Eagle fills gaps, through TRACK_PRECEDENCE, an explicit [target, trackField, eagleField] map rather than {...eagle, ...track}. A spread overwrites with undefined and would erase real data: 12 Track records have no abbreviation.
• isPublished mirrors the read[] ACL and is never an independent signal, on both the Track and the Eagle-only path.
• buildProjectIndex(projects).resolve(ref) returning null is load-bearing. A row whose parent does not resolve is dropped, never given a fabricated parent. That is what replaces fuzzy name matching.
• Raw payloads are retained unindexed under sources.track and sources.eagle, so a re-merge never re-fetches upstream and any field is traceable to its origin. The API never reads them.
• Coordinates are GeoJSON [longitude, latitude] throughout, with no axis swap. 7 of 382 Track records carry a positive longitude, a dropped minus sign; validCoordinates negates and re-validates against a BC bounding box, recovering 6. Sparwood Wells #04 (lat 45.861, lng 53.354) has both values wrong and gets no centroid rather than an invented one.

Containers

The table lists the containers this page names. azure/modules/cosmos-nosql.bicep declares all of them, each owned by a module under src/repositories/.

Container	Partition key	Why
projects	/id (Track id)	Point reads or full lists; no grouping dimension exists
documents	/projectId	?project=X is the dominant list, so single-partition
chunks	/documentId	Delete-then-reinsert stays single-partition
boundaries	/type	The only filter that exists. 281 rows: 28 regional districts, 160 municipalities, 93 electoral districts
wildfires	/id	Spatial index, 7-day TTL. The sync re-upserts every fire still in the DataBC feed, so anything that drops out expires itself
apikeys	/id	Service-credential store, see ADR-007-Service-to-Service-Credentials
config	/id	Runtime config documents
userdata	/userId	One partition per user; every read and write of a user's own data is single-partition. See User-Data
users	/id	Eagle user mirror. The partition key is also the scope field, so a project-scoped key reads no user
groups	/projectId	Eagle project contact groups. Indexes projectId
inspections	/inspection	Eagle inspections, elements and items, told apart by kind. One inspection and everything under it is one partition. Indexes kind, projectId, element

boundaries and wildfires are enrichment data and carry rows in test only; prod serves Eagle and Track data alone. Sources-and-Status owns that contract.

Indexing excludes /* and includes only filtered paths. Two are load-bearing: /read/[]/? must be indexed or every ACL read is a full scan, and /name must be indexed or ORDER BY fails outright.

Cosmos NoSQL caps an item at 2 MB. Boundaries reach 1.58 MB (Peace River Regional District; 9 exceed 1 MB); everything else is under 6 KB.

Seeding

How the containers are filled from Track and Eagle, and the gates that stop a bad seed, is Seeding.

Authorization

Two orthogonal dimensions. ADR-004-Read-ACL-Authorization-Model has the full rationale.

read[] holds role types only (public, sysadmin, staff, demi-admin, demi-service-read), so it stays bounded and indexed. Project scope rides the partition key instead, which is already the partition boundary and so costs nothing extra.

project:207        -> scoped to project 207 (a canonical project id = the partition key)
staff, compliance  -> role TYPES, matched against read[]

There are three access tiers: privileged, scoped and public. rolesFor() strips project:* from the role list so a project id can never land in the read[] IN clause. No project role at all means not scoped (public tier), which is distinct from an explicit projectScope: [] meaning scoped to nothing (scopeClause → false). Privileged roles ignore scope entirely. Scope values are ids rather than names, which keeps resolveAccess synchronous and lookup-free on every request.

readClause(roles) in src/helpers/access-sql.js is the only place a visibility predicate is built:

(EXISTS(SELECT VALUE r FROM r IN c.read WHERE r IN (@role0, @role1))
 OR ((NOT IS_DEFINED(c.read) OR ARRAY_LENGTH(c.read) = 0) AND c.isPublished = true))

EXISTS-with-subquery, not ARRAY_CONTAINS_ANY — the latter does not use the index.

The unsetIsPublic option drops the isPublished half of the second branch, so a row carrying neither field is visible. One container passes it: boundaries (src/repositories/boundaries.js), whose seeded rows carry no ACL at all. A boundary given an explicit read[] is still gated by the first branch.

systemAccess() is the one context that reads every item regardless of ACL. It is built from the normal privileged tier and resolves to true through readClause rather than being a bypass flag, so the SQL-asserting tests cover it. It takes no arguments, so it can never be derived from a request.

Database-level ACLs driven by Keycloak are not available: Cosmos NoSQL data-plane RBAC is Entra-only and its finest scope is a container, and resource tokens are key-derived (incompatible with disableLocalAuth), expire in 1–5 hours, and cannot express role types. The browser never touches Cosmos, so the API is the trust boundary.

The double gate

A document is public only when its own read[] contains public and its parent project's does. The two are kept in step by denormalisation, not by a join: a project visibility flip re-derives read[] and cascades it into that project's documents, intersecting rather than overwriting (src/controllers/nosql/project.js, documents.setAclForProject). The seed enforces the same invariant before it writes, and a document that is public under a non-public project fails the gate. Publishing a document under an unpublished project returns 409.

Chunks are gated by their parent document, resolved at query time: the chunk search path reads the parent documents under the caller's own access and drops any chunk whose parent did not come back (src/controllers/search.js). A chunk's own read[] is an ingest-time snapshot that may lag, so it can never leak text on its own. Sync-Architecture explains why eagle-search's re-stamping machinery must not be ported here.

Data access

There is no Mongo-to-SQL translator. The operators where the two disagree ($ne, $exists, $size) are exactly what a read filter is built from, and there are only about 12 distinct query shapes in the whole application.

query(name, spec, opts) throws unless the spec is {query: string, parameters: array}. Parameters only, never interpolation.

• No generic find(filter). Each method owns its SQL and cannot emit an unfiltered read.
• countWhere shares selectWhere, so a count can never drift from its list predicate and leak the size of a set the caller cannot read.
• Paging uses continuation tokens, not skip/take. Cosmos has no efficient offset. An anonymous caller is capped at pageSize 100 and refused above it; any authenticated caller may ask for up to 1000.
• patch() is capped at 10 operations by Cosmos and guarded. Use it for partial updates; upsert() replaces the whole item and will erase fields written by another path.
• Point reads bypass the query predicate, so canRead() is mandatory after readItem().

In the controllers: a hidden project returns 404 rather than 403, since a 403 would confirm the id exists. Update paths refuse to reassign a partition key (id on projects, projectId on documents, type on boundaries) because in Cosmos that is a delete-and-reinsert, not an update. And resolveDocumentAcl() is used by both document write paths, so an intake upload cannot be published under a private project.

Layering

Writes served from controllers/native follow one import direction: routes.js -> controllers/native/<kind>.js -> domain/<kind>.js -> repositories/. No route uses the path yet. Organization create and update are the first planned.

• Route: decides who may call, by role only. The guard admits the roles eagle-api lists for the route in its x-security-scopes.
• Controller: an HTTP adapter. writeHandler(domainFn) resolves the caller's access, calls the domain function, maps its status to a response (200 or 201, 400, 403, 404, 409, or 503 when an etag race is lost after retries), and redacts the saved row. A controller that imports domain/ never writes res.status or res.json and never imports db/.
• Domain: the business rules. A function checks the body with _schema.check, loads the row through a repository with the caller's access (an unreadable row is not found), refuses fields the caller may not write with refusedWriteKeys, applies its named rules, then writes with writeRow. It never imports http/ or db/.
• Repository: storage only. It never imports the domain, the controllers, utils/audit or vis/redact.

_schema.check compares a body to an allowlist of typed fields (string with a maximum length, boolean, enum, date, id, array of ids). An unknown key is an error. It returns {status: 'ok', value} or {status: 'invalid', fields} with every field error at once.

writeRow writes through the etag-guarded retry loop. It returns {status, value, created}, writes one audit event when the save succeeds, and throws if the caller gave no audit description.

Authorization sits in three places. The route guard checks the role. Record-level access is resolveAccess in the controller and the access-filtered repository read in the domain. Which fields a caller may change on a row is refusedWriteKeys in the domain. There is no separate authorization layer.

Two tests keep the layers apart. test/domain/layering.test.js scans the source text for imports and res calls that break the rules above. test/controllers/route-inventory.test.js reads one inventory file per object under test/controllers/route-inventory/. Every non-GET route needs an audit action or a reason it is exempt. Ported routes also carry the eagle-api roles, and the test checks they match the route guard.

The per-object tracker, with the eagle-api rules for each object, is docs/eagle-api-port.md in the eagle-demi repository.

Deletion semantics

Action	What it does
Unpublish (PUT /documents/:id/published)	isPublished: false, read[] loses public, the document's AI Search ACL is rewritten, and its chunks are patched to the same ACL. This is the hide mechanism
Hard delete (DELETE /documents/:id)	Removes the Cosmos item, its chunks, and both the document and chunk entries in the search index
The stored blob	Never deleted by any request path. The delete response says so explicitly

Index entries are removed directly rather than through the change feed, which emits no deletes in latest-version mode. Doing it directly is what makes a soft-delete marker unnecessary. Index removal is best-effort — the record is already gone from Cosmos, so an index failure must not turn a successful delete into a 500, and the response reports the outcome explicitly.

Object storage

src/storage/ is the single entry point and exposes two operations, because that is all the application does with stored files:

getDownloadUrl(key, opts)       -> short-lived URL   (download endpoint, 5-minute default expiry)
putFile(key, filePath, ctype)   -> stored key        (upload)

No bucket, container or client escapes the module. The backend is chosen by an explicit STORAGE_BACKEND (minio | azure) and an unknown value throws at load; it is never inferred from whichever credentials happen to be present. The MinIO backend owns the environment key prefix, so no caller can forget it.

The Azure Blob backend and azure/modules/document-storage.bicep are written and not deployed (deployDocumentStorage = false).

	MinIO	Azure Blob
Auth	access key + secret	Entra managed identity, no keys (allowSharedKeyAccess: false)
Environment isolation	one bucket, nested key prefix	one container per environment
Download URL	presigned GET	user delegation SAS, sp=r, https-only
Container creation	on demand	never — comes from Bicep

Three Azure Blob traps:

• Storage Blob Delegator is required and is not implied by Storage Blob Data Contributor. Without it getUserDelegationKey fails, and with shared-key access disabled a user delegation SAS is the only way to sign a download link, so every download breaks.
• The delegation key's signedStartsOn and signedExpiresOn must be Date objects. The generated mapper types them as String, but generateBlobSASQueryParameters calls toISOString() on them.
• The delegation key is cached for 30 minutes (it is valid up to 7 days). Uncached, every download adds a round trip; cached too long, it silently produces SAS URLs that fail authentication.

src/scripts/copy-blobs-to-azure.js copies MinIO objects across. Dry run by default, resumable, and it never imports the MinIO write operation — a test greps the source to keep it that way.

Map boundary layers

Boundary polygons are pre-generated static assets under frontend/public/assets/geojson/, not live WFS calls: loading the three layers from BC OpenMaps WFS costs 3–8 s per layer and hits rate limits. scripts/export-topological-boundaries.js generates them, and load time is under 30 ms. Municipalities are 552 KB, regional districts 1.13 MB, electoral districts 1.33 MB.

Each layer is simplified through Mapshaper's shared topological arc graph (-clean -simplify visvalingam keep-shapes), never polygon by polygon. Neighbouring polygons store shared edges in reverse vertex order, so downsampling them separately breaks shared boundaries and leaves overlaps and slivers.

The seeder also reads these files, which is why scripts/package-api.py re-includes that one path out of the otherwise-excluded frontend/ tree. Without it the seed fails with ENOENT on regional_districts.geojson in Azure only, never locally.

See also

• User-Data — the /me routes, saved map areas and preferences
• Seeding — filling the containers from Track and Eagle
• Identity-Model — which id means what
• Sync-Architecture — how records arrive, and what a visibility change cascades
• Environment-Reality-and-Operational-Gotchas — what exists in Azure, and the traps
• Search-and-Retrieval — AI Search configuration and the retrieval measurements
• Extraction-Pipeline — how documents become chunks
• ADR-004-Read-ACL-Authorization-Model — the read-ACL authorization decision