Skip to content

Eagle DEMI

Jump to bottom
Daniel Truong edited this page Oct 14, 2026 · 24 revisions

Eagle DEMI

DEMI (Document Extraction & Machine Intelligence) is EPIC's central data store. demi-api is the master directory for projects, documents, chunks and administrative boundaries, and it serves the search that eagle-public uses.

DEMI runs on Azure, not OpenShift: Azure App Service for the API, Cosmos DB for NoSQL for the data, Azure AI Search for the indexes, and the NRS object store for files. Cosmos and AI Search are reachable through private endpoints only.

Repository: digitalspace/eagle-demi. Its own wiki owns the detail; this page covers what an EPIC developer needs to know from the Eagle side.

Where it fits

flowchart LR
    Track["epictrack-api<br/>master project registry"]
    EagleAPI["eagle-api<br/>MongoDB"]
    DEMI["demi-api"]
    Cosmos[("Cosmos DB for NoSQL")]
    Search[("Azure AI Search<br/>demi-search-*")]
    Store[("NRS object store")]
    Public["eagle-public"]

    Track -->|"static snapshot, checked in"| DEMI
    EagleAPI -->|"PUT /api/eagle/* on every write"| DEMI
    DEMI --> Cosmos
    DEMI --> Search
    DEMI --> Store
    Public -->|"/demi-search/search"| DEMI
Loading

Project identity comes from Track: a DEMI project's id is the Track project id as a string. The Eagle _id is kept on the record as eagleId so Eagle URLs still resolve. An Eagle project with no Track counterpart is kept, not dropped: it gets id of eagle-<eagleId>, a null trackProjectId and sourceSystem: 'eagle'.

eagle-api pushes every write to DEMI. DEMI writes into Eagle only for ENGAGE-owned comment periods, through its sync-out worker and eagle-api's protected /api/commentperiod endpoints. Nothing writes Eagle's MongoDB directly. See ENGAGE–EAGLE Integration for the flow and the DEMI wiki page Sync-Architecture and ADR-014-Sync-Out for the worker.

Environments

Environment Resource group API Search service
Test c4b0a8-test-rg demi-api-test.azurewebsites.net demi-search-test
Prod rg-demi-prod demi-api-prod.azurewebsites.net demi-search-prod

eagle-public reaches demi-api through rproxy at /demi-search/search, which keeps the browser same-origin. SEARCH_API_PATH in the eagle-api Config document selects it: /demi-search in both test and prod. See Eagle Search.

That one URL carries the Project, Document and DocumentChunk searches. eagle-public reads everything else from eagle-api: the project record, participating Indigenous Nations, comment periods, recent activities, dropdown lists, and document downloads.

/config/public serves DEMI's live mirror of eagle-api's /api/config, reachable through rproxy at /demi-search/config. eagle-api keeps it current with PUT /eagle/config/public, fired whenever a read of /api/config sees the served payload change. It answers 503 when the document has not been seeded rather than a default payload. See Configuration Management for the fetch order and push mechanism, and the eagle-demi wiki Seeding page for the bootstrap command.

API

demi-api mounts its router at both /api and /. Reads are open and filtered by ACL; every mutation is gated.

Endpoint Method Access
/api-docs, /health GET Open
/config GET Open
/config/public GET Open
/search GET ACL-filtered
/search/summary GET Privileged
/projects, /projects/:id GET ACL-filtered
/projects, /projects/:id POST, PUT, DELETE Write
/documents, /documents/:id GET ACL-filtered
/documents/:id/download GET ACL-filtered, presigned URL
/documents, /documents/:id POST, PUT, DELETE Write
/documents/extract POST Write
/documents/:id/published PUT Write
/documents/:id/chunks POST Write
/boundaries, /boundaries/:id GET ACL-filtered
/boundaries, /boundaries/:id POST, PUT, DELETE Write
/eagle/projects/:eagleId, /eagle/documents/:eagleId PUT Write
/eagle/config/public PUT Write
/db/stats, /admin/index-progress GET Privileged
/admin/sync/wildfires POST Write
/admin/api-keys, /admin/api-keys/:id GET, POST, DELETE Write

ACL-filtered reads run passive authentication: a valid credential populates the caller's identity, anything else continues as anonymous, and visibility is decided by the record's read[] array rather than by rejecting the request. A rejected credential is logged, so a forged or expired token is distinguishable from a logged-out visitor.

Authentication

Two credential types reach the same identity shape.

Registry API key, sent as X-Api-Key. Used for service-to-service calls, including the eagle-api push. Keys are issued through /admin/api-keys, carry their own role set and an optional project scope, and the plaintext is returned once at creation.

Keycloak bearer token, sent as Authorization: Bearer <token>. Used by the DEMI frontend and by staff users. Signatures are validated against the realm JWKS at SSO_JWKSURI. Setting KEYCLOAK_ENABLED=false skips signature validation and is for local work only.

Roles:

Set Roles Grants
Privileged read sysadmin, demi-admin, demi-service-read, demi-service-write Bypasses publication gating; reaches privileged reads
Write sysadmin, staff, demi-admin, demi-service-write Mutations
Admin sysadmin, staff, demi-admin /admin/* routes

staff is not a privileged read role. A staff caller sees rows through the read[] ladder below and reads fields at level 2.

A missing or invalid credential returns 401. A valid credential without the required role returns 403.

Visibility

A document is public only if the document and its parent project are both public — a double gate. Chunk visibility is derived from the parent document at query time and project names are resolved live, so a rename or a stale chunk ACL cascades nothing. When a project's publication state changes, DEMI re-derives the denormalised document read[].

Row access uses the ladder tokens in read[] plus the privileged read roles above:

Level Token Reader
1 team Project team
2 staff All EAO staff
3 idir Any IDIR user
4 public Anyone

Levels 2 to 4 nest: a level 3 row carries staff and idir. A row with no ladder token opens to privileged callers and to the roles it names.

An Eagle record keeps Eagle's own read[], capped under its parent by capRead. The readers of the stored read[] are always a subset of the readers of the parent's cap.

  • A read[] with no ladder token is never widened, so ['sysadmin','inspector'] stays as it is.
  • capRead never returns an empty list. The fallback is ['sysadmin'].
  • An empty or blank-only Eagle read[] is stored as ['sysadmin'].
  • A missing Eagle read[] is stored at level 2 (staff).

The code is eagleBaseAcl and eagleReadUnder in src/helpers/eagle-acl.js, and capRead, levelOfRead, readForLevel and noLadderToken in src/helpers/access-sql.js.

Eagle push contract

eagle-api sends the raw Eagle record, not a DEMI-shaped one, because DEMI owns the merge rules. Seven kinds always mirror this way: projects, documents, comment periods, comments, organizations, notifications (Eagle's ProjectNotification), and updates (Eagle's RecentActivity).

PUT {DEMI_API_BASE}/api/eagle/projects/{eagleId}
PUT {DEMI_API_BASE}/api/eagle/documents/{eagleId}
PUT {DEMI_API_BASE}/api/eagle/updates/{eagleId}
PUT {DEMI_API_BASE}/api/eagle/commentperiods/{eagleId}
PUT {DEMI_API_BASE}/api/eagle/comments/{eagleId}
PUT {DEMI_API_BASE}/api/eagle/organizations/{eagleId}
PUT {DEMI_API_BASE}/api/eagle/notifications/{eagleId}
Ocp-Apim-Subscription-Key: {DEMI_APIM_KEY}

{ "doc": { ... raw Eagle document ... } }

body.doc._id must match the :eagleId in the path or the call returns 400. Document pushes also carry a labels object resolving the List ObjectId references (type, milestone, projectPhase, documentAuthorType) to names.

Each push makes two attempts with a 10-second timeout. It is disabled unless both DEMI_API_BASE and DEMI_APIM_KEY are set. The write controller waits for the push before it answers; see Write response.

Five more kinds push only when DEMI_PUSH_OPT_IN_KINDS names them. The setting is a comma-separated list of route segments, unset by default:

Segment Record Route
users user /eagle/users/:id
groups group /eagle/groups/:id
inspections inspection /eagle/inspections/:id
inspection-elements inspection element /eagle/inspection-elements/:id
inspection-items inspection item /eagle/inspection-items/:id

A kind that is not listed does nothing: no database read and no HTTP call. The setting does not affect the other kinds above. User pushes leave out the legacy password and salt fields. A deleted group is pushed with isDeleted: true. DEMI serves all five routes. The setting names all five kinds on test and prod (values-test.yaml, values-prod.yaml).

DEMI merges the project and document push against the existing record's sources.track block if there is one, preserves every other sources block, writes an audit event, and cascades document visibility when the project's publication state flips. An update push stores the record in the updates Cosmos container keyed by the Eagle id, with projectId, isPublished set from the Eagle record's active field, and a notifiedAt claim used by the eagle-notify push below.

A DEMI-side narrowing or takedown of a project or document survives later Eagle pushes and re-seeds. DEMI stamps levelHeldAt on the row and keeps the lower of the stored and pushed read[] until a widen in DEMI reaches level 4 or Eagle's level. pdfTitle and the extraction fields are kept on every push as well.

eagle-api hard-deletes records. A document or comment-period delete pushes a delete marker (isDeleted: true). An update delete pushes the record once more with active: false, so DEMI unpublishes it the same way an edit would. A project delete (soft delete) pushes the project as well. Every kind not named here produces no delete push. Reconciling those needs an id-set comparison.

Write response

Every write that pushes waits for the push before replying: projects, documents, updates, comment periods, comments, organizations, notifications, groups, inspections and users, including the ENGAGE (met-api) and DEMI sync-out calls on /commentperiod and /projectNotification.

Variable Default Meaning
DEMI_PUSH_AWAIT_MS 25000 Ms a request waits
DEMI_PUSH_QUEUE_MAX 1000 Queue cap per pod; extra pushes drop
Case Response
Push lands 200, the saved record plus "mirrored": true
Push fails 200, the saved record plus "mirrored": false
Pushes off, or kind not in DEMI_PUSH_OPT_IN_KINDS Plain 200, no mirrored field

A push fails when the push client drops it (retries used up, queue full), DEMI rejects it, the deadline passes, or the parent is not in DEMI. The Mongo write stays; no 502. The row gets demiPushPending: true, demiPushFailedAt (last failure) and demiPushError (reason).

Clients must not retry on mirrored: false: a retry duplicates the record and file. eagle-admin shows "Saved. It will appear in search within a few minutes."

Failures log at error: [demiPush] not-mirrored <kind> <id>: <reason>. A later good write clears the fields.

A hard delete (DELETE /document, DELETE /commentperiod, DELETE /project/{id}/group/{groupId}) leaves no row to flag. A failed delete push writes a row in the demi_push_tombstones collection instead, and the reply stays 200 with mirrored: false. A soft delete (project) flags the live row and writes no tombstone. Fields and retry: Eagle DEMI Push Sweep.

Parent refusals

DEMI refuses a document, comment period, comment, group or inspection whose parent it lacks. The answer is 404 with the body { "error", "code" }:

Code Meaning
PARENT_NOT_FOUND The parent id is well formed; DEMI has no such row
PARENT_REF_INVALID The parent reference is missing or malformed

Both fail the push, as does a 404 without a code. The Eagle DEMI Push Sweep retries every flagged row until it lands.

Lists

eagle-api pushes no List rows. DEMI gets them only from eagle-demi seed-public-reads.js --only lists. DEMI copies List names into document labels and project fields.

Migrations

A migration that changes a mirrored record needs a re-push. The rule and the DEMI: note format are in eagle-api migrations/README.md. A test fails yarn test when a migration names a mirrored type without the note.

eagle-notify push contract

demi-api posts the event to eagle-notify when an update it holds transitions to published. Because DEMI owns updates, DEMI is the side that announces the transition, not eagle-api.

POST {NOTIFY_API_BASE}/api/events
x-functions-key: {NOTIFY_API_KEY}

{
  "kind": "project-updated",
  "serviceName": "project:{projectId}",
  "title": "...",
  "url": "...",
  "projectName": "...",
  "excerpt": "...",
  "idempotencyKey": "{update.id}"
}

serviceName is eao:updates when the update has no project. idempotencyKey is the update's id (the Eagle _id). An atomic notifiedAt claim on the DEMI updates record, set with a conditional Cosmos patch, stops two concurrent pushes of the same newly published update from notifying twice. Unpublishing or deleting the update pushes the same idempotencyKey with cancelled: true instead of the content fields. A failed push releases the claim, so the next push retries.

The push needs both NOTIFY_API_BASE and NOTIFY_API_KEY app settings set; either unset means pushes are dark. NOTIFY_API_KEY comes from Key Vault secret notify-api-key, sourced from the OpenShift secret demi-app-secrets by scripts/deploy-infra.sh.

eagle-notify turns the event into a paced campaign after a 10-minute recall window. See the eagle-notify repo docs/STATUS.md for that side.

Related

Eagle Documentation

Home

Operations

Architecture

Configuration

Help

Repositories

Environments

Clone this wiki locally