Repository navigation
Search and Retrieval
Azure AI Search Basic, canadacentral, keyless, private endpoint only — demi-search-test in test,
demi-search-prod in prod. Estates: Environment-Reality-and-Operational-Gotchas.
Retrieval is classic lexical BM25 over three indexes. There are no vector fields and no embeddings.
Chunk results are reordered by Azure's semantic ranker — a reranker over the top 50 BM25 hits,
not a second retrieval path. It cannot surface anything the keyword query missed, and the measured
found@50 is identical with it on and off. AI's role is a final summarizer over results the lexical
index already returned. Cosmos full-text search was evaluated and rejected —
ADR-005-Cosmos-Full-Text-Search-Ruled-Out.
The summarizer is the fifth step and the only one that touches a model. Steps 1–4 do not:
| Step | Function | AI? | |
|---|---|---|---|
| 1 | tokenize, build the Lucene query |
ai-search.tokenize, ai-search.buildQuery
|
no |
| 2 | BM25 + the OData ACL filter |
ai-search.runSearch, access-odata.filterFor
|
no |
| 3 | top N chunk ids, ranked by BM25 score | ai-search.searchChunks |
no |
| 4 | fetch those chunks' text from Cosmos | repositories/chunks.getById |
no |
| 5 | LLM composes a short answer with [n] citations |
controllers/search.summarize |
yes |
Step 4 point-reads from Cosmos as the second ACL gate. content is retrievable, so it is the
select list in searchChunks that keeps chunk text out of responses. Delete the model deployment
and steps 1–4 still work; only the summary panel disappears. Design, rejected alternatives and
limitations: ADR-006-AI-Summarization-over-BM25.
Costed against the real corpus with the Azure retail pricing API, canadacentral, measured 2026-07-30.
| Option | Cost/mo | Fuzzy | Verdict |
|---|---|---|---|
| Cosmos DB full-text search | ~$9 | no — silent no-op | ruled out, ADR-005-Cosmos-Full-Text-Search-Ruled-Out |
| Azure AI Search Basic | ~$75–81 | yes, GA | chosen. Also gives highlighting, faceting and synonyms natively |
| Typesense Dedicated D8 | ~$700 | yes, GA | ruled out, ADR-001-Typesense-for-DFL-Search |
Typesense keeps indexed fields in RAM with no on-disk mode, and its own sizing rule is 2–3× RAM against searched-field size, which puts this corpus at 17–26 GB against the 8 GiB Container Apps Consumption ceiling. Its clustering is replication rather than sharding, so a 3-node cluster is 3× the RAM, not a third each.
EAO's other search product, EPIC.search, is a vector/RAG stack and shares none of this
implementation. Where DEMI's rules differ from it is in EPIC-Search-Business-Rules.
- Search-Index-Reference — indexes, indexers, data sources, api-versions, analyzers, field metadata and what the schema cannot express.
- Search-Query-Construction — how a request becomes a Lucene query and an OData filter, plus highlighting, deletes, paging and the search controller's decisions.
- Search-Measurements — the retrieval scorecard, the hypotheses tested and rejected, and the rules a probe has to follow.
- Architecture — containers, ACL model, deletion semantics
- BC-Gov-Azure-Landing-Zone — private endpoints, DNS and the policies that force them
- Extraction-Pipeline — how the indexed text is produced
- ADR-001-Typesense-for-DFL-Search — superseded, kept for why Typesense was chosen and dropped
- Environment Reality & Gotchas
- Architecture
- Seeding — filling Cosmos from Track and Eagle
- Sources and Status — every source, one state
- Search & Retrieval — overview
- Search Index Reference
- Search Query Construction
- Search Measurements
- Extraction Pipeline
- Extraction Quality — coverage, word-joining, tiling
- Identity Model — which id means what
- Attribute-Level Access — field-visibility levels, catalog
- Sync Architecture — how records get in
- Eagle Mirror Reference — mirrored kinds, refusals, drift classes
- ENGAGE Rows Reference — ENGAGE-owned comment periods, reconcile
- Public-Read Backfill — filling and repairing the mirror containers
-
Short Links —
/s/{code}redirect and/api/links -
Recent Uploads —
/api/documents/recent-uploads, eagle-public homepage panel - Reports — comment export and BCGW CSV reads
-
User Data — saved map areas, preferences,
/meroutes -
Bulk Download —
/api/bulk-downloads, zip jobs - Bulk Download: Operations — enable, poison-queue runbook
- Document Intake — upload link, malware scan, release
- Connecting an Application to DEMI
- Azure Environments
- Document Backup — archive copy of originals, run and check
- Document Backup: Revert and Drill — put documents back from the archive
- PDF Titles — public name written into stored PDFs, run, restore
- Secret Sync — Key Vault, secret-sync Function app, required names
-
Operating DEMI in Azure — deploy, verify
- Operating the DEMI Data Plane — devbox access, probes, audit and cost reads
- Edge ban detector — hourly timer, modes, kill switch, soak and promote
-
CI Workflows — every GitHub Actions workflow, trigger, identity, roles
- CI Deploy Workflows — eagle-demi staging and production deploys
- CI Admin Frontend Workflows — eagle-demi-admin checks and deploys
- Release Process — cutting, tagging, shipping and rolling back a version
- Runbook: Search Outage — index/app schema drift, diagnose, fix, roll back
- BC Gov Landing Zone
- Business Rules
- EPIC.search Business Rules — where DEMI differs from EPIC.search
- 001: Typesense for DFL Search — superseded
- 002: Async Processing (MongoDB Queue) — superseded
- 003: Rule-Based Classification
- 004: Read ACL Authorization Model
- 005: Cosmos Full-Text Search Ruled Out
- 006: AI Summarization over BM25
- 007: Service-to-Service Credentials
- 008: Audit and Usage Analytics Store
- 009: Track Feed — proposed
- 010: Attribute-Level Access Control
- 011: Public Read Surface
- 012: Public Config Surface
- 013: Stored Project Summaries — proposed
- 014: Sync-out to Eagle
- 015: Malware scanning on intake