Repository navigation
EPIC Search Business Rules
Daniel Truong edited this page Oct 15, 2026
·
4 revisions
EPIC.search is a separate EAO search product: a React web app, two Flask services (search-api for
orchestration and model calls, search-vector-api for retrieval) and a Postgres/pgvector embedder.
Its source is the EPIC.search repository; a workspace clone goes in EPIC.search/ beside eagle-demi.
None of its rules govern DEMI. Nothing in eagle-demi/src references the product, and no rule read
out of it has been promoted into DEMI policy. DEMI's rules are Business-Rules (BR-01 to BR-09),
its retrieval design is Search-and-Retrieval, and its access model is
ADR-004-Read-ACL-Authorization-Model with Identity-Model.
| Topic | EPIC.search | DEMI |
|---|---|---|
| Retrieval | Vector search over pgvector, re-ranked by a cross-encoder. | Lexical BM25 in Azure AI Search, no embeddings — ADR-005-Cosmos-Full-Text-Search-Ruled-Out, ADR-006-AI-Summarization-over-BM25. |
| Role of the model | The model gates query scope, classifies query complexity, infers project and document-type filters, and rewrites the query before retrieval. | The model summarises chunks the lexical query already returned. It never selects documents, re-ranks or rewrites (src/ai/summarize.js). |
| Summary grounding | The summariser prompts carry no grounding, citation or no-fabrication rule. | The summariser answers only from the numbered sources, cites every claim, and says so when the sources do not answer (src/ai/summarize.js). |
| Public access | Every search needs a BC Gov IDIR login; there is no anonymous route. | Anonymous search is served, and the caller's read[] decides what it returns (src/http/routes.js). |
| Published gating |
document_status is stored and indexed at ingest but never used as a search filter. |
A document is public only when it and its parent project are both public — BR-04. |
| Metadata classification | The embedder tags every chunk against a fixed 111-term vocabulary, by literal or semantic match. | Reuse of that embedder is rejected in favour of rule-based suggestions a person confirms — ADR-003-Rule-Based-Classification. |
| Ingest admission |
pdf, images, docx and plain text only; doc, xlsx and pptx are refused. |
pdf, doc, docx, pptx and xlsx are all extracted (src/extract.js:32-38). |
| Chunking | 1000 characters with 200 overlap, chunked one page at a time. | Sections accumulate to a target size and overlap across block boundaries; pageNumber is a sequence number, not a PDF page (src/chunker.js). |
- Environment Reality & Gotchas
- Architecture
- Seeding — filling Cosmos from Track and Eagle
- Sources and Status — every source, one state
- Search & Retrieval — overview
- Search Index Reference
- Search Query Construction
- Search Measurements
- Extraction Pipeline
- Extraction Quality — coverage, word-joining, tiling
- Identity Model — which id means what
- Attribute-Level Access — field-visibility levels, catalog
- Sync Architecture — how records get in
- Eagle Mirror Reference — mirrored kinds, refusals, drift classes
- ENGAGE Rows Reference — ENGAGE-owned comment periods, reconcile
- Public-Read Backfill — filling and repairing the mirror containers
-
Short Links —
/s/{code}redirect and/api/links -
Recent Uploads —
/api/documents/recent-uploads, eagle-public homepage panel - Reports — comment export and BCGW CSV reads
-
User Data — saved map areas, preferences,
/meroutes -
Bulk Download —
/api/bulk-downloads, zip jobs - Bulk Download: Operations — enable, poison-queue runbook
- Document Intake — upload link, malware scan, release
- Connecting an Application to DEMI
- Azure Environments
- Document Backup — archive copy of originals, run and check
- Document Backup: Revert and Drill — put documents back from the archive
- PDF Titles — public name written into stored PDFs, run, restore
- Secret Sync — Key Vault, secret-sync Function app, required names
-
Operating DEMI in Azure — deploy, verify
- Operating the DEMI Data Plane — devbox access, probes, audit and cost reads
- Edge ban detector — hourly timer, modes, kill switch, soak and promote
-
CI Workflows — every GitHub Actions workflow, trigger, identity, roles
- CI Deploy Workflows — eagle-demi staging and production deploys
- CI Admin Frontend Workflows — eagle-demi-admin checks and deploys
- Release Process — cutting, tagging, shipping and rolling back a version
- Runbook: Search Outage — index/app schema drift, diagnose, fix, roll back
- BC Gov Landing Zone
- Business Rules
- EPIC.search Business Rules — where DEMI differs from EPIC.search
- 001: Typesense for DFL Search — superseded
- 002: Async Processing (MongoDB Queue) — superseded
- 003: Rule-Based Classification
- 004: Read ACL Authorization Model
- 005: Cosmos Full-Text Search Ruled Out
- 006: AI Summarization over BM25
- 007: Service-to-Service Credentials
- 008: Audit and Usage Analytics Store
- 009: Track Feed — proposed
- 010: Attribute-Level Access Control
- 011: Public Read Surface
- 012: Public Config Surface
- 013: Stored Project Summaries — proposed
- 014: Sync-out to Eagle
- 015: Malware scanning on intake