Repository navigation
ADR 015 Malware Scanning on Intake
Daniel Truong edited this page Oct 14, 2026
·
3 revisions
Accepted 2026-10-14. The pipeline is in Document-Intake.
Uploads today go to eagle-api on OpenShift, which scans them with ClamAV. The documents themselves live in the NRS object store, an external S3-compatible service. DEMI has no intake of its own, and the briefs ask for malware scanning on upload.
The BC Gov landing zone sets Defender for Storage on every subscription, with on-upload scanning on (BC-Gov-Azure-Landing-Zone). Each storage account can opt out. Defender scans blobs in Azure Storage only. It cannot scan the NRS object store.
DEMI scans uploads with Defender for Storage on-upload scanning, on one staging account per environment.
-
One scanned account. Browsers upload to
demiin<env>. Defender scans each blob and writes the result to a blob index tag. A timer reads the tag and only a clean file moves on to the NRS object store. See Document-Intake. -
Every other account opts out. Each DEMI storage module sets
overrideSubscriptionLevelSettings: trueandisEnabled: false, so those accounts are not scanned and not billed for it. - Defender is the only scanner. A file in the object store passed Defender. Nothing scans it again, and stored files are assumed safe.
-
Fail closed. Anything other than
No threats foundis rejected. A file with no result after 6 hours is rejected.
- Each environment pays about 10 USD a month for the scanned account, plus 0.15 USD per GB scanned.
- A monthly cap limits scan spend: 5 GB on test, 25 GB on prod. At the cap, scanning stops and uploads are rejected until the month ends or the cap is raised.
- Files over 2 GB are not scanned and are rejected.
- Files already in the NRS object store, and anything written to it by another route, are not scanned.
- The intake account needs a public blob endpoint, because the browser uploads to it. Access is limited by Entra auth, a 15-minute write-only SAS and the access rule in Document-Intake.
- On 2026-10-14 the test deploy of an Event Grid topic
demi-intake-testfor scan results was denied. The landing-zone policy assignmentDeny-PublicPaaSEndpoints(definition "Azure Event Grid topics should disable public network access", effect Deny) blocks a topic with public network access. Microsoft documents: "Event Grid topics that don't have public network access enabled, for example, those using private endpoints, aren't supported for scan-result delivery." The sweep reads the result from the blob index tag instead. A result reaches the row up to about 5 minutes after the scan. - The pipeline is built and parked as of 2026-10-14. The first test deploy failed on a full private endpoint subnet and on role conditions, and nothing is enabled on test or prod. See Status in Document-Intake.
- A role assignment condition may only name actions its role grants. Storage Blob Data
Contributor does not grant
blobs/tags/write, so its conditions deny only theBlob.Write.WithTagHeaderssub-operations. Only the Owner grant onincomingalso deniesblobs/tags/write. - ClamAV on OpenShift stays in place for eagle-api. This decision does not change it.
- ClamAV self-hosted in Azure. It runs always on and costs about 31 to 53 USD a month per environment. A cold start takes minutes. DEMI would own the signature updates and the host.
- Defender on every storage account. It scans files that never come from an upload. Each account costs about 0.63 CAD a day (Azure-Environments).
- Document-Intake — pipeline, settings, how to resume
- Azure-Environments — Defender state per account
- Environment Reality & Gotchas
- Architecture
- Seeding — filling Cosmos from Track and Eagle
- Sources and Status — every source, one state
- Search & Retrieval — overview
- Search Index Reference
- Search Query Construction
- Search Measurements
- Extraction Pipeline
- Extraction Quality — coverage, word-joining, tiling
- Identity Model — which id means what
- Attribute-Level Access — field-visibility levels, catalog
- Sync Architecture — how records get in
- Eagle Mirror Reference — mirrored kinds, refusals, drift classes
- ENGAGE Rows Reference — ENGAGE-owned comment periods, reconcile
- Public-Read Backfill — filling and repairing the mirror containers
-
Short Links —
/s/{code}redirect and/api/links -
Recent Uploads —
/api/documents/recent-uploads, eagle-public homepage panel - Reports — comment export and BCGW CSV reads
-
User Data — saved map areas, preferences,
/meroutes -
Bulk Download —
/api/bulk-downloads, zip jobs - Bulk Download: Operations — enable, poison-queue runbook
- Document Intake — upload link, malware scan, release
- Connecting an Application to DEMI
- Azure Environments
- Document Backup — archive copy of originals, run and check
- Document Backup: Revert and Drill — put documents back from the archive
- PDF Titles — public name written into stored PDFs, run, restore
- Secret Sync — Key Vault, secret-sync Function app, required names
-
Operating DEMI in Azure — deploy, verify
- Operating the DEMI Data Plane — devbox access, probes, audit and cost reads
- Edge ban detector — hourly timer, modes, kill switch, soak and promote
-
CI Workflows — every GitHub Actions workflow, trigger, identity, roles
- CI Deploy Workflows — eagle-demi staging and production deploys
- CI Admin Frontend Workflows — eagle-demi-admin checks and deploys
- Release Process — cutting, tagging, shipping and rolling back a version
- Runbook: Search Outage — index/app schema drift, diagnose, fix, roll back
- BC Gov Landing Zone
- Business Rules
- EPIC.search Business Rules — where DEMI differs from EPIC.search
- 001: Typesense for DFL Search — superseded
- 002: Async Processing (MongoDB Queue) — superseded
- 003: Rule-Based Classification
- 004: Read ACL Authorization Model
- 005: Cosmos Full-Text Search Ruled Out
- 006: AI Summarization over BM25
- 007: Service-to-Service Credentials
- 008: Audit and Usage Analytics Store
- 009: Track Feed — proposed
- 010: Attribute-Level Access Control
- 011: Public Read Surface
- 012: Public Config Surface
- 013: Stored Project Summaries — proposed
- 014: Sync-out to Eagle
- 015: Malware scanning on intake