Document Intake

Document Intake

Reference for the pipeline that takes a new document from a browser, scans it for malware, and releases it to storage. The decision is in ADR-015-Malware-Scanning-on-Intake. The pipeline is defined in the eagle-demi repository.

Status

The pipeline is built and merged (eagle-demi main 9dc8858, PRs #606 and #607) and parked, as of 2026-10-14. deployDocumentIntake is false in azure/main.test.bicepparam and azure/main.prod.bicepparam. No INTAKE_* app setting is set on test or prod, so POST /intake/uploads answers 503 and the sweep does nothing. How to resume is under Resume.

The first test infra deploy (infra-fnd-9dc8858-234425) failed on two things. The resources it created were deleted from test on 2026-10-15: the storage account demiintestvymaysch2agdq, the identity demi-intake-test, the Defender setting, the three alert rules and the Storage Blob Data Owner grant. Test holds no intake resources and nothing bills. The foundation deploy recreates them all when the module is turned back on.

• Private endpoint demi-intake-pe-test. The subnet c4b0a8-test-cond-ext-pe-subnet (10.46.52.32/28) is full: 7 live endpoints hold all 11 usable IPs, and none is orphaned. The prod subnet has 5 free IPs. The privateEndpoint subnet (10.46.52.144/28) in the same vnet has 3 free IPs. It is the candidate for privateEndpointSubnetId on test.
• Role assignment conditions. Azure rejected the Storage Blob Data Contributor conditions with InvalidRoleAssignmentCondition: Action pattern .../blobs/tags/write is not referenced in role ba92f5b4-.... A condition may only name actions its role grants. The fix is in a pending PR: the Contributor grants deny only the Blob.Write.WithTagHeaders sub-operations, and the Owner grant on incoming also denies blobs/tags/write.

The deploy showed that Azure accepts a container-scoped Storage Blob Data Owner grant with a condition that has no OR branch.

Not yet verified: the casing of the tag key Malware scanning scan result, the key and format of the scan time tag, and the rest of the API identity's condition. The EICAR proof on test checks them, and it needs the private endpoint first.

Flow

1. The client calls POST /intake/uploads with projectId, fileName and optionally requestedRead. The call creates a documents row with scanStatus: 'pending' and read: [].
2. The response carries documentId, uploadUrl and expiresAt. The link is a user-delegation SAS for incoming/<documentId> and lasts 15 minutes. The client PUTs the file to it.
3. Defender for Storage scans the blob on upload. It writes the blob index tags Malware scanning scan result and Malware scanning scan time.
4. The timer intakeScanSweep in demi-api-fc-<env> runs every 5 minutes. It reads the tags of each pending row's blob with demi-intake-<env>.
5. The sweep applies the result (table under Scan results).

Resources

Resource	Name	Notes
Storage account	demiin<env><hash>	Defined in azure/modules/document-intake.bicep. Shared keys off. Public blob endpoint on, because the browser uploads to it. Private endpoint for the API app. Blob soft delete on.
Containers	incoming, quarantine, deadletter	Uploads land in incoming. Rejected files are copied to quarantine. Defender skips quarantine and deadletter.
Identity	demi-identity-<env>	The API identity. Limited by the access rule below.
Identity	demi-intake-<env>	Second identity on the API app. Reads blob tags in the sweep and writes the quarantine copy. It can never write a tag.
Alerts	demi-intake-scan-cap-75-<env>, demi-intake-scan-cap-100-<env>, demi-intake-sweep-silent-<env>	Notify demi-alerts-<env>. See Scan cap.

Defender settings

Set on the intake account by Microsoft.Security/defenderForStorageSettings.

Property	Value
isEnabled	true
overrideSubscriptionLevelSettings	true
malwareScanning.onUpload.isEnabled	true
malwareScanning.onUpload.capGBPerMonth	intakeCapGBPerMonth: 5 on test, 25 on prod
filters.excludeBlobsWithPrefix	quarantine, deadletter
filters.excludeBlobsLargerThan	2147483648 bytes (2 GB)
malwareScanning.blobScanResultsOptions	blobIndexTags
sensitiveDataDiscovery.isEnabled	false

A file over 2 GB is not scanned. The sweep rejects it as not-scanned. Every other storage account in DEMI has Defender off (Azure-Environments).

Scan results

The sweep matches the value of the Malware scanning scan result tag. Key names match without regard to case. Only the exact value No threats found is clean. Other values match by prefix.

Tag value	Row scanStatus	scanRejectReason	File
No threats found	clean	none	Streamed to the NRS object store at <projectId>/<id>.<ext>, then deleted from incoming. The row takes its requestedRead list.
Starts with Malicious	rejected	malicious	Copied to quarantine.
Starts with Not scanned	rejected	not-scanned	Copied to quarantine.
Starts with Scan failed or Error	rejected	scan-error	Copied to quarantine.
Any other value	rejected	unknown-result	Copied to quarantine.

The sweep sets scanStatus, scanResult (the raw tag value), scanRejectReason, scannedAt, scanSizeBytes and malwareNames. scannedAt is the time of the sweep run, not the scan time. malwareNames is always empty, because tags carry no malware names. A rejected row has read: [] and isPublished: false. Rejected rows are hidden from every read except the status route below. No admin list of rejected rows exists.

Upload status

GET /intake/uploads/:id is for the uploader. It returns documentId, scanStatus, scanRejectReason, scannedAt and createdAt. It answers 404 for a row without a scanStatus, so it never reads an ordinary document, and 503 when intake is off.

The sweep writes a row only while it is pending, so a repeated run changes nothing. A clean tag whose Malware scanning scan time is older than the blob's last write is ignored until the new bytes are scanned. A missing or unreadable scan time also leaves the row pending.

Sweep

The timer intakeScanSweep runs every 5 minutes (INTAKE_SWEEP_SCHEDULE). Each run has two passes.

1. Result pass. It walks pending rows oldest first, in pages of 100. For each row it stats the blob in incoming, then reads the tags. A row with no result tag stays pending. The pass stops after 500 rows or 20 minutes per run. The rows left are logged as a warning with rowsLeft and wait for the next run. A failed tag read leaves the row pending, is logged as an error and never rejects it.
2. Timeout pass. It rejects rows still pending after 6 hours (INTAKE_PENDING_TIMEOUT_HOURS).

The documents container indexes /scanStatus/?, /createdAt/? and /scannedAt/?. The result pass needs the first two for its query and its ordering.

A result reaches the row up to about 5 minutes after the scan ends. Scanning takes up to 3 hours, so a rejection by the timeout pass is final.

scanRejectReason	Meaning
timeout	The file arrived and no result came. The file is copied to quarantine.
no-upload	The link was issued and never used. The result pass sets it once the row is older than the 15-minute link plus 5 minutes and has no blob.

Access rule

The API identity demi-identity-<env> holds Storage Blob Data Contributor on the intake account with an ABAC condition. All three parts must hold.

• A blob read is allowed only in incoming, and only when the blob's Malware scanning scan result tag equals No threats found. Listing is not restricted.
• Writes and deletes are allowed only in incoming.
• No write may set the Malware scanning scan result tag.

The identity also holds Storage Blob Delegator, which signs the upload links.

demi-intake-<env> holds these assignments:

• Storage Blob Data Contributor on the account, with a condition that denies tag writes.
• Storage Blob Data Owner on the incoming container, with the same condition. Owner supplies the tag read that Contributor lacks.
• Storage Blob Delegator, without a condition.

Neither conditioned assignment can write a tag, so the sweep cannot mark a blob clean. A wrong sweep decision cannot release an unscanned file: the API identity's read of incoming gets 403 until the clean tag exists.

App settings

Set on demi-api-fc-<env> by azure/modules/api-function-flex.bicep. When INTAKE_STORAGE_ACCOUNT is empty, the app registers neither intake function and POST /intake/uploads answers 503.

Setting	Value
INTAKE_STORAGE_ACCOUNT	the intake account name
INTAKE_INCOMING_CONTAINER	incoming
INTAKE_QUARANTINE_CONTAINER	quarantine
INTAKE_QUARANTINE_CLIENT_ID	client id of demi-intake-<env>
INTAKE_SWEEP_SCHEDULE	0 */5 * * * *
INTAKE_PENDING_TIMEOUT_HOURS	6
INTAKE_CAP_GB_PER_MONTH	intakeCapGBPerMonth, the same cap Defender uses

Scan cap

Defender stops scanning for the rest of the month when the cap is reached.

The sweep intakeScanSweep logs this line on every run, every 5 minutes. scannedBytes is the sum of scanSizeBytes over rows whose scannedAt is in the current UTC month. capBytes is INTAKE_CAP_GB_PER_MONTH in bytes. The cap alerts read the newest such line:

[intake] month-to-date scannedBytes=<N> capBytes=<M>

One alert fires when scannedBytes reaches 75% of capBytes (severity 2), one at 100% (severity 1). Both run hourly and notify demi-alerts-<env>.

demi-intake-sweep-silent-<env> checks every 30 minutes and fires when no such line has appeared for 2 hours. The sweep is then not running, so pending uploads are not decided or timed out and the cap alerts cannot fire. It fires once on resume if the code deploy comes more than 2 hours after the infra pass.

At the cap, new uploads receive no scan result. The sweep rejects them after 6 hours, so the pipeline fails closed. To recover, raise intakeCapGBPerMonth in the environment's bicepparam and rerun infra (Operating-DEMI-in-Azure).

Cost

Measured from Eagle prod staff uploads, 2026-10-14: mean 1.06 GB and 130 files a month, peak 5.28 GB in 2025-05. The account costs about 9.80 USD a month plus 0.15 USD per GB scanned.

Environment	Expected	At the cap
Test	about 9.80 USD a month	10.55 USD
Prod	about 9.96 USD a month, peak month about 10.60 USD	13.55 USD

Resume

Infra goes before the code that reads it. Test first, then prod.

1. Pick a private endpoint subnet with free IPs. Override privateEndpointSubnetId in azure/main.test.bicepparam, or get the landing-zone subnet enlarged.
2. Set deployDocumentIntake = true for test.
3. Run the infra pass: scripts/deploy-infra.sh test --foundation --live.
4. Set INTAKE_STORAGE_ACCOUNT through the normal code deploy.
5. Apply the documents datasource: scripts/demi-devbox.sh apply --env test --datasources demi-documents-ds --yes.
6. Prove the pipeline with an EICAR test file built at run time, never committed. Upload it and a clean file through POST /intake/uploads and watch GET /intake/uploads/:id. The EICAR file must be tagged Malicious, sit in quarantine, and its row must be rejected. The clean file must reach the object store with its row clean and no blob left in incoming.

If a clean release returns 403, check the casing of the tag key first. Also check that the Malware scanning scan time tag parses as a date.

Before prod:

• Replace the Storage Blob Data Owner grant with a custom role that holds only blobs/tags/read.
• The first prod foundation deploy after the merge reindexes the documents container online.

See also

• ADR-015-Malware-Scanning-on-Intake — why Defender, why one account
• Azure-Environments — Defender state on the other accounts, spend
• Architecture — document storage