Edge Ban Detector

Edge ban detector

The edgeBan timer in the DEMI API finds addresses that pull many documents through Front Door with no sign of a browser, and keeps the banauto rule's address list in step with the bans it holds. Code: src/scripts/edge-ban.js, src/repositories/edge-bans.js, timer in api/index.js. The Front Door rules, the placeholder and the manual list are in the eagle-dev-guides wiki page Front-Door-Ban-Ladder.

What one run does

The timer runs hourly (EDGE_BAN_SCHEDULE, default 0 0 * * * *). A run:

1. Reads the Front Door access log in eagle-logs-<env> for the last 24 hours, as demi-identity-<env>.
2. Groups requests by socket address: IPv4 as /32, IPv6 as /64.
3. Counts distinct documents per address. A document is a path matching documents/<id>/download or /fetch. Range requests for one document count once.
4. Counts browser evidence per address: SPA loads (spafallback matched), analytics beacons (usagealias or /api/analytics) and calls to /demi-search/search.
5. Judges each address with more than documentFloor documents.
6. Records bans in the edgeBans container, then rebuilds the list from unexpired bans.

An address's traffic before its last ban ended is not counted again.

Judge order

First match wins.

Verdict	When
never ban	Address is in the policy allow list
never ban	Address is in a published crawler range (Google, Bing, OpenAI feeds)
never ban	User agent holds a declared crawler name (built-in list plus policy crawlerAgents)
never ban	Owner type is government, institution, mobile or relay
leave	Browser evidence is above evidenceFloor
ban	Owner type is datacenter or vpn
watch	Anything else (residential, unknown): reported, never banned

Owner type comes from an RDAP lookup (rdap.arin.net) of the network name and registrant names, matched against word patterns. A lookup that fails gives unknown. A known row's owner is reused.

The crawler range feeds are cached in edgeBans for 24 hours. A failed fetch falls back to the cached copy at any age.

Escalation

A ban lasts 1 hour for the first offence, 4 hours for the second and 24 hours from the third on (banHours). The count comes from the address's row, which expires 7 days after its last write. An address quiet for 7 days starts at 1 hour again.

Guards

Guard	Value
List entries	IPv4 /32 and IPv6 /64 only. A wider value in a row is refused with a warning.
List length	maxListLength 100. Over it, the latest-expiring bans are kept and a warning is logged.
Bans added per run	maxNewBansPerRun 20. The rest are marked deferred and judged again next run.
RDAP lookups per run	maxLookups 50. Past it, owner is unknown, so the address is only watched.
Crawler ranges	With no cached or fresh crawler ranges, the run bans nothing.
Empty list	The placeholder 192.0.2.1/32 is written.

Modes

EDGE_BAN_MODE sets what a run writes.

Mode	Cosmos edgeBans	banauto rule
off	Not read or written	Placeholder written. Needs no policy. Kill switch.
shadow	Bans and crawler cache written	Not touched
write	Bans and crawler cache written	Address list written (conditions only, other conditions kept)

A run is skipped, with an info line, when EDGE_BAN_MODE, EDGE_LOG_WORKSPACE_ID or EDGE_BAN_RULE_ID is empty. shadow and write are also skipped, with an error line, when the policy is empty or its Key Vault reference does not resolve. A rule write that gets 409 is retried by the next run. banmanual is never written.

The timer is registered only when EDGE_BAN_MODE is set. Deployed apps always set it.

Kill switch

Set app setting EDGE_BAN_MODE=off on the function app. The next hourly run writes the placeholder into banauto. No deploy is needed. Front Door takes 7 to 25 minutes to apply it.

App settings

Setting	Bicep parameter	Meaning
EDGE_BAN_MODE	edgeBanMode	off, shadow or write. Default off.
EDGE_BAN_SCHEDULE	edgeBanSchedule	NCRONTAB, six fields. Default hourly.
EDGE_BAN_POLICY	edgeBanPolicySecretUri	Key Vault reference to secret edge-ban-policy
EDGE_LOG_WORKSPACE_ID	edgeLogWorkspaceId	Customer id of eagle-logs-<env>
EDGE_BAN_RULE_ID	edgeBanRuleId	Resource id of the rulesdemidocs rule banauto

The test values are in azure/main.test.bicepparam.

Policy secret

Key Vault secret edge-ban-policy (demi-kv-<env>) holds JSON. Keys left out take the default.

Key	Default	Meaning
documentFloor	300	An address needs more distinct documents than this in 24 h to be judged
evidenceFloor	3	More browser evidence than this means leave alone
banHours	[1, 4, 24]	Ban length for the first, second and later offences
maxListLength	100	List cap
maxNewBansPerRun	20	Bans added per run
maxLookups	50	RDAP lookups per run
allow	[]	CIDRs never banned
crawlerAgents	[]	Extra user-agent words treated as declared crawlers

To set it, run from the devbox:

az keyvault secret set --vault-name demi-kv-<env> --name edge-ban-policy --file <policy.json>

Then list edge-ban-policy in optionalSecretNames of the environment's param file.

The edgeBans container

Cosmos NoSQL container edgeBans, partition key /id. One row per banned prefix: owner type, owner name, reason, offence times with document and evidence counts, banCount and expiresAt. The id is the prefix with / replaced by _. Every write resets a 7-day TTL. Row feed:crawlers holds the cached crawler ranges. The container is not application data and has no ACL.

Dry run

The CLI reads the access log and prints each candidate. It writes nothing: no Cosmos, no rule.

node src/scripts/edge-ban.js --dry-run --env test [--floor N]

--floor replaces documentFloor. The workspace comes from EDGE_LOG_WORKSPACE_ID, else from the workspace named eagle-logs-<env>. The caller needs Log Analytics Reader on it. Policy comes from EDGE_BAN_POLICY when it holds JSON, else the defaults.

Soak and promote

Each stage ends with an exit check. Each prod step needs explicit approval.

1. Set EDGE_BAN_MODE=shadow and leave the list at the placeholder. Run 7 days, including weekday peaks. Exit: every row in edgeBans has zero browser evidence, is in no never-ban family, and either overlaps the scraper alert or has a reason a person reviewed.
2. Set EDGE_BAN_MODE=write on test. The edge action edgeBanAction is already block, so every address the timer writes into banauto gets 302 to /blocked.html as soon as the rule updates. There is no count stage. Run 3 days. Exit: in the soak query (see Front-Door-Ban-Ladder), banauto matches only listed addresses, matches start within 25 minutes of a write, an expired ban stops matching, and block-page hits come only from listed addresses, with no support reports.
3. Set EDGE_BAN_MODE=write on prod, with approval. Same exit check for 7 days.

Rollback at any stage: set EDGE_BAN_MODE=off. To stop turning requests away without waiting for a list change, set edgeBanAction=count and deploy.