`m365_defender`: `alert`/`incident` pagination is fragile under high volume

[kcreddy]

Summary

The alert and incident data streams paginate the Microsoft Graph security API using $top + incrementing $skip while holding a fixed $filter. This collides with three hard limits of the Graph security API and leads to 400 errors, wasted/stalled polling, and a latent data-loss edge case under high volume. The current default config mostly avoids the worst symptom by accident, but the mechanism is not sound and should be reworked before we lean on these streams for agentless GA.

vulnerability is not affected — it uses a different API (Defender machines API) and already paginates correctly via @odata.nextLink.

Affected

• packages/m365_defender/data_stream/alert/agent/stream/httpjson.yml.hbs ($top default batch_size=2000, interval=5m)
• packages/m365_defender/data_stream/incident/agent/stream/httpjson.yml.hbs ($top default batch_size=50, interval=1m, $expand=alerts)

Background — Graph security API constraints

Per the Microsoft Graph Security API error responses / Constraints:

• $top is capped at 1000. Values above are clamped, not rejected.
• $skip is capped at 500 — $skip=501 returns 400 Bad Request.
• The documented way to page past these limits is to advance a $filter on the timestamp (eventDateTime/lastUpdateDateTime), i.e. move the time boundary forward — not $skip.

Additionally, empirically, alerts_v2 does not return @odata.nextLink (verified below), so nextLink-based pagination is not available for these two streams.

The problem

Current pagination (both streams) keeps $filter fixed and grows $skip:

response.pagination:
  - set: url.params.$filter = '[[.last_response.url.params.Get "$filter"]]'   # unchanged
  - set: url.params.$skip   = '[[if (eq (len .last_response.body.value) BATCH)]][[add prevSkip BATCH]][[end]]'

Because $filter is only used as a cross-interval cursor (never advanced within a run), within-run paging is pure $skip offset paging and therefore subject to the $skip ≤ 500 cap. Behavior by batch_size:

batch_size	Behavior	Result
2000 (current alert default)	server clamps page to ≤1000; predicate len==2000 never true → stops after 1 page	no $skip paging, but ≤1000 records/interval; the 2000 is dead/misleading
1500	same as 2000 (≤1000 ≠ 1500)	≤1000 records/interval
1000	full page len==1000==batch → page 2 $skip=1000 > 500 → 400	errors every high-volume interval, stall
≤500 (e.g. incident 50)	real $skip paging until $skip crosses 500 → 400 on the deep page under load	plus offset-shift risk (below)

Consequences:

1. 400 errors + stalled catch-up whenever a poll window has more records than the offset cap allows.
2. Slow real-time ingestion — with batch_size > 1000 the stream fetches ≤1000 records per interval, so a backlog drains at ≤1000/interval and new alerts queue behind it.
3. Latent within-run data loss (edge case): the order/cursor key lastUpdateDateTime is also the mutable field. If a record is updated mid-pagination it shifts offsets, and an item at the boundary can be skipped; if its timestamp is below the final cursor, the next interval's ge filter misses it → permanent loss. Requires concurrent updates during a multi-page $skip run (so mainly the smaller-batch incident stream).

No duplicate documents result from boundary re-fetches: both streams set a deterministic _id via fingerprint (including lastUpdateDateTime), so re-fetching the same version overwrites.

Evidence

• $top is clamped, not rejected: Live API response - runs alert with $top=2000 and returns 200.
• No @odata.nextLink: a live GET /v1.0/security/alerts_v2?$top=5 returns only @odata.context + value (5 items) and no @odata.nextLink / @odata.count, despite more alerts existing.
• Precision: returned timestamps are 7-digit sub-second (e.g. 2026-04-15T09:45:29.3666667Z).
• Docs: $top ≤ 1000, $skip ≤ 500 (constraints); List incidents; Paging.

Proposed fix

Rework alert/incident pagination to advance $filter instead of $skip (keeping within-run pagination so backfill stays fast):

• Keep $orderby lastUpdateDateTime asc.
• In response.pagination, when the page is full (len == batch_size): set $filter = 'lastUpdateDateTime gt <max lastUpdateDateTime in page>' and reset $skip=0; otherwise stop (non-full page = last page).
• Use gt at full timestamp precision (do not truncate to milliseconds — that would reintroduce ties). At 7-digit (100 ns) precision, exact ties are negligible; gt avoids the ge infinite-loop-on-identical-timestamp-page case, and the fingerprint _id absorbs any rare boundary re-fetch.
• Correct the alert batch_size default (2000 is above the Graph cap and never triggers within-run paging); pick a value ≤ 1000.

This keeps $skip at 0 (no 400), drains a full backlog in one run (fast catch-up, decoupled from interval), and needs no @odata.nextLink.

Alternatives considered

• Follow @odata.nextLink (like the vulnerability stream): not viable — alerts_v2 doesn't return it.
• MDE-style cross-interval gt cursor with no within-run pagination (microsoft_defender_endpoint/data_stream/log): correct and simple, but fetches only one page per interval, so backfill/catch-up time scales with interval × backlog — poor for a high-volume stream and sensitive to user-set interval/initial_interval.

Notes / related

• Surfaced during the agentless ORR for m365_defender; the package currently requests 4Gi for agentless (manifest.yml) citing in-memory volume. Pagination and per-page memory are being reviewed together.
• Enforcing numeric bounds on batch_size is tracked separately in elastic/package-spec#1176.

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)