[O365] - Prune stale listing URLs from cursor to recover from long outages#18510
Merged
ShourieG merged 6 commits intoelastic:mainfrom Apr 20, 2026
Merged
[O365] - Prune stale listing URLs from cursor to recover from long outages#18510ShourieG merged 6 commits intoelastic:mainfrom
ShourieG merged 6 commits intoelastic:mainfrom
Conversation
…'s accepted window so that prolonged periods without successful fetches (e.g. auth failures during a credential rotation, or agent downtime longer than seven days) self-heal on resume instead of looping on AF20055.
ShourieG
added
Integration:o365
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
ShourieG
requested review from
chrisberkhout,
efd6 and
kcreddy
and removed request for
kcreddy
🚀 Benchmarks reportTo see the full report comment with |
efd6
reviewed
Apr 20, 2026
Comment on lines
+176
to
+213
| ).as(state, | ||
| // Drop listing links whose startTime has aged past the API's accepted | ||
| // window. The Office 365 Management API rejects requests whose startTime | ||
| // is more than 7 days in the past with AF20055; a link that was queued | ||
| // before a prolonged period without successful fetches (agent downtime, | ||
| // auth failures, credential rotation gaps, etc.) may have aged past | ||
| // that boundary while parked in the cursor. The listing-error branch | ||
| // cannot evict it reliably because its map-shaped error return is | ||
| // treated by the runtime as a freeze signal that discards cursor | ||
| // mutations, so the only way out is to prune up front. Once pruned, | ||
| // the bottom branch generates a fresh link clamped to the valid | ||
| // window on the next iteration. | ||
| (now() - duration(state.base.maximum_age)).as(oldest_allowed_start, | ||
| state.with( | ||
| { | ||
| "cursor": state.cursor.with( | ||
| { | ||
| "todo_links": state.cursor.todo_links.filter(link, | ||
| link.parse_url().RawQuery.parse_query().as(q, | ||
| // Listing URLs generated by this program use "startTime", but | ||
| // NextPageUri values returned by the API have been seen with | ||
| // the lowercase "starttime" variant (see #15325), so accept | ||
| // either. If no startTime can be extracted, keep the link | ||
| // and let the API itself decide. | ||
| q.?startTime.orValue(q.?starttime.orValue([])).as(st, | ||
| (st.size() == 0) ? | ||
| true | ||
| : | ||
| st[0].parse_time(time_layout.RFC3339) >= oldest_allowed_start | ||
| ) | ||
| ) | ||
| ), | ||
| } | ||
| ), | ||
| } | ||
| ) | ||
| ) | ||
| ).as(state, |
Contributor
There was a problem hiding this comment.
This could be folded into the previous block.
state.base.content_types.split(",").map(t, t.trim_space()).as(configured_types,
state.with(
{
"cursor": state.?cursor.orValue({}).with(
{
// Listing links have the content type embedded in the URL query
// string, so links for removed types would produce API errors
// if not pruned here.
//
// Links whose startTime has aged past maximum_age are also
// pruned. The Office 365 Management API rejects requests whose
// startTime is more than 7 days in the past with AF20055; a
// link queued before a prolonged period without successful
// fetches (agent downtime, auth failures, credential rotation
// gaps, etc.) may have aged past that boundary while parked in
// the cursor. The listing-error branch cannot evict it reliably
// because its map-shaped error return is treated by the runtime
// as a freeze signal that discards cursor mutations. Once
// pruned, the bottom branch generates a fresh link clamped to
// the valid window on the next iteration.
"todo_links": state.?cursor.todo_links.orValue([]).filter(link,
configured_types.exists(ct, link.to_lower().contains("contenttype=" + ct.to_lower()))
&&
link.parse_url().RawQuery.parse_query().as(q,
// Listing URLs generated by this program use "startTime",
// but NextPageUri values returned by the API have been seen
// with the lowercase "starttime" variant (see #15325), so
// accept either. If no startTime can be extracted, keep the
// link and let the API itself decide.
q.?startTime.orValue(q.?starttime.orValue([])).as(st,
(st.size() == 0) ?
true
:
now() - st[0].parse_time(time_layout.RFC3339) <= duration(state.base.maximum_age)
)
)
),
"todo_content": state.?cursor.todo_content.orValue([]).filter(item,
configured_types.exists(ct, ct.to_lower() == item.?contentType.orValue("").to_lower())
),
}
),
}
)
)
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
💚 Build Succeeded
History
cc @ShourieG |
efd6
approved these changes
Apr 20, 2026
|
Package o365 - 3.8.1 containing this change is available at https://epr.elastic.co/package/o365/3.8.1/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type of change
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots