Skip to content

Handle an error with a paging endpoint#364

Merged
michael-richey merged 24 commits intomainfrom
michael.richey/handle-bad-batch
Aug 13, 2025
Merged

Handle an error with a paging endpoint#364
michael-richey merged 24 commits intomainfrom
michael.richey/handle-bad-batch

Conversation

@michael-richey
Copy link
Copy Markdown
Collaborator

@michael-richey michael-richey commented Aug 8, 2025

What does this PR do?

If a paginated request fails try with smaller page sizes. In some cases this will solve the problem, in other cases there may be 1 resource that causes the endpoint to throw a 500 even by itself. Get as many resources as we can from the page skipping over and warning about the bad resource.

Description of the Change

Page number and page size make this very difficult to deal with. The fact that different endpoints use different methods for pagination make it worse. This approach halves the batch size until things start functioning. If batch size gets all the way down to 1 and the request still fails then skip over it. The tricky part is then to keep pulling at the smaller batch sizes until we can scale back up to the original batch size.

This change surfaced an error with team membership pagination. The PaginationConfiguration instance changes as we page through it so we can't use the same instance for each paginated request.

There's an async bug where remaining isn't always recalculated causing the loop to hang: remaining was always artificially set to 1 to get into the loop the first time. Now we immediately switch remaining to 0 until we can recalculate it. Probably need to spend some time refactoring this whole loop.

The actual github secret scanner doesn't want us to commit the cassette for the SDS because it has fake secrets in it as examples.

@michael-richey michael-richey marked this pull request as ready for review August 13, 2025 14:29
@michael-richey michael-richey requested a review from a team as a code owner August 13, 2025 14:29
@michael-richey michael-richey merged commit 3ea9d9c into main Aug 13, 2025
24 of 25 checks passed
@michael-richey michael-richey deleted the michael.richey/handle-bad-batch branch August 13, 2025 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants