Fix stackoverflow with large pages in paginator #6668
Merged
+217
−111
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, signalig onNext() to the subscriber was done via recursion, pulling elements from an iterator over the current page returned by the service. However, this can quickly lead to a stackoverflow error since the stack will grow linearly with the size of the page. This commit fixes this issue by using SdkPublisher's builtin flatMapIterable(), which uses a loop to signal onNext(), and also ensures that it does not call itself recursively. fixes #6411
dagnir
force-pushed
the
dongie/gh-6411-2
branch
from
fa2b2c7 to
c40873e
Compare
12 tasks
dagnir
commented
Jan 12, 2026
| public static final Function<Integer, Iterator<Integer>> EMPTY_ITERATOR = response -> new ArrayList<Integer>().listIterator(); | ||
| @Mock | ||
| AsyncPageFetcher asyncPageFetcher; | ||
| PaginatedItemsPublisher<Integer, Integer> itemsPublisher; |
Contributor
Author
There was a problem hiding this comment.
Note: this test is just testing features of SdkPublisher such as limit(), not PaginatedItemsPublisher itself, so just removed this class entirely
alextwoods
approved these changes
Jan 12, 2026
...gen/src/main/java/software/amazon/awssdk/codegen/poet/paginators/AsyncResponseClassSpec.java
Show resolved
Hide resolved
...ed-classes-test/src/test/java/software/amazon/awssdk/services/paginators/PaginatorsTest.java
Outdated
Show resolved
Hide resolved
zoewangg
approved these changes
Jan 12, 2026
...n/awssdk/codegen/poet/paginators/PaginatedOperationWithResultKeyAndMoreResultsPublisher.java
Show resolved
Hide resolved
dagnir
added
the
no-api-surface-area-change
|
|
This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, signalig onNext() to the subscriber was done via recursion, pulling elements from an iterator over the current page returned by the service. However, this can quickly lead to a stackoverflow error since the stack will grow linearly with the size of the page.
This commit fixes this issue by using SdkPublisher's builtin flatMapIterable(), which uses a loop to signal onNext(), and also ensures that it does not call itself recursively.
fixes #6411
Motivation and Context
Modifications
Testing
Screenshots (if appropriate)
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License