fix/ensure that split pdf requests are retried #160
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We discovered that when a pdf is split into smaller chunks, those requests are not being retried. Now that we have `allow_failed=False`, this results in the whole document failing as soon as any of the child requests hit a transient error. The fix is to reuse the `utils.Retry` logic that the main code path uses. Copying the retry config in the hook logic is not great, and we can work with Speakeasy to make the internal logic more modular. But for now, this will address the current failures while we work on a better implementation. Testing: See the added unit test. The existing retry logic works for the final split page, everything else needs to use the new logic. To test this, I mocked a response from the server to return 502 for a low `starting_page_number`, which we know will have to be handled by the hooks.
pawel-kmiecik
approved these changes
Aug 22, 2024
awalker4
force-pushed
the
fix/retry-split-requests
branch
from
c385af5 to
f18f2b4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We discovered that when a pdf is split into smaller chunks, those requests are not being retried. Now that we have
allow_failed=False, this results in the whole document failing as soon as any of the child requests hit a transient error. The fix is to reuse theutils.Retrylogic that the main code path uses. Copying the retry config in the hook logic is not great, and we can work with Speakeasy to make the internal logic more modular so we can reuse more. But for now, this will address the current failures while we work on a better implementation.Testing:
See the added unit test. The existing retry logic works for the final split page, everything else needs to use the new logic. To test this, I mocked a response from the server to return 502 for a low
starting_page_number, which we know will have to be handled by the hooks.Other changes:
Remove the "Not splitting" log. When the final split page is retried, it triggers all the hooks again. We need to force
split_pdf_page=Falsein this request, and we don't need additional logging when this code is hit again.