Improve reliability of source calendar retrieval by enabling retries #403
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…for intermittent HTTP failures
This was referenced Jan 10, 2024
|
@octogonz I finally implemented this solution in my version and it seems to be making callWithBackoff work as intended now. The only thing I tweaked was adding an error catch in backoffRecoverableErrors for 404 (because that was the easiest way to break my calendar URL to test this!). Not sure if we would need that error in the real world though. I closed my #392 PR in favor of this solution. |
|
Seems fine to me. @jonas0b1011001 - any thoughts? |
jonas0b1011001
approved these changes
May 12, 2024
This was
linked to
issues
May 15, 2024
jonas0b1011001
pushed a commit
that referenced
this pull request
May 22, 2024
…for intermittent HTTP failures (#403)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was having problems where my entire calendar would get deleted temporarily, only to be recreated an hour later, due to intermittent HTTP errors being reported by the calendar source. It is the same issue discussed in #343.
Related work
PR #392 tries to solve this same problem by silently skipping sync when such errors occur. I'm uncomfortable with that solution, because I would prefer "obviously wrong" (entire calendar is missing) versus "deceptively wrong" (calendar looks okay but is actually telling me wrong information because it has stopped syncing). Skipping sync would be a good solution if we ALSO provided better mechanism for communicating problems, for example email notifications. Ideally we should pursue that.
A different approach
But then I found a superior fix: This PR completely solves the root cause in my case. That is why I have not invested the time to investigate better error notifications. For my own purposes at least, this PR entirely eliminated the need for PR #392.
Fixes #343
What I changed
Track the failures. If an HTTP request ultimately fails (excluding success after retrying), throw a top-level exception so that the Google Apps Script "Executions" dashboard reports the execution as failing. Example shown below:
Reattempt HTTP requests. Improve
callWithBackoff()so that it retries HTTP requests for all status codes that are known to be intermittent problems.With these changes, I can see that the failures are no longer occurring. The longest reattempt took 8 tries over 14 seconds, as seen in this log:
But since yesterday, every execution ultimately succeeded. 👍