Configure idempotent upload to Google Cloud Storage#4269
Merged
leonardehrenfried merged 1 commit intoJul 13, 2022
Merged
Conversation
Codecov Report
@@ Coverage Diff @@
## dev-2.x #4269 +/- ##
=============================================
- Coverage 56.89% 56.87% -0.02%
+ Complexity 10586 10583 -3
=============================================
Files 1402 1402
Lines 57170 57170
Branches 6603 6603
=============================================
- Hits 32526 32518 -8
- Misses 22630 22638 +8
Partials 2014 2014
Continue to review full report at Codecov.
|
Member
|
Since this is a sandbox feature, we can merge that in with one approval, preferably by someone at Entur. |
hannesj
approved these changes
Jul 13, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The goal of this PR is to prevent failure to upload graph building artifacts to Google Cloud Storage due to transient network issues/ server errors as described in issue #4268
By default the Google Storage client library does not retry failed upload operations.
See https://cloud.google.com/storage/docs/retry-strategy#java for more details about which operations are retried by default.
To activate retry on error, it is necessary to make the operation "conditionally idempotent" by detecting concurrent update:
Expected behavior: if a transient network issue/server error prevents an upload operation to complete, it will be retried with the default retry settings: 6 attempts, exponential backoff factor: 2, initial delay: 1s
(See https://cloud.google.com/storage/docs/retry-strategy#client-libraries for details on the default retry settings)
Side effect: if another process concurrently updates the blob, the upload process will fail.
Issue
closes #4268
Unit tests
✅
Documentation
No