Skip to content

Configure idempotent upload to Google Cloud Storage#4269

Merged
leonardehrenfried merged 1 commit into
opentripplanner:dev-2.xfrom
entur:otp2_configure_idempotent_upload_to_gcs
Jul 13, 2022
Merged

Configure idempotent upload to Google Cloud Storage#4269
leonardehrenfried merged 1 commit into
opentripplanner:dev-2.xfrom
entur:otp2_configure_idempotent_upload_to_gcs

Conversation

@vpaturet

@vpaturet vpaturet commented Jul 13, 2022

Copy link
Copy Markdown
Contributor

Summary

The goal of this PR is to prevent failure to upload graph building artifacts to Google Cloud Storage due to transient network issues/ server errors as described in issue #4268
By default the Google Storage client library does not retry failed upload operations.
See https://cloud.google.com/storage/docs/retry-strategy#java for more details about which operations are retried by default.
To activate retry on error, it is necessary to make the operation "conditionally idempotent" by detecting concurrent update:

  • if the blob already exists in the target bucket and must be updated: check that the blob version ("generation") did not change between the beginning and the end of the upload operation.
  • if the blob does not exist yet in the target bucket and must be created: check that the blob still does not exist when the upload operation completes.

Expected behavior: if a transient network issue/server error prevents an upload operation to complete, it will be retried with the default retry settings: 6 attempts, exponential backoff factor: 2, initial delay: 1s
(See https://cloud.google.com/storage/docs/retry-strategy#client-libraries for details on the default retry settings)

Side effect: if another process concurrently updates the blob, the upload process will fail.

Issue

closes #4268

Unit tests

Documentation

No

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

Merging #4269 (608c7d1) into dev-2.x (c14f86b) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             dev-2.x    #4269      +/-   ##
=============================================
- Coverage      56.89%   56.87%   -0.02%     
+ Complexity     10586    10583       -3     
=============================================
  Files           1402     1402              
  Lines          57170    57170              
  Branches        6603     6603              
=============================================
- Hits           32526    32518       -8     
- Misses         22630    22638       +8     
  Partials        2014     2014              
Impacted Files Coverage Δ
...tripplanner/ext/datastore/gs/GsFileDataSource.java 0.00% <0.00%> (ø)
...pplanner/ext/datastore/gs/GsOutFileDataSource.java 0.00% <0.00%> (ø)
...ph_builder/module/ned/MissingElevationHandler.java 93.70% <0.00%> (-1.58%) ⬇️
...pplanner/graph_builder/module/osm/OSMDatabase.java 76.47% <0.00%> (-0.95%) ⬇️
...lanner/routing/vehicle_parking/VehicleParking.java 82.03% <0.00%> (-0.79%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c14f86b...608c7d1. Read the comment docs.

@leonardehrenfried leonardehrenfried added the +Sandbox This will be implemented as a Sandbox feature label Jul 13, 2022
@vpaturet vpaturet marked this pull request as ready for review July 13, 2022 09:49
@vpaturet vpaturet requested a review from a team as a code owner July 13, 2022 09:49
@leonardehrenfried

Copy link
Copy Markdown
Member

Since this is a sandbox feature, we can merge that in with one approval, preferably by someone at Entur.

@leonardehrenfried leonardehrenfried merged commit 98d422c into opentripplanner:dev-2.x Jul 13, 2022
t2gran pushed a commit that referenced this pull request Jul 13, 2022
@t2gran t2gran added this to the 2.2 milestone Jul 14, 2022
@t2gran t2gran added the !Bug Apply to issues describing a bug and PRs witch fixes it. label Jul 14, 2022
@t2gran t2gran deleted the otp2_configure_idempotent_upload_to_gcs branch July 21, 2022 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

!Bug Apply to issues describing a bug and PRs witch fixes it. +Sandbox This will be implemented as a Sandbox feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

File upload to Google Cloud Storage does not recover from transient network failures/server errors

5 participants