Retry storage operation on 409 conflict#325
Conversation
| } | ||
| if err := storageClient.WriteProtobuf(ctx, uri, storage.Options{}, literalMap); err != nil { | ||
|
|
||
| err = async.RetryOnSpecificErrors(5, async.RetryDelay, func() error { |
| } | ||
|
|
||
| func isRetryableError(err error) bool { | ||
| if e, ok := err.(*googleapi.Error); ok && e.Code == 409 { |
There was a problem hiding this comment.
A bit hard to reproduce this as my local version does not connect to GCP and even if I am not sure I could get this error to reappear. I used this https://github.com/graymeta/stow/blob/973a61f346d598a566affb53c4698764a67df164/google/location.go#L38-L42 to understand how to parse it
There was a problem hiding this comment.
I think this looks right, but it also looks like flytestdlib wraps the error: https://github.com/flyteorg/flytestdlib/blob/15cac7bec8470974050ed02343cda8191e5f14bc/storage/stow_store.go#L264 so you may need to unwrap the error. I think IsCausedBy works here to get the underlying googleapi error.
Mind adding a unit test to verify this?
sonjaer
force-pushed
the
add-retry
branch
2 times, most recently
from
e9c716c to
1d66278
Compare
sonjaer
requested review from
EngHabu,
anandswaminathan,
katrogan,
kumare3 and
wild-endeavor
as code owners
Codecov Report
@@ Coverage Diff @@
## master #325 +/- ##
==========================================
+ Coverage 58.10% 58.16% +0.06%
==========================================
Files 148 148
Lines 10691 10707 +16
==========================================
+ Hits 6212 6228 +16
Misses 3796 3796
Partials 683 683
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
sonjaer
force-pushed
the
add-retry
branch
4 times, most recently
from
a169adc to
e4ac309
Compare
Signed-off-by: Sonja Ericsson <sonjae@spotify.com>
|
Congrats on merging your first pull request! 🎉 |
* add retry wip Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * add tests Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix Signed-off-by: Sonja Ericsson <sonjae@spotify.com> * fix lint Signed-off-by: Sonja Ericsson <sonjae@spotify.com>
2 participants
TL;DR
Fix for flyteorg/flyte#1997
Type
Are all requirements met?
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
Remove the 'fixes' keyword if there will be multiple PRs to fix the linked issue
fixes https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/