feat(providers): expand PutBlob API to allow for idempotent puts #1654
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jkowalski
reviewed
Jan 13, 2022
Codecov Report
@@ Coverage Diff @@
## master #1654 +/- ##
==========================================
- Coverage 70.72% 70.69% -0.04%
==========================================
Files 386 386
Lines 30760 30775 +15
==========================================
Hits 21755 21755
- Misses 7336 7351 +15
Partials 1669 1669
Continue to review full report at Codecov.
|
|
Is this only going to work for GCS? What about S3-like providers? |
Currently only GCS, yes. If/When S3 has support for write-once then we can extend. |
adowair
force-pushed
the
idempotent-put-blob
branch
from
9ab4138
to
ac8ce66
Compare
jkowalski
changed the title
adowair
force-pushed
the
idempotent-put-blob
branch
from
ac8ce66
to
54ee2ce
Compare
When `DoNotRecreate` is set as true, the blob put operation should only succeed if no blob with the given blob ID already exists. Othwerwise, `ErrBlobAlreadyExists` is returned.
By default, storage providers should not support idempotent creates. This commit adds error handling to exit early if `DoNotRecreate` is set to true. The commit also verifies this behavior in the provider validation test.
When PutBlob options were introduced, error handling logic for them was implemented for the Sharded storage interface. However, the behavior of different providers that implement Sharded can be different, so it's better to push the options down to be processed in the provider implementations.
adowair
force-pushed
the
idempotent-put-blob
branch
from
54ee2ce
to
48aaf2d
Compare
To unify error handling code and make it more maintainable, introduce a new error type `blob.ErrUnsupportedPutBlobOption`, which is to be returned whenever a storage provider implementation is given put options it does not support.
adowair
force-pushed
the
idempotent-put-blob
branch
from
48aaf2d
to
4fd99d9
Compare
|
LGTM, although I'm still unsure how broadly useful this will be, if GCS is the only provider that can support it. |
jkowalski
approved these changes
Jan 27, 2022
|
Thank you @jkowalski, I will also investigate other storage providers and add support in future PRs as is possible. |
jkowalski
added a commit
to jkowalski/kopia
that referenced
this pull request
Jan 29, 2022
There was a small regression in kopia#1654 - only in negative tests. Also removed unnecessary retries on ErrUnsupportedPutBlobOption + switched provider test to run once per day, instead of 12 times/day.
julio-lopez
pushed a commit
that referenced
this pull request
Jan 29, 2022
There was a small regression in #1654 - only in negative tests. Also removed unnecessary retries on ErrUnsupportedPutBlobOption + switched provider test to run once per day, instead of 12 times/day.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today, executing a PutBlob operation will always overwrite a blob if it exists. It would be helpful for some customer workflows to be able to write blobs only if they don't exist (i.e. don't overwrite/recreate blobs). This PR expands the PutBlob API with an additional option
DoNotRecreate, which toggles this feature. Note that not all backends support idempotent creates, so this PR also implements error logic to rejects such PutBlobs if they can't be executed.