pkg/blobserver/google/cloudstorage: don't overwrite a blob that's already present #1287
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've been encountering rate-limit-exceeded errors trying to dump a lot of files into a GCS backend (via
pk-mount
).I thought what I needed was a rate limiter. That's #1285.
But the problem persisted and I discovered that the GCS blobserver was repeatedly re-uploading the same blob over and over (to wit: my PGP public key). So I wrote this PR, to skip uploading a blob if it's already present.
But that's still not good enough to make the rate-limit-exceeded errors go away. I figure this change is still a good one anyway, so here it is while I dig a bit more.Fixed in 18e8c5b by using an explicit call toAttrs
before the call toNewWriter
.