feat(services): support if_match conditional writes for azblob and gcs#7890
feat(services): support if_match conditional writes for azblob and gcs#7890shanielh wants to merge 1 commit into
Conversation
Azure Blob's Put Blob API supports If-Match with an arbitrary ETag, so wire it in alongside the existing If-None-Match support. GCS's JSON API has no ETag-based conditional write mechanism, only generation-number query params. Implement write_with_if_match and write_with_if_none_match on top of ifGenerationMatch/ifGenerationNotMatch, reusing the object generation already exposed via Metadata::version(). S3 is unchanged: its If-None-Match header only accepts the literal wildcard, which is already covered by if_not_exists().
e5dd147 to
253431f
Compare
Xuanwo
left a comment
There was a problem hiding this comment.
Thank you for working on this.
The azure part seems good to me, but I strongly againest the gcs change.
| pub write_with_cache_control: bool, | ||
| /// Indicates if conditional write operations using If-Match are supported. | ||
| /// | ||
| /// The value passed to `if_match` is normally a literal ETag. GCS is an exception: |
There was a problem hiding this comment.
No, we don't do this. It's very clear that if-match accepts an etag.
Think about this, we now have a user want to support all of s3, azblob and gcs.
The user is using APIs like:
let expected_etag = op.stat(pathx).await?.etag();
let _ = op.write_with(pathx, data).if_match(expected_etag).await?;Then this user will go crazy to find that it doesn't work on gcs, because if_match always didn't match the give etag. After reading docs, they realized that they need to do:
let expected_etag = if op.scheme() == "gcs" {
op.stat(pathx).await?.version()
} else {
op.stat(pathx).await?.etag()
};
let _ = op.write_with(pathx, data).if_match(expected_etag).await?;I don't think they will love opendal again.
| /// Indicates if conditional write operations using If-None-Match are supported. | ||
| /// | ||
| /// The value passed to `if_none_match` is normally a literal ETag. GCS is an exception: | ||
| /// it has no ETag-based conditional write API, so it repurposes this field to carry |
| write_with_content_type: true, | ||
| write_with_content_encoding: true, | ||
| write_with_user_metadata: true, | ||
| write_with_if_match: true, |
There was a problem hiding this comment.
Please don't declare a feature that services didn't support.
|
@Xuanwo maybe we should add a new capability for write if version match? I believe the write if none is fine for GCS. If so. I'll be happy to implement it for GCS. |
Which issue does this PR close?
Closes #7889
Rationale for this change
Conditional-write support (
if_match/if_none_match) was inconsistent across object-storage services: S3 supportedif_matchbut not arbitrary-ETagif_none_match(a genuine AWS API limitation — S3'sx-amz-if-none-matchonly accepts the literal*), Azure Blob supportedif_none_matchbut notif_match, and GCS supported neither for writes. This PR closes the two gaps that are actually fixable given each provider's API (Azure'sif_match, GCS's generation-based conditional writes), while leaving S3 as-is since its gap is not implementable through the S3 API.What changes are included in this PR?
write_with_if_matchcapability and wired theIf-Matchheader intoazblob_put_blob_request, mirroring the existingif_none_matchhandling.write_with_if_matchandwrite_with_if_none_matchcapabilities. GCS's JSON API has no ETag-based conditional write mechanism, only generation-number query params, so these are implemented on top ofifGenerationMatch/ifGenerationNotMatch. The object generation number is exposed via the existingMetadata::version()field (already populated by GCS's stat/read/list paths), so no newMetadatafields were needed. Non-numericif_match/if_none_matchvalues return anUnsupportederror.core/core/src/types/capability.rsthat GCS repurposesif_match/if_none_matchto carry a generation number rather than a literal ETag; added a corresponding note to GCS'sdocs.md.test_write_with_if_match/test_write_with_if_none_matchtests (core/tests/behavior/async_write.rs) to use the object's generation (Metadata::version()) as the match token for GCS and its ETag for all other backends, since enabling the new capability flags makes these shared tests exercise GCS for the first time.Are there any user-facing changes?
Yes:
if_matchtowrite_with, matching the behavior already available forif_none_match.if_match/if_none_matchtowrite_withfor optimistic-concurrency writes, but must pass the object's generation number (fromMetadata::version()), not a literal ETag — this is called out in the capability doc comments and GCS'sdocs.md.AI Usage Statement
This PR was developed with Claude Code (Anthropic), model Claude Sonnet 5, including codebase investigation, implementation, and verification (
cargo fmt,cargo clippy -D warnings,cargo check --all-features, doc tests). Live behavior tests against real GCS/Azure backends were not run — no credentials were available in the development environment.