Ceph RGW rejects quoted ETags in If-Match (412) — relay CAS / S3 conformance probe fails on Ceph-backed object storage

[darox]

Summary

When buzz-relay is backed by an S3-compatible object store implemented by Ceph RGW, every conditional-write (CAS) — including the startup S3 conformance probe — fails with HTTP 412, causing the relay to CrashLoop on startup.

Root cause

Ceph RGW returns ETags in quoted form (e.g. "deadbeef", per RFC 7232) in GET responses, but rejects quoted ETags in the If-Match precondition header (returns 412) while accepting the unquoted form (e.g. deadbeef). This is non-compliant with RFC 7232, which requires If-Match to carry a quoted entity-tag.

buzz-relay takes the ETag it received from a GET response and forwards it verbatim into the If-Match header of the subsequent PUT. Because the stored ETag is quoted, every If-Match carries a quoted value that Ceph RGW rejects → 412 → CAS fails.

Relevant code: crates/buzz-relay/src/api/git/store.rs, put_pointer(), the Precond::IfMatch(ETag(tag)) arm:

Precond::IfMatch(ETag(tag)) => {
    headers.insert(
        axum::http::header::IF_MATCH,
        tag.parse().map_err(|_| {
            StoreError::Backend(S3Error::HttpFailWithBody(
                400,
                format!("invalid etag {tag}"),
            ))
        })?,
    );
}

This single path covers both the conformance probe (etag_consistency phase, which calls get_pointer → put_pointer(IfMatch)) and cas_publish.rs (Precond::IfMatch(e.clone())).

Reproduction

• S3 backend: Ceph RGW ( Reef / any recent release).
• Configure buzz-relay against a Ceph-backed bucket (BUZZ_S3_ENDPOINT, BUZZ_S3_BUCKET, credentials).
• Start the relay: the startup S3 conformance probe fails with 412; the relay exits / CrashLoops.

Empirically confirmed with a stdlib SigV4 probe against our Ceph RGW:

• GET returns ETag: "deadbeef" (quoted).
• If-Match: "deadbeef" (quoted) → 412 Precondition Failed.
• If-Match: deadbeef (unquoted) → 200 OK.

Suggested fix

Normalize the ETag before inserting it into If-Match — strip surrounding double quotes:

Precond::IfMatch(ETag(tag)) => {
    // Ceph RGW rejects quoted If-Match values (RFC 7232 non-compliant) but
    // accepts the unquoted form. AWS S3 and MinIO accept both, so stripping
    // is safe across backends.
    let unquoted = tag.trim_matches('"');
    headers.insert(
        axum::http::header::IF_MATCH,
        unquoted.parse().map_err(|_| {
            StoreError::Backend(S3Error::HttpFailWithBody(
                400,
                format!("invalid etag {unquoted}"),
            ))
        })?,
    );
}

This is harmless for AWS S3 and MinIO, which accept both the quoted and unquoted forms, so it does not regress those backends.

Alternatively, a more general fix could normalize the ETag on read (when received from GET) so callers always see the unquoted form, but that is a larger change with broader blast radius.

Environment

• buzz-relay built from chart tag chart-v0.1.6
• S3 backend: Ceph RGW

Happy to open a PR if this approach is acceptable.

[darox]

Opened #3003 with the fix: strip surrounding quotes from the ETag before inserting it into If-Match. AWS S3 and MinIO accept both forms, so no regression on those backends.