v0.3.164 - Object-Storage Upload Adapter (Stage 2): Real SigV4 R2/S3 Transport

v0.3.164 - Object-Storage Upload Adapter (Stage 2): Real SigV4 R2/S3 Transport

v0.3.164 lands Stage 2 of the WOM #11 live object-storage upload adapter: a real,
hand-rolled AWS SigV4 R2/S3-compatible upload transport. WOM is now
network-CAPABLE for an approved object-storage upload. Capable is not automatic:
a live --approve upload still fails closed without env-only credentials, a safe
--reviewed-by, a resolvable endpoint/bucket, and a met tiered tiny-first gate.

This release ships with unproven_against_live_provider: true on the release note
and every execution receipt, and stays that way until the tiny-first human runbook
below confirms the first live object.

What ships

• Real SigV4 transport (S3CompatibleTransport). PUT and UploadPart sign the
  real lowercase-hex payload sha256; HEAD/GET sign UNSIGNED-PAYLOAD. The
  canonical URI/query encoding is RFC-3986 correct (unreserved A-Za-z0-9-._~,
  everything else uppercase %XX, content-addressed sha256/<first2>/<hex>
  key separators preserved, path encoded once). Header canonicalization,
  SignedHeaders, the 4-line string-to-sign, and the 4-stage signing key
  (kDate→kRegion→kService→kSigning, raw-byte intermediates) are all
  byte-exact. Region is auto for Cloudflare R2 (threaded into both the
  credential scope and kRegion); generic-s3 requires an explicit region.
  The SigV4 core is pinned against the published AWS documented example — the
  canonical request, string-to-sign, and final signature all match the third-party
  values.
• Whole-object verification by re-download-and-hash (the showstopper this stage
  resolves). The executor compares a lowercase-hex sha256 at HEAD-after. Rather
  than depend on a provider-stored SHA-256 — which is not reliably available: R2
  does not implement GetObjectAttributes, marks the x-amz-checksum-* headers
  "Feature Not Implemented", and a SHA-256 multipart checksum can only be COMPOSITE
  (never the whole-object hash) — the transport verifies by HeadObject (presence
  • size) followed by GetObject and re-hashing the returned bytes to hex. Uploads
    still sign the real payload SHA-256 in SigV4 (x-amz-content-sha256) for on-wire
    integrity, but no provider checksum surface is trusted. The shipped executor
    comparison is unchanged and now passes for a genuine upload.
• Multipart. Objects at or above the 5 GiB threshold use a plain multipart
  upload (create → part PUTs → complete). No x-amz-checksum-type: FULL_OBJECT is
  sent — that combination is unsupported for SHA-256 on both AWS S3 and R2. The
  CompleteMultipartUpload request carries only the <Part> list and an explicit
  text/xml Content-Type (AWS rejects application/x-www-form-urlencoded for that
  call). Whole-object integrity is verified by the same HEAD+GET-rehash path. If the
  re-download does not hash to the content id, the object FAILS — it never passes on
  ETag or size, and the completed-but-wrong object is deleted so no orphan accrues
  storage cost. A rate_limited failure mid-multipart is retried by the bounded
  loop to the attempt ceiling, exactly like the single-PUT path.
• Bounded retry + cost ceilings. A per-object retry loop backs off exponentially
  with jitter up to a hard attempt ceiling on rate_limited (429/503/SlowDown/
  InternalError/conn-reset), then fails closed as failed_rate_limited. An auth
  status (403/400/SignatureDoesNotMatch/InvalidAccessKeyId/RequestTimeTooSkewed)
  fails closed immediately with ZERO retries — retrying a bad signature burns
  Class-A ops and can never succeed. A HARD cumulative provider-PUT ceiling
  (OBJECT_STORAGE_TOTAL_PUT_CEILING) bounds cost across the whole run,
  independent of --max-objects. retry_summary.backoff_ms_total is now a real
  accumulated value.
• Tiered tiny-first gate. A run may not exceed the live-acceptance tier the
  store's prior durable execution receipts have proved. A bulk first-live run
  REFUSES with tiered_gate_unmet until the single small object is proved. Tier
  advancement is derived from durable receipt facts — result_status,
  bytes_uploaded, and the count of distinct successful one-per-object receipts —
  not a bare flag the caller sets. A batch (tier 3) requires a proven large-object
  / multipart upload (tier 2) plus enough distinct landed objects. Each live tier
  stays a human --approve step.
• Single networking seam. Every transport method builds a fully-signed request
  and hands it to an injected send. The only place stdlib networking is reachable
  is one _default_urllib_sender() factory; the CLI wires it only for a real
  --approve run. No dependency was added (still PyYAML only).
• Secret discipline extended. The direct-value containment guard now also covers
  the derived SigV4 signing key, as belt-and-suspenders over the primary structural
  guarantee that signing material lives in transport locals only. No request
  headers, Authorization, StringToSign, CanonicalRequest, or provider error
  body is ever recorded.

Capable != automatic

live_object_upload_adapter_implemented and provider_api_call_implemented are
now true — the live adapter really ships. But archive object-storage-upload --approve still fails closed unless every gate is met: env-only credential refs,
a safe --reviewed-by, a resolvable non-secret endpoint/bucket, and a met tiered
gate. Read-only object-storage-upload-plan/-verify and the read-only MCP tools
are unaffected.

Residual (only a real endpoint proves these)

Carried as unproven_against_live_provider: true until the first live object:
signature ACCEPTANCE by R2's authorizer; real read-after-write consistency of the
HEAD+GET verification path; real 429/SlowDown timing; the ±15-minute clock-skew
window; and real Class-A/B billing (note the HEAD+GET verification adds one
GetObject read per uploaded object). Whole-object integrity itself no longer
depends on any provider checksum surface, so it is not a live unknown.

Human tiny-first runbook (tier 1, first live object)

Upload exactly one small object, verify it end-to-end by hand, then advance tiers.

WOM_R2_ACCESS_KEY_ID=<id> WOM_R2_SECRET_ACCESS_KEY=<secret> \
archive object-storage-upload <archive-root> \
  --provider-kind cloudflare-r2 \
  --store-ref storage:account:<label> \
  --endpoint-host <account>.r2.cloudflarestorage.com \
  --bucket <bucket-name> \
  --access-key-id-ref env:WOM_R2_ACCESS_KEY_ID \
  --secret-access-key-ref env:WOM_R2_SECRET_ACCESS_KEY \
  --only sha256:<one-small-object-hex> \
  --max-objects 1 \
  --skip-uploaded \
  --reviewed-by kim \
  --approve

After it succeeds, verify by hand: the execution receipt under
receipts/providers/object-storage-executions/, the manifest transition to a
wom_uploaded location, and the remote after-HEAD. Then advance the tiers, each a
separate human --approve run:

1. T1 one small object (above).
2. T2 one object at or above the multipart threshold.
3. T3 a small batch (~10) with a deliberate mid-batch abort proving the
   manifest lock + resume ledger.
4. T4 a --skip-uploaded re-run proving no declared_uploaded false-skip.
5. T5 the full set.

Only after the first live object is confirmed does the doc caveat flip to
proven_against_live_provider: cloudflare-r2, one object.

What is deliberately NOT in this release

No live object has been uploaded from this release — the tiny-first runbook is a
human step. Real signature acceptance, checksum surfacing, multipart, throttle
timing, and cost are validated live, tier by tier, behind explicit human approval.