refactor(streaming): drop buffered fallback in S3/ECR upload paths#785
Merged
vieiralucas merged 1 commit intomainfrom Apr 26, 2026
Merged
refactor(streaming): drop buffered fallback in S3/ECR upload paths#785vieiralucas merged 1 commit intomainfrom
vieiralucas merged 1 commit intomainfrom
Conversation
Pre-1.0, no need to keep two ingestion modes for the same op. The streaming dispatch always wires body_stream for these routes; the test helper does the same for direct service calls. Buffered branches were dead in production. S3 - put_object: stream is now required, errors with MalformedRequestBody when missing. Spool-to-tempfile path is the only path. SSE-KMS reads the spool back once, non-KMS hands BodySource::File straight to the store. - upload_part: same. Single Bytes-allocation branch removed. - compute_md5 import dropped from objects.rs (now unused — md5 is computed by spool_request_stream). ECR OCI - blob_upload_patch / blob_upload_finish: stream-only via append_stream. Buffered append_bytes_sync branch deleted from these paths. JSON UploadLayerPart still uses append_bytes_sync because that protocol carries a base64-decoded chunk inline. Tests - make_request now wires body_stream from the same bytes it puts in body, so direct unit-test calls to put_object / upload_part take the streaming path. Buffered handlers (put_object_tagging, …) ignore the stream and read req.body as before. Net: 144 lines removed, 93 added.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
5 tasks
vieiralucas
added a commit
that referenced
this pull request
Apr 26, 2026
PR #785 (drop buffered fallback) made S3 PutObject require the streaming dispatch path. The streaming_route guard rejected single- segment paths to avoid mis-routing CreateBucket as PutObject, but that also rejected virtual-hosted PutObject (`/key` with bucket in Host header), causing a 400 MalformedRequestBody. Now the guard also accepts single-segment PUT when the Host header carries a virtual-hosted bucket. Restores e2e::host_routing::s3_virtual_hosted_put_get_delete and the dotted /aws-dash-separated variants alongside it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pre-1.0 cleanup. PR #782 left two ingestion modes side-by-side: streaming spool plus a buffered `BodySource::Bytes` branch from `req.body`. Now that the streaming dispatch + test helper always wire `body_stream`, the buffered branches are dead. Remove them.
Net: -144 / +93 lines.
Test plan
Summary by cubic
Require streaming request bodies for S3 and ECR uploads by removing the buffered fallback. This unifies ingestion to a single streaming path, lowers memory pressure, and simplifies the code.
Refactors
fakecloud-s3PutObject: streaming-only. Spools to a tempfile; SSE-KMS reads the spool back once; non-KMS passesBodySource::Fileto the store.fakecloud-s3UploadPart: streaming-only. Spools and computes MD5/size in constant memory.fakecloud-ecrOCI blob PATCH/PUT: streaming-only viaappend_stream. JSONUploadLayerPartstays buffered (base64 inline).body_streamalongsidebodyso PutObject/UploadPart use the streaming path. Buffered endpoints keep readingbody.Migration
request.body_stream(can be created from the same bytes asrequest.body).Written for commit 7116cad. Summary will update on new commits.