fix(s3): set NextMarker in ListObjects V1 response when truncated#1048
Conversation
Coverage Report for CI Build 24995061090Coverage increased (+0.005%) to 71.588%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats💛 - Coveralls |
| Name: v2Result.Name, | ||
| Prefix: v2Result.Prefix, | ||
| Marker: v2Result.NextContinuationToken, | ||
| ...(v2Result.IsTruncated && v2Result.NextContinuationToken |
There was a problem hiding this comment.
This element is returned only if you have the delimiter request parameter specified
I think we should handle this condition as well and adjust tests accordingly
There was a problem hiding this comment.
Done in ada90f1. Gated the NextMarker spread on command.Delimiter and flipped the no-delimiter test to assert NextMarker is undefined.
| EncodingType: list.responseBody.ListBucketResult.EncodingType, | ||
| Name: v2Result.Name, | ||
| Prefix: v2Result.Prefix, | ||
| Marker: v2Result.NextContinuationToken, |
There was a problem hiding this comment.
| Marker: v2Result.NextContinuationToken, | |
| ...(command.Marker !== undefined ? { Marker: command.Marker } : {}), |
@supabase/storage I think we should update this separately for compliance. It could be breaking theoretically for some but I guess minimal risk
There was a problem hiding this comment.
Held off on this one in the current push. Details in the top-level reply above: the pre-existing list all keys with pagination test paginates via resp.Marker, and swapping to command.Marker makes that undefined on the first page and breaks external callers doing the same. Happy to fold it in here with a test migration, or split it into a follow-up PR so this can merge cleanly. Let me know which you prefer.
There was a problem hiding this comment.
I think this change is good as it is to fix NextMarker.
Let's prepare Marker issue separately to make it obvious so that we can refer people to the exact minimal change if they are affected and mark it breaking.
|
Thanks @ferhatelmas! Addressed the NextMarker + delimiter condition in ada90f1. Tightened the condition to require On the Happy to either (a) add it here with a test migration to |
S3 V1 ListObjects is expected to return NextMarker alongside IsTruncated so pagination-aware clients (like the AWS SDK v3 paginator, which reads output.NextMarker before falling back to the last Contents key) can continue iterating. The handler was never populating it, so a client that specifically keyed off NextMarker got undefined. Sets NextMarker to the underlying V2 NextContinuationToken (which is the plain last-key when cursorV1 is set). Only emitted when IsTruncated is true and the value is present, so the XML serializer does not render an empty <NextMarker/> element in the non-truncated case. Marker is intentionally left unchanged here. It is still populated with the next-page hint rather than the echoed input Marker the spec asks for, but fixing that is a backwards-incompatible change for clients that coincidentally work against today's value; tracked separately.
Biome flagged the client.send(new ListObjectsCommand(...)) call in the new truncation test as a multi-line expression that should collapse onto a single line. Autoformat applied, no behavior change.
Per the S3 ListObjects V1 spec, NextMarker is only returned when the Delimiter request parameter is specified. Tighten the condition to require command.Delimiter in addition to IsTruncated, and flip the no-delimiter test to assert NextMarker is omitted.
ada90f1 to
50b8b1c
Compare
Closes #1047.
Summary
S3
ListObjectsV1 has to returnNextMarkeralongsideIsTruncatedso that pagination-aware clients can advance without reverse-engineering the cursor offContents.S3ProtocolHandler.listObjectsnever emitted it. The next-page hint was leaking into theMarkerfield instead, and any client that specifically readNextMarkergotundefined.Sets
NextMarkerto the V2NextContinuationToken, which undercursorV1: trueis the plain last key (results.nextCursorKey). Only emitted whenIsTruncated === trueand the token is present, via a conditional spread, so the XML serializer does not render an empty<NextMarker/>element in the non-truncated case.Markeris intentionally left unchanged. It is still populated with the next-page hint rather than the echoed inputMarkerper the spec. The AWS SDK paginator prefersNextMarkernow that it is available, and changingMarker's value is a backwards-incompatible change for any client that has grown to depend on it. Tracked separately in the issue.Test plan
Three tests added / adjusted in
src/test/s3-protocol.test.tsunderListObjectCommand:sets NextMarker when the response is truncated. Flat listing, confirmsNextMarkeris defined and usable as next request'sMarker.omits NextMarker when the response is not truncated. Confirms the field is absent (not empty string).sets NextMarker when truncated with a delimiter. Delimiter listing, confirmsNextMarkercarries the cursor and feeding it back returns the remaining common prefixes.All
ListObjectCommandtests pass against a local Docker storage stack; existingListObjectsV2Commandtests are unaffected.