S3: paginator returns incorrect NextContinuationToken

[mikedambrogia]

moto 4.2.12 installed via pip install on 3.10.12

What actually happens: the pagination NextContinuationToken value returned is actually a key in the list of s3 file contents returned, and not a unique value/formatted as s3 returns the value. s3 returns a long hash of some sort.

That said, the pagination mock, when passed the non-standard NextContinuationToken value does return the correct next result set values.

Debugging the code snippet below will help explain this.

I'd expect: a NextContinuationToken value formatted as some sort of long hash that AWS provides. But there isn't much value in asserting that the NextContinuationToken matches some sort of hash pattern. I'm left scratching my head while trying to determine if this is actually just an anomoly, or if this is actually a bug.

import boto3
import json
from moto import mock_s3


def get_files(NextContinuationToken=None):

    s3client = boto3.client("s3")
    paginator = s3client.get_paginator("list_objects_v2")

    paginate_options = {
        "Bucket": "test_bucket",
        "PaginationConfig": {
            "PageSize": 5,
            }
        }

    if NextContinuationToken is not None:
        paginate_options["PaginationConfig"]["StartingToken"] = NextContinuationToken

    result = paginator.paginate(
        **paginate_options
    )

    data = {}
    data["_metadata"] = {}
    data["_metadata"]["query"] = {}

    keys = []
    try:
        for page in result:
            if "NextContinuationToken" in page.keys():
                NextContinuationToken = page["NextContinuationToken"]
            else:
                NextContinuationToken = ""
            break
    except Exception as e:  # noqa F841
        message = "invalid value for NextContinuationToken"
        raise Exception(message)

    data["_metadata"]["NextContinuationToken"] = NextContinuationToken

    if "Contents" in page.keys():
        for file in page["Contents"]:
            keys.append(file["Key"])

    data["results"] = keys

    return data


@mock_s3
def test_list_files_post():

    # setup s3 mock bucket
    bucket = "test_bucket"
    region = "us-west-2"
    response = boto3.client("s3").create_bucket(
        Bucket=bucket,
        CreateBucketConfiguration={
            'LocationConstraint': region,
        }
    )
    assert response['ResponseMetadata']['HTTPStatusCode'] == 200

    # create 10 mock s3 files to test for NextContinuationToken
    s3_client = boto3.client("s3", region_name=region)
    body = "this is my test data"
    for count in range(9):
        key = f"filename-{count}.json"
        try:
            s3_client.put_object(
                Bucket=bucket,
                Key=key,
                Body=body
                )
        except Exception as e:
            print(str(e))

    # this has been returning filename-4.json as my NextContinuationToken value
    response1 = get_files()

    # passing ini filename-4.json as the NextContinuationToken does return keys 5-9 correctly
    response2 = get_files(response1["_metadata"]["NextContinuationToken"])

[bblommers]

Hi @mikedambrogia! The client.list_objects_v2 documentation explicitly states:

ContinuationToken is obfuscated and is not a real key.

So Moto should indeed do the same thing. Having an obfuscated token would also allow us to add more validation that subsequent requests use the same attributes.

As far as I can see, there is no actual bug here - as the AWS API still behaves as expected. But I'll mark it as an enhancement.

[bblommers]

@mikedambrogia Because this is a breaking change, sending an obfuscated token is part of the Moto v5 release.

We've released an early alpha to get some feedback: moto==5.0.0alpha2

Note that as part of this release, there is only a single decorator now:

@mock_aws

This replaces all the service-specific decorators.

See the release announcement here for more details: #7198

[bblommers]

Moto v5 has now been released with this change: https://pypi.org/project/moto/5.0.0/