Allow NextToken to be None

[tmshn]

Summary

In many clients, NextToken parameter is used to iterate over all resources, with say while loop.
However, you can't pass NextToken param for the first fetch, which makes codes complicated.
I believe allowing NextToken to be None will solve this.

Example

Here's well-work example to get all cloudwatch alarms.

cloudwatch = boto3.client('cloudwatch')

alarms = cloudwatch.describe_alarms() # First fetch: can't pass NextToken param
# ...
# Do something
# ...
while 'NextToken' in alarms:
    alarms = cloudwatch.describe_alarms(NextToken=alarms['NextToken'])
    # ...
    # Do same thing again
    # ...

If NextToken can be None, the code can be simpler:

cloudwatch = boto3.client('cloudwatch')

alarms = {'NextToken': None}
while True:
    try:
        alarms = cloudwatch.describe_alarms(NextToken=alarms['NextToken'])
    except KeyError:
        break
    else:
        # ...
        # Do something
        # ...

Currently, above code will get following error

ParamValidationError: Parameter validation failed:
Invalid type for parameter NextToken, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

How to fix

For me, botocore is too complicated to make some changes, but I found following code may have something to do with the issue.

https://github.com/boto/botocore/blob/develop/botocore/client.py#L547

One idea is adding following lines:

if 'NextToken' in api_params and api_params['NextToken'] is None:
    del api_params['NextToken']

If you tell me where to fix, I feel no reluctance in making a pull request.

My envoironment

>>> platform.system()
'Darwin'
>>> platform.mac_ver()
('10.11.3', ('', '', ''), 'x86_64')
>>> platform.python_version()
'2.7.11'
>>> botocore.__version__
'1.4.28'
>>> boto3.__version__
'1.3.1'

[JordonPhillips]

The issue is that None could be a valid value for a service. If we start using it as if it were a noop, then we would not be able to support that behavior. If you would like to make pagination simpler, I recommend using our paginator.

[ryguyrg]

@JordonPhillips there are not paginators for all operations.

Only:
CloudWatch.Paginator.DescribeAlarmHistory
CloudWatch.Paginator.DescribeAlarms
CloudWatch.Paginator.ListMetrics

I support @tmshn suggestion of allowing None for nextToken.

[tmshn]

The value None can't be used to represent "no value is set"; since it could be a valid value in some APIs.
IMHO, the better suggestion is to add new (singleton) class like NotSet to represent "no value is set".

It could be like this (an example from PyGithub):
https://github.com/PyGithub/PyGithub/blob/v1.29/github/GithubObject.py#L37-L42

What do you say, @JordonPhillips, @ryguyrg ?

[gmanfunky]

Because ended up here searching for NextToken usage, I wanted to point out that providing a boto.describe_things(NextToken='') seems to result in the behavior @tmshn desired.

[hammadzz]

@gmanfunky not always, you can try ssm api for get_parameters_by_path

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the GetParametersByPath operation: 1 validation error detected: Value '' at 'nextToken' failed to satisfy constraint: Member must have length greater than or equal to 1

Now there is a pagination supported in the latest release of botocore but I am using AWS Lambda which lags behind in updating the SDK. I know I can push the latest botocore package but I am trying to keep my Lambda slim for cold start times.

[ehues]

It would be nice to have a NotSet as suggested by @tmshn.

Failing that, the code in the example can be simplified:

    cw = boto3.client('events')

    extraArgs = {}
    while True:
        rulesPage = cw.list_rules(**extraArgs)

        for rule in rulesPage['Rules']:
            # do something
        
        if 'NextToken' in rulesPage:
            extraArgs['NextToken'] = rulesPage['NextToken']
        else:
            break

it isn't pretty, but it allows you to avoid repeating something.

[0xW1sKy]

For anyone else reading this, i went a similar route but handled as a function to manage it.

ORG=boto3.client('organizations')

def get_org_accounts(**args):
    response = ORG.list_accounts(**args)
    accounts = response["Accounts"]
    if "NextToken" in response:
        args['NextToken'] = response['NextToken']
        accounts += get_org_accounts(**args)
    return accounts

also, there is a new thing you can do with while loops as of python 3.8

ORG=boto3.client('organizations')

response = ORG.list_accounts()
accounts = response["Accounts"]
while token := response.get('NextToken'):
    response = ORG.list_accounts(NextToken=token)
    accounts.extend(response["Accounts"])