Skip to content

Conversation

@w0rks74710n
Copy link
Contributor

When trying to restore tables that were generated using the backup functionality of dynamodump, the Table and GSI schema from the describe_table boto3 API call is not fully compatible with the create_table operation of the API. This PR makes sure that the correct payloads are passed to the create_table API call when GSIs are present in the backup data. It also makes sure that when a backup Table is configured in on-demand mode (provisioned throughput set to 0 for all WCU and RCU) the default value of RESTORE_WRITE_CAPACITY is used instead (otherwise the parameter validation of the API will fail).

Without these changes, the output of the process will throw the following error when restoring a table with GSIs:

Traceback (most recent call last):
  File "dynamodump/dynamodump/dynamodump.py", line 1389, in <module>
    main()
  File "dynamodump/dynamodump/dynamodump.py", line 1354, in main
    do_restore(
  File "dynamodump/dynamodump/dynamodump.py", line 831, in do_restore
    dynamo.create_table(
  File "/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/lib/python3.8/site-packages/botocore/client.py", line 677, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/lib/python3.8/site-packages/botocore/client.py", line 725, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/lib/python3.8/site-packages/botocore/validate.py", line 337, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid value for parameter ProvisionedThroughput.ReadCapacityUnits, value: 0, valid min value: 1
Unknown parameter in GlobalSecondaryIndexes[0]: "IndexStatus", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[0]: "IndexSizeBytes", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[0]: "ItemCount", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[0]: "IndexArn", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[0].ProvisionedThroughput: "NumberOfDecreasesToday", must be one of: ReadCapacityUnits, WriteCapacityUnits
Invalid value for parameter GlobalSecondaryIndexes[0].ProvisionedThroughput.ReadCapacityUnits, value: 0, valid min value: 1
Unknown parameter in GlobalSecondaryIndexes[1]: "IndexStatus", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[1]: "IndexSizeBytes", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[1]: "ItemCount", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[1]: "IndexArn", must be one of: IndexName, KeySchema, Projection, ProvisionedThroughput
Unknown parameter in GlobalSecondaryIndexes[1].ProvisionedThroughput: "NumberOfDecreasesToday", must be one of: ReadCapacityUnits, WriteCapacityUnits
Invalid value for parameter GlobalSecondaryIndexes[1].ProvisionedThroughput.ReadCapacityUnits, value: 0, valid min value: 1

@w0rks74710n
Copy link
Contributor Author

Closed in order to fix failing tests first

@w0rks74710n w0rks74710n reopened this Sep 22, 2021
Copy link
Owner

@bchew bchew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @w0rks74710n

@bchew bchew merged commit 5bdaef5 into bchew:master Sep 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants