Fix UnprocessedKeys bug when restoring from dynamodb #977
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes an issue where we were appending a list of lists to a list of key dicts, which resulted in the following error since boto's batch_get_items expects only a list of key dicts:
We suspect that we've been avoiding this issue because we chunk at 100 keys ourselves before sending to batch_get_items, but this particular job has an extremely large command and therefore set of data and we may have reached the 16MB limit first.
This patch allowed tron to restart cleanly in pnw-devc.
boto3 docs seem to imply
Keys
will always be an array, soextend
seems ot be the right call here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/client/batch_get_item.html