AWS account credentials in the Model Meta class is not used in threaded code

[atsuoishimoto]

Problem

An UnrecognizedClientException exception is thrown when using threads to reference a table.

How to reproduce

1. Save the following code as repro.py and modify VALID_ACCESS_KEY and VALID_SECRET_ACCESS_KEY with valid AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

import time, sys, os, traceback, io
from concurrent.futures import ThreadPoolExecutor
from pynamodb.models import Model
from pynamodb.attributes import UnicodeAttribute


# Set the credentials not to be read from ~/.aws/credentials
os.environ['AWS_ACCESS_KEY_ID'] = "INVALID_KEY"
os.environ['AWS_SECRET_ACCESS_KEY'] = "INVALID_SECRET"

class UserModel(Model):
    class Meta:
        table_name = 'test_accout_dynamodb222'
        region = 'us-west-1'
        aws_access_key_id = 'VALID_ACCESS_KEY' 
        aws_secret_access_key = 'VALID_SECRET_ACCESS_KEY'

    email = UnicodeAttribute(hash_key=True)
    first_name = UnicodeAttribute()
    last_name = UnicodeAttribute()

def init():
    # Initialize test table
    UserModel.create_table(wait=True,read_capacity_units=1, write_capacity_units=1)
    user = UserModel('test@example.com', first_name='Samuel', last_name='Adams')
    user.save()
    print("Initialized")
    return

if len(sys.argv) == 2 and int(sys.argv[1]):
    init()
    sys.exit(1)

def get_item(n):
    for i in range(10):
        for user in UserModel.query("test@example.com"):
            return print(n, i, user)

with ThreadPoolExecutor(max_workers=10) as e:
    futs = [e.submit(get_item, i) for i in range(10)]
    for fut in futs:
        try:
            fut.result()
        except Exception:
            traceback.print_exc()
            pass

2. Create table with following command.

python3 repro.py 1

3. Run the test with the following command.

python3 repro.py

Running repro.py will cause the following error.

Traceback (most recent call last):
  File "/Users/ishimoto/src/PynamoDB/pynamodb/connection/base.py", line 1387, in query
    return self.dispatch(QUERY, operation_kwargs, settings)
  File "/Users/ishimoto/src/PynamoDB/pynamodb/connection/base.py", line 340, in dispatch
    data = self._make_api_call(operation_name, operation_kwargs, settings)
  File "/Users/ishimoto/src/PynamoDB/pynamodb/connection/base.py", line 469, in _make_api_call
    raise VerboseClientError(
pynamodb.exceptions.VerboseClientError: An error occurred (UnrecognizedClientException) on request (QOC25V56NVI2T6UC2QM09B5TOVVV4KQNSO5AEMVJF66Q9ASUAAJG) on table (test_accout_dynamodb) when calling the Query operation: The security token included in the request is invalid.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ishimoto/src/PynamoDB/reproduce.py", line 54, in <module>
    fut.result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/ishimoto/src/PynamoDB/reproduce.py", line 47, in get_item
    for user in UserModel.query("test@example.com"):
  File "/Users/ishimoto/src/PynamoDB/pynamodb/pagination.py", line 193, in __next__
    self._get_next_page()
  File "/Users/ishimoto/src/PynamoDB/pynamodb/pagination.py", line 179, in _get_next_page
    page = next(self.page_iter)
  File "/Users/ishimoto/src/PynamoDB/pynamodb/pagination.py", line 113, in __next__
    page = self._operation(*self._args, settings=self._settings, **self._kwargs)
  File "/Users/ishimoto/src/PynamoDB/pynamodb/connection/table.py", line 274, in query
    return self.connection.query(
  File "/Users/ishimoto/src/PynamoDB/pynamodb/connection/base.py", line 1389, in query
    raise QueryError("Failed to query items: {}".format(e), e)
pynamodb.exceptions.QueryError: Failed to query items: An error occurred (UnrecognizedClientException) on request (QOC25V56NVI2T6UC2QM09B5TOVVV4KQNSO5AEMVJF66Q9ASUAAJG) on table (test_accout_dynamodb) when calling the Query operation: The security token included in the request is invalid.

Cause

When botocore client lost credentials here, a new Session object is created with default credentials if the running thread does not own Session.

Resolution

Specify credentials when creating Session object.

[atsuoishimoto]

Any progress on this issue?
Please let me know if you have any questions.

[ikonst]

Thanks for the fix, and sorry for the delay.

[atsuoishimoto]

Thank you for the review! I deeply appreciate it.