DynamoDB client behaves differently depending on whether client was directly created or obtained from a resource

[nrosner]

Describe the bug
If you create a DynamoDB resource and later obtain its client instance

res = boto3.resource('dynamodb')
cli = res.meta.client

that client behaves differently from one that is created directly, as in

cli = boto3.client('dynamodb')

Code written for the latter breaks with the former. This was already reported in issue #2125. The issue was closed, but I believe the problem still prevails.

Steps to reproduce

A simple query will raise a ClientError when the client was obtained from the resource. You will need a table, otherwise the non-existent table exception will mask the ClientError.

For simplicity, let's use the DynamoDB local server which has zero configuration. Download the local server and launch it with

java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb

Now add a table:

import boto3

dynamodb = boto3.resource('dynamodb',
    endpoint_url='http://localhost:8000',
    aws_access_key_id='anythingWorks-LocalServerDoesntCare',
    aws_secret_access_key='anythingWorks-LocalServerDoesntCare',
    region_name='us-west-2'
)

table = dynamodb.create_table(
    TableName='Users',
    KeySchema=[{'AttributeName': 'userid', 'KeyType': 'HASH'}],
    AttributeDefinitions=[{'AttributeName': 'userid', 'AttributeType': 'S'}],
    ProvisionedThroughput={'ReadCapacityUnits': 10, 'WriteCapacityUnits': 10}
)

table.wait_until_exists()

This query succeeds with a DynamoDB client obtained via boto3.client():

import boto3

client = boto3.client('dynamodb',
    endpoint_url='http://localhost:8000',
    aws_access_key_id='anythingWorks-LocalServerDoesntCare',
    aws_secret_access_key='anythingWorks-LocalServerDoesntCare',
    region_name='us-west-2'
)

response = client.query(
    TableName="Users",
    KeyConditionExpression="userid = :u",
    ExpressionAttributeValues={":u": {"S": "jsmith"}}
)

The same query fails if the client was obtained from a DynamoDB resource:

import boto3

resource = boto3.resource('dynamodb',
    endpoint_url='http://localhost:8000',
    aws_access_key_id='anythingWorks-LocalServerDoesntCare',
    aws_secret_access_key='anythingWorks-LocalServerDoesntCare',
    region_name='us-west-2'
)

# Obtain the client from the resource
client = resource.meta.client

# The same query fails if the client was obtained from a resource
response = client.query(
    TableName="Users",
    KeyConditionExpression="userid = :u",
    ExpressionAttributeValues={":u": {"S": "jsmith"}}
)

The above yields the following error:

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: One or more parameter values were invalid: Condition parameter type does not match schema type

The query only works with such a client if the ExpressionAttributeValues are deserialized:

# If the client was obtained from the resource, the query must be
response = client.query(
    TableName="Users",
    KeyConditionExpression="userid = :u",
    ExpressionAttributeValues={":u": "jsmith"}
)

Expected behavior
I would expect the client to always behave in the same way. Regardless of whether you obtained it from a resource, it is still a client. Once you obtain the low-level client, you should use low-level representations, right?

The DynamoDB resource is a higher-level abstraction. It is very nicely Pythonic, especially in terms of implicit (de)serialization. I use it whenever I can, but sometimes you still need the client, because the resource doesn't cover the whole API.

Being able to obtain the client from the resource gives developers the best of both worlds.

Stack trace

>>> response = client.query(
...     TableName="Users",
...     KeyConditionExpression="userid = :u",
...     ExpressionAttributeValues={":u": {"S": "jsmith"}}
... )

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: One or more parameter values were invalid: Condition parameter type does not match schema type

[swetashre]

Thank you for your post. Currently this is the expected behavior. In your case when you are creating client from resource it is using resource's function to transform parameter before making api call. So you are getting error.
In this case i would recommend using low level client instead of constructing from resource.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[orfisko]

This is ridiculous. The boto3 library is a nightmare, specifying types for writing data to a database.... Everything is specified elsewhere, why would we need to specify it again?