Spikes in response time when calling DynamoDb

[LeonWinst]

Describe the bug

Bumping AWSSDK.SSO from version 4.0.1.3 to 4.0.1.6 and AWSSDK.SSOIDC from version 4.0.2 to 4.0.2.3 has resulted in frequent spikes when calling DynamoDb.

We have fixed the issue by reverting back to versions 4.0.1.3 and 4.0.2.3.

The spikes seemed to mainly effect queries that were querying GSIs. No throttle exceptions were logged inside the DynamoDb console. The query was searching on the primary & secondary keys so response times upwards of 20 seconds are not expected.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

DynamoDb queries do not result in frequent spikes.

Current Behavior

Querying GSIs result in frequent spikes. Example from an Xray trace - a batch of 9 queries running in parallel results in two taking 30 seconds:

Reproduction Steps

Use the following packages:

• AWSSDK.SSO: 4.0.1.6
• AWSSDK.SSOIDC: 4.0.2.3
• AWSSDK.DynamoDbv2: 4.0.6.3

Query a GSI using a PK & SK lookup in a Parallel.ForeachAsync() with a MaxDegreeOfParalllelism of 32

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

• AWSSDK.SSO: 4.0.1.6
• AWSSDK.SSOIDC: 4.0.2.3
• AWSSDK.DynamoDbv2: 4.0.6.3

Targeted .NET Platform

.NET 8

Operating System and version

AmazonLinux & Windows10

[dscpinheiro]

Can you share more detailed reproduction code? How your table is configured would also help.

[dscpinheiro]

Looking at the versions though, it does look like it could be related to #4053

There are more details on that issue, but I had to revert a change with how credentials are refreshed (because the previous behavior was broken).

[LeonWinst]

We have a GSI setup projecting several attributes. The primary key is the same as the main table. However, the sort key has been swapped out for another guid for us to query.

Read capacity: on-demand
Write capacity: on-demand
The GSI holds around 800MB with an item count of 1.5m

Example of the query structure:

{
	PK - equals - guid.toString()
	SK - equals - guid.toString()
	AND - AttributeOne - equals - true
	AND - AttributeTwo - equals - "approved"
	AND - AttributeThree - notEquals - true

----

       ConsistentRead = false
}

Example of a repository method with the issue:

public async Task<List<Item>> ExampleMethod(Guid pk, List<Guid> sks)
    {
        var listOfItems = new List<Item>();
        await Parallel.ForEachAsync(
            sks,
            new ParallelOptions { MaxDegreeOfParallelism = 32 },
            async (sk, _) =>
            {
                var query = createQuery(pk, sk)

                var results = await _context.query(query, gsi).ConfigureAwait(false);

                lock (listOfItems)
                {
                    listOfItems.AddRange(results);
                }
            }).ConfigureAwait(false);

        return listOfItems;
    }

It is also worth noting, when caling the repo method with a list of 50 sks to lookup - we only ever see 1-2 of the requests having the spike in response time. All others complete within milliseconds (as suggested my the XRay screenshot)

Please let me know if there is anything else I could share to help

[dscpinheiro]

Got it, thanks for the reply. Since you're doing the requests in parallel it does look like it's related to the revert we had to do in c43d041

We're going to re-introduce this behavior, this is being tracked in #4024 (that issue also has more details on the problems we were seeing, the performance degradation is obviously not ideal but our implementation had a bug where expired credentials were being returned).

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.