Dynamodb QueryAsync to List performance issues

[Crypth]

After upgrading to first 2.0.2.2 and 2.0.2.3 the performance of enumerating a query to a list with 25k items went from roughly 4 seconds to almost 19 seconds. The query itself takes less than a second which is what you'd expect and just fine.

We reverted back to 1.5.37.0 and the issue is gone. I can't really explain why, but perhaps the single core small and medium instances aren't fond of context/thread swapping. The CPU spikes to 100% after retrieving the json data from dynamodb, rendering the instance virtually unusable for the duration.

I boiled down the issue to a simple test by just a single query in a vs unit test which confirmed where the issue lies. Basically it looks like:

var asyncSearch = Context.QueryAsync<MyItemType>(HashKey);
var searchResult = (await asyncSearch.GetRemainingAsync()).ToList();

Compared to

var result = await Task.FromResult(Context.Query<MyItemType>(HashKey));

Did we misunderstand the usage?

It's worth mentioning that some of the data is binary, in effect: it's being base64decoded but it's rarely above 64 bytes. The rest is numbers, with a hash key and a range key.

[normj]

Thanks for letting us know about your performance issue. I'm doing some investigation into this and let you know what I find.

[normj]

Hi Crypth

I wanted to give you an update on this performance issue. I'm still investigating the performance differences between V1 and V2 and have made good progress. I want to do a bit more performance tuning and testing on my fixes before releasing the changes.

[Crypth]

That is splendid news 👍 , I'm just sorry I don't have the time to help out. Out of curiosity though, where was the bottleneck? Or perhaps have the release notes disclose it instead of wasting time on comments here is a better idea.

[normj]

We released version 2.0.4 of the SDK recently that contains performance improvements. I'd appreciate it if you could download it and see what improvements you see.

There wasn't any one thing causing the performance bottleneck. We switched to a new Json parser for V2 and we needed to rework how we were using it.

[Crypth]

Cheers Normj, I've checked the performance, not quite as thoroughly as i have time for at the moment but it seems it's quite an improvement. A quick test with 30k items or so was 12 seconds with the 2.0.4.1 SDK while my slightly modified 1.5.37.0 ran at 7.5 seconds. This makes sense though as I've implemented gzip in my version which shaves off quite a bit of time from the response. In fact the rework in 2.0.4.1 might even be faster. As far as I'm concerned this issue is resolved, many thanks. I'll let you know more when I've stuck gzip support into the 2.0.4.1 if you'd like.

I'm not sharing that code unless someone asks for it because of the implementation it needs to check the checksum for both data streams, thus making it slightly unsafe. Unless we can somehow learn the size of when the dynamodb server starts gzipping the response, i can't think of any better way though.