Count limits

[Tinche]

This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

[Tinche]

The mypy failure is at src/aiodynamo/http/httpx.py:45: error: Argument "data" to "post" of "AsyncClient" has incompatible type "bytes"; expected "Dict[Any, Any]" which I haven't touched. Should probably be fixed separately.

[dimaqq]

The mypy failure is at src/aiodynamo/http/httpx.py:45: error: Argument "data" to "post" of "AsyncClient" has incompatible type "bytes"; expected "Dict[Any, Any]" which I haven't touched. Should probably be fixed separately.

Will be done in #87

[dimaqq]

Code-wise LGTM from me.

@ojii if the new functionality is reasonable.

[dimaqq]

I've merged #87, if you rebase the tests should be all green ❇️ :)

[ojii]

what's the reason for repeating this 20 times?

[Tinche]

I wanted to call it more than a few times to ensure the limit parameter was being correctly applied. The number 20 is arbitrary.

[ojii]

I like that idea, but I think the loop adds little value. Instead, how about testing these scenarios:

• 1 item table, limit 1 (tests a full count despite limit)
• 2 item table, limit 1 (tests a single page partial count)
• table above page size, limit larger than page size but not all items (tests a multi page partial count)

There's lots of other potential scenarios, but I feel like these are the most important. The main point I'm trying to suggest is to test specific scenarios, rather than the same one 20 times.

[Tinche]

Sounds good, but what is the page size? I know from experience on our production DynamoDB instances some tables would return around 10k items using a single query. I think it depends on the sizes of the items?

[Tinche]

Hm I see

aiodynamo/tests/integration/test_client.py

Line 259 in 29c7ae7

async def test_query_with_limit(client: Client, high_throughput_table: TableName):

already dealing with pagination, maybe I can use this approach for the test.

[Tinche]

@ojii Test reworked!

[ojii]

This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

Would a count_single_page be useful for your use case?

[Tinche]

This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

Would a count_single_page be useful for your use case?

Yes, had this been available I would have used it. On a high level though, what I really want is to count with a limit so that's what I implemented.

[dimaqq]

maybe I'll slop limit onto the docs after merge :clever:

[dimaqq]

Documentation needs to be updated :)