DynamoDB Scan Request Doesn't Work As Expected

[akefirad]

Describe the bug
I've got a simple Scan query (copied from the request body view of the plugin):

{
  "FilterExpression": "fooId = :val_1",
  "ExpressionAttributeValues": {
    ":val_1": {
      "S": "some value"
    }
  },
  "TableName": "my-table",
  "ReturnConsumedCapacity": "TOTAL"
}

where fooId is not a key (hence it's a scan call). This request returns no items, but when I run it on AWS web console (not the actual JSON, but using the provided form) it correctly returns three items.

Steps to reproduce
Not entirely sure what is the root cause here. Sometimes scan calls return some result sometimes not.

Expected behavior
It should return the correct items.

Environment information:
IntelliJ IDEA 2023.2 (Ultimate Edition)
Build #IU-232.8660.185, built on July 26, 2023
Runtime version: 17.0.7+7-b1000.6 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 13.5
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 12
Metal Rendering is ON
Non-Bundled Plugins:
org.dynamodb4idea (2023.2.1-232)
Kotlin: 232-1.9.0-IJ8660.185

[AntonShuvaev]

Hi,

What you're experiencing is the expected behavior of a DynamoDB Scan request. The Scan operation in DynamoDB reads items up to the limit of 1 MB before applying the filter expression, which means the result can be empty even if the table has items that match the filter expression. If the data in the table is constantly changing, the result can differ each time you run the same query.

In the plugin, you can use pagination to iterate through the data and find the items you're looking for. This is achieved by utilizing the LastEvaluatedKey in the ExclusiveStartKey of your Scan request. However, depending on your specific needs and use case, it might be more efficient to revise your request to use a Query instead of a Scan.

[akefirad]

I see, so you mean after the first result (which might have zero items), I need to use the "next page" icon to continue the pagination, right?
On that note, does it make sense to have auto-pagination feature? Like continue the pagination until the given number of items are retrieved? (My impression is that AWS Web Console does something like that, but I'm not sure.)