Query on fields without index #1174
Replies: 1 comment
-
@Marcvander We aren't solving this case. It's up to you to handle that edge case. Dynamoose will return the items we get back from DynamoDB. We also return the Now if you don't want to manually write code to make more requests, we also have an But overall, it's all constrained by DynamoDB. We aren't doing anything special behind the scenes to bypass any limits or anything. To specifically address your questions:
We aren't. We only return the data to you that DynamoDB returns to us. So in this case you'll likely only receive a few items, along with a
Only if you are using the Dynamoose |
Beta Was this translation helpful? Give feedback.
-
Coming from MongoDB and MySQL, we can run queries using fields that are not indexed. This gives a lot of flexibility on your queries, but it also allows you to run poor performance queries. DynamoDB on the other hand lets you only run queries on indexed fields. This makes sure that the performance will always be very good with a 1GB or 100TB data set.
There are two concepts in DynamoDB to run queries:
-
KeyConditionExpression
: using indexed fields with a haskKey and rangeKey-
FilterExpression
: using any fieldsBut the problem is explained here: "However, the key point to understand is that the Query and Scan operations will return a maximum of 1MB of data, and this limit is applied in step 1, before the filter expression is applied."(https://www.alexdebrie.com/posts/dynamodb-filter-expressions/). This means that using the
FilterExpression
on a dataset larger than 1MB won't give you the desired result, since the filter will only be applied on the first 1MB of data.In Dynamoose documentation, on the Query page (https://dynamoosejs.com/guide/Query), it is written: "Model.query() combines both the KeyConditionExpression and the FilterExpression from DynamoDB. If you query for an attribute that you defined as your hashKey or rangeKey DynamoDB will use KeyConditionExpression. This could be the most performant and cost efficient way to query for. If querying for attributes that are not defined as your hashKey or rangeKey DynamoDB might select more documents at first and then filter the result which could have a bad impact on performance and costs.".
Let's say I query using a field that has no index (neither a hashKey nor a rangeKey), the documentation tells me that the
FilterExpression
will be used. I'd like to understand how are you solving the 1MB limit problem in this case? Are you running a recursion of queries until no more result are found?Beta Was this translation helpful? Give feedback.
All reactions