Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paginate large filter results #199

Closed
Abhiek187 opened this issue Apr 11, 2024 · 2 comments
Closed

Paginate large filter results #199

Abhiek187 opened this issue Apr 11, 2024 · 2 comments
Labels
enhancement New feature or request
Projects

Comments

@Abhiek187
Copy link
Owner

This app has grown to the point that there are now hundreds of recipes stored in MongoDB. To make queries more efficient and to encourage lazy-loading on the front-end, see https://www.mongodb.com/docs/atlas/atlas-search/paginate-results/ for how to paginate $search results.

@Abhiek187 Abhiek187 created this issue from a note in EZ Recipes (To do) Apr 11, 2024
@Abhiek187 Abhiek187 added the enhancement New feature or request label Apr 11, 2024
@Abhiek187
Copy link
Owner Author

Abhiek187 commented Apr 27, 2024

Here's how we can do this:

Server-side:

  • Find query
    • Initial: find(filter, { limit: 100 })
    • Subsequent: find({...filter, _id: {$gt: ObjectId("last _id")}}, { limit: 100 })
  • Aggregate query
    • Initial: $search --> $limit 100 --> $addFields searchSequenceToken
    • Subsequent: $search searchAfter lastBase64Token --> $limit 100 --> $addFields searchSequenceToken

Client-side:

  • Call initial when first searching
  • As the user scrolls down, call subsequent
  • Keep the entire array in memory

We want to avoid the skip operator since MongoDB has to traverse all the documents anyway. For find queries, we can take advantage of the default _id index and directly query documents that come after the last document in the previous query. MongoDB can compare ObjectIds. Similarly, for search queries, we can use the searchSequenceToken to directly jump to search results after the last token. Limits will also optimize the query by limiting the number of documents that should be returned before the query finishes.

The UI will need to save the last ObjectId/token and pass it to the BFF to paginate the results. We can probably add token as a query parameter to the filter API, which can accept either an ObjectId or a searchSequenceToken, depending on the query. Both will be treated as a string, but for the ObjectId, the BFF will just need to convert it to an ObjectId for the query.

@Abhiek187 Abhiek187 moved this from To do to In progress in EZ Recipes Apr 28, 2024
@Abhiek187
Copy link
Owner Author

Abhiek187 commented Apr 28, 2024

Just ran a benchmark: returning all documents took 1.7s with 4.3 MB of data. By limiting to 100 documents, it now takes 0.43s with 0.64 MB of data. That's 4x faster with 6.7x less data!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
EZ Recipes
In progress
Development

No branches or pull requests

1 participant