Paginate large filter results

[Abhiek187]

This app has grown to the point that there are now hundreds of recipes stored in MongoDB. To make queries more efficient and to encourage lazy-loading on the front-end, see https://www.mongodb.com/docs/atlas/atlas-search/paginate-results/ for how to paginate $search results.

[Abhiek187]

Here's how we can do this:

Server-side:

• Find query
  • Initial: find(filter, { limit: 100 })
  • Subsequent: find({...filter, _id: {$gt: ObjectId("last _id")}}, { limit: 100 })
• Aggregate query
  • Initial: $search --> $limit 100 --> $addFields searchSequenceToken
  • Subsequent: $search searchAfter lastBase64Token --> $limit 100 --> $addFields searchSequenceToken

Client-side:

• Call initial when first searching
• As the user scrolls down, call subsequent
• Keep the entire array in memory

We want to avoid the skip operator since MongoDB has to traverse all the documents anyway. For find queries, we can take advantage of the default _id index and directly query documents that come after the last document in the previous query. MongoDB can compare ObjectIds. Similarly, for search queries, we can use the searchSequenceToken to directly jump to search results after the last token. Limits will also optimize the query by limiting the number of documents that should be returned before the query finishes.

The UI will need to save the last ObjectId/token and pass it to the BFF to paginate the results. We can probably add token as a query parameter to the filter API, which can accept either an ObjectId or a searchSequenceToken, depending on the query. Both will be treated as a string, but for the ObjectId, the BFF will just need to convert it to an ObjectId for the query.

[Abhiek187]

Just ran a benchmark: returning all documents took 1.7s with 4.3 MB of data. By limiting to 100 documents, it now takes 0.43s with 0.64 MB of data. That's 4x faster with 6.7x less data!