Keyset pagination

[aeriksson]

Is your feature request related to a problem? Please describe.
Offset-based pagination is problematic when dealing with indexes that change over time, because offsets held client-side are invalidated when documents are inserted or removed server-side. The client has no way of dealing with this, and as a result, documents end up being skipped over or appearing multiple times when paginating.

This is especially jarring when dealing with "scroll to load more"-style activity feeds (as is common in social media apps).

Describe the solution you'd like
The standard solution dealing with this problem is to use "keyset" or "seek" pagination. With this approach, the client holds a reference to the last element to appear in the latest query (this can be either a direct reference, such as a document id, or an indirect reference, such as a timestamp). This reference is then included in queries (via something like after=my-document-id or after=2020-07-04T12:07:44.480Z) in lieu of the offset parameter.

Since MeiliSearch already supports associating a primary key with each document, the natural approach to implementing keyset pagination would be to support keyset pagination on the primary key. I'd propose:

• Adding a query param called after (or similar) to the search endpoint, that takes a primary key.
• Accepting after in place of offset, but rejecting requests that include both.
• If a document with the primary key specified in after exists, returning documents starting from the document after that one.
• If no document with the given primary key exists, returning either an empty result or an error.

I'm not familiar with MeiliSearch's data model, so I'm not sure what the exact implementation would look like. If there's a way to seek to a document with a given primary key and iterate from there, however, then that could be facilitated to speed up 'deep' pagination (because unlike with offset-based pagination, there's no need to scan all documents 'before' the target document).