-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Cursor Based Pagination #430
Comments
@rflihxyz Below are the reasons why I picked keyset-based pagination(cursor-based).
|
I think @rflihxyz is talking about the pagination that is going to be implemented at services that handle the unified API endpoints. So, when we create the connection with any provider, it triggers the initial sync. The sync uses the third-party provider's pagination (It might be possible that each third party has a different way of handling pagination) to fetch a certain amount of data and store it in Panora's DB. Now, The services that handle the unified endpoint use cursor pagination and fetch data from Panora's DB. Here, it is also possible that the user has a large amount of data in the connected provider. As a reason, a certain amount of data is stored in Panora's DB after initial sync. The remaining data can be fetched on-demand using the unified API endpoint. |
For now, our favourite approach is Cursor based pagination, as the underlying data changes frequently.
Other approaches like offset-based paginations would frequently lead to double entries.
Scope: All unified endpoints
Open to community suggestions :)
The text was updated successfully, but these errors were encountered: