Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Cursor Based Pagination #430

Closed
rflihxyz opened this issue May 16, 2024 · 2 comments · Fixed by #479
Closed

feat: Cursor Based Pagination #430

rflihxyz opened this issue May 16, 2024 · 2 comments · Fixed by #479
Assignees
Labels

Comments

@rflihxyz
Copy link
Contributor

rflihxyz commented May 16, 2024

For now, our favourite approach is Cursor based pagination, as the underlying data changes frequently.
Other approaches like offset-based paginations would frequently lead to double entries.

Scope: All unified endpoints

Open to community suggestions :)

@rflihxyz rflihxyz changed the title Cursor Based Pagination featL: Cursor Based Pagination May 16, 2024
@rflihxyz rflihxyz changed the title featL: Cursor Based Pagination feat: Cursor Based Pagination May 16, 2024
@rajeshj11
Copy link

rajeshj11 commented May 16, 2024

@rflihxyz Below are the reasons why I picked keyset-based pagination(cursor-based).
please refer to this link:
https://docs.gitlab.com/ee/api/rest/index.html#pagination


Offset-based pagination. The default method and available on all endpoints except, in GitLab 16.5 and later, the users endpoint.
Keyset-based pagination. Added to selected endpoints but being [progressively rolled out](https://gitlab.com/groups/gitlab-org/-/epics/2039).
note:
**For large collections, you should use keyset pagination (when available) instead of offset pagination, for performance reasons.**

@mit-27
Copy link
Contributor

mit-27 commented May 16, 2024

I think @rflihxyz is talking about the pagination that is going to be implemented at services that handle the unified API endpoints. So, when we create the connection with any provider, it triggers the initial sync. The sync uses the third-party provider's pagination (It might be possible that each third party has a different way of handling pagination) to fetch a certain amount of data and store it in Panora's DB. Now, The services that handle the unified endpoint use cursor pagination and fetch data from Panora's DB.

Here, it is also possible that the user has a large amount of data in the connected provider. As a reason, a certain amount of data is stored in Panora's DB after initial sync. The remaining data can be fetched on-demand using the unified API endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Ready
Development

Successfully merging a pull request may close this issue.

3 participants