Description
The pipeline currently generates offset-based pagination. Add cursor-based pagination as an option, which is better suited for real-time data, infinite scroll, and large datasets.
Why
Offset pagination has well-known issues with large datasets (performance degrades as offset increases) and consistency issues (items shift between pages as new data is inserted). Cursor-based pagination solves both. The `pipeline.config.json` already has `cursorPaginationAvailable: true` but no implementation exists.
Current State
`pipeline.config.json`:
```json
"pagination": {
"defaultLimit": 20,
"maxLimit": 100,
"strategy": "offset",
"cursorPaginationAvailable": true
}
```
Acceptance Criteria
Description
The pipeline currently generates offset-based pagination. Add cursor-based pagination as an option, which is better suited for real-time data, infinite scroll, and large datasets.
Why
Offset pagination has well-known issues with large datasets (performance degrades as offset increases) and consistency issues (items shift between pages as new data is inserted). Cursor-based pagination solves both. The `pipeline.config.json` already has `cursorPaginationAvailable: true` but no implementation exists.
Current State
`pipeline.config.json`:
```json
"pagination": {
"defaultLimit": 20,
"maxLimit": 100,
"strategy": "offset",
"cursorPaginationAvailable": true
}
```
Acceptance Criteria
```json
{
"data": [...],
"meta": {
"hasNextPage": true,
"hasPreviousPage": false,
"startCursor": "abc123",
"endCursor": "xyz789"
}
}
```