feat: Search pipeline run API uses pagination cursor

[yuechao-qin]

TL;DR

Replaced offset-based pagination with cursor-based pagination for searching pipeline runs.

Old format (offset-based, base64-encoded)

The page_token was a base64-encoded JSON blob containing offset, filter, and filter_query:

base64({"offset": 10, "filter": null, "filter_query": null})
  = eyJvZmZzZXQiOiAxMCwgImZpbH=

API request for page 2:

GET /api/pipeline_runs?page_token=eyJvZmZzZXQiOiAxMCwgImZpbH=

New format (cursor-based, plain text)

The page_token is now a plain-text created_at~id cursor pointing to the last row of the previous page:

2024-02-01T09:00:00+00:00~018d8fff

API request for page 2:

GET /api/pipeline_runs?page_token=2024-02-01T09:00:00+00:00~018d8fff

What changed?

Functional

• Pagination mechanism: Switched from offset/limit to cursor-based pagination using created_at and id fields

Other

• Database optimization: Added composite index ix_pr_created_at_desc_id_desc on (created_at DESC, id DESC) for efficient cursor queries
• Query structure: Modified SQL queries to use tuple comparison (created_at, id) < (cursor_created_at, cursor_id) instead of OFFSET
• API compatibility: Maintained existing page_token parameter while changing internal implementation

How to test?

uv run pytest tests/test_api_server_sql.py tests/test_filter_query_sql.py

• Verify pagination works correctly with multiple pages of pipeline runs
• Test that cursor pagination maintains stable ordering even when new records are inserted between page requests
• Confirm no duplicate records appear across paginated results
• Validate that invalid page tokens raise appropriate errors
• Test pagination combined with filter queries

Why make this change?

Cursor-based pagination provides several advantages over offset-based pagination:

• Performance: Eliminates expensive OFFSET (reads all rows that are skipped) operations on large datasets
• Consistency: Prevents duplicate or missing records when data is inserted/deleted during pagination
• Scalability: Query performance remains constant regardless of page depth
• Stability: Results remain consistent even with concurrent modifications to the dataset

[yuechao-qin]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: Add include_sql to Search Pipeline Run API #135
• feat: Search pipeline run API uses pagination cursor #131 👈 (View in Graphite)
• feat: Search within date range in pipeline run API #130
• feat: Search pipeline name in pipeline run API #129
• refactor: Convert legacy filter to be filter_query #128
• feat: Search created by user in pipeline run API #108
• feat: Search annotations in pipeline run API #107
• feat: Create Pydantic models for Search Pipeline Run API #106
• refactor: Search pipeline run refactoring and baseline unit tests #105
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.