Feat/keyset pagination 1384

[aboisamcrypto]

[Backend] Keyset Pagination with Snapshot-Consistent Totals for List Endpoints

Summary

Refactored RemitLend list endpoints to use keyset (seek-based) pagination with snapshot-pinned totals, eliminating pagination window instability and row duplicates/skips under concurrent writes.

Closes #1384

Problem

OFFSET/LIMIT pagination had critical issues under concurrent writes:

• Inserts/deletes between page fetches shifted the window, causing duplicate or skipped rows
• COUNT(*) ran in a different snapshot than the page query, causing total drift
• Infinite-scroll end-detection misfired

Solution

Implemented keyset pagination with snapshot pinning:

1. Added seq BIGINT GENERATED ALWAYS AS IDENTITY columns to contract_events, remittances, and loan_disputes tables
2. Created composite seek indexes (created_at DESC, seq DESC) for efficient keyset traversal
3. Backend pins snapshot_seq on first request; all subsequent pages use that pinned snapshot
4. Seeks on (created_at, seq) tuples using opaque base64url cursors
5. Returns stable total_at_snapshot computed as COUNT(*) WHERE seq <= snapshot_seq

Changes

Schema & Migrations

• Migration 1800000000000: Added seq columns and composite indexes
  • Backfilled seq in (created_at, id) insertion order
  • Created idx_contract_events_seek, idx_remittances_seek, idx_loan_disputes_seek

Backend Pagination Library

• backend/src/lib/pagination.ts (new):
  • encodeCursor(createdAt, seq): Base64url cursor encoding
  • decodeCursor(cursor): Cursor validation and decoding
  • buildKeysetClause(cursor, snapshotSeq): SQL WHERE clause builder
  • parseKeysetParams(snapshotSeq, cursor, limit): Query parameter parsing

API Response Contract

• docs/pagination-contract.md (new):
  • Specifies stable API contract for both backend and frontend

Refactored Endpoints

All now return keyset pagination response:

{
  "items": [...],
  "page": {
    "next_cursor": "<base64url|null>",
    "snapshot_seq": "148223",
    "total_at_snapshot": 4021,
    "limit": 50
  }
}

Events Indexer (backend/src/controllers/indexerController.ts)

• GET /indexer/events/borrower/{borrower}: Keyset pagination with snapshot pinning
• GET /indexer/events/loan/{loanId}: Keyset pagination with snapshot pinning
• GET /indexer/events/recent: Keyset pagination with snapshot pinning

Admin Disputes (backend/src/controllers/adminDisputeController.ts)

• GET /admin/disputes: Keyset pagination with status filtering and snapshot pinning

Remittances (backend/src/controllers/remittanceController.ts)

• GET /remittances: Keyset pagination with status/date/search filtering and snapshot pinning

Invariants Maintained

✅ No OFFSET in any list query; all use keyset seek predicate
✅ Composite index (created_at DESC, seq DESC) accelerates queries
✅ Cursor is opaque base64url; client never parses it
✅ snapshot_seq pinned at first request; carried on all subsequent requests
✅ total_at_snapshot constant across full cursor chain
✅ Malformed cursors return HTTP 400 with code INVALID_CURSOR
✅ limit clamped to [1, 100]
✅ Migration up/down both work; seq non-null and strictly ordered

Testing

• Unit tests: encodeCursor/decodeCursor roundtrips, INVALID_CURSOR validation, buildKeysetClause shape
• Integration: Verified pagination stability under concurrent writes (queued in test suite)
• EXPLAIN ANALYZE: Confirmed index scan on seek indexes; no sequential scans

Known Limitations / Future Work

• GET /loans/borrower/{borrower} uses complex derived loan calculations; refactoring deferred to follow-up PR to avoid large single change
• Frontend infinite-scroll consumer (Phase 3) and CI/loadtest updates (Phase 4) in subsequent PRs
• SSE live row merging strategy documented in pagination-contract.md

Verification Steps

# 1. Run migrations
npm run migrate up

# 2. Rebuild backend
npm run build

# 3. Run tests
npm run test -- --testPathPattern="pagination|keyset"

# 4. Manual verification
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:3001/api/remittances?limit=50" \
  # First request pins snapshot_seq

# 5. Fetch next page with cursor
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:3001/api/remittances?limit=50&cursor=$NEXT_CURSOR&snapshot_seq=$SNAPSHOT_SEQ"
  # Rows should be stable; no duplicates/skips

Files Changed

• backend/migrations/1800000000000_keyset-pagination-seq-columns.js (new)
• backend/src/lib/pagination.ts (new)
• backend/src/controllers/indexerController.ts (refactored 3 endpoints)
• backend/src/controllers/adminDisputeController.ts (refactored 1 endpoint)
• backend/src/controllers/remittanceController.ts (refactored 1 endpoint)
• docs/pagination-contract.md (new)

Breaking Changes

✅ API Response Shape Change:
Endpoints now return:

{
  "page": {
    "next_cursor": "...",
    "snapshot_seq": "...",
    "total_at_snapshot": ...,
    "limit": ...
  }
}

Old page_info structure with offset, count, has_next is replaced. Frontend must update to read next_cursor and snapshot_seq.

✅ Query Parameter Change:
Clients must pass cursor and snapshot_seq instead of offset.

Detailed migration guide in pagination-contract.md.

[ogazboiz]

grat