Skip to content

perf: Polling worker fetches all pending events in one unbounded request — add pagination #337

Description

@Xhristin3

Problem Statement
worker.ts calls GET /streams/pending and expects an array of all pending events. With no pagination, a backlog of thousands of events is returned in a single HTTP response, consuming significant memory.

Why It Matters
An unbounded poll response is a memory bomb during catch-up after downtime. It also makes the worker's memory footprint unpredictable.

Acceptance Criteria

  • GET /streams/pending supports limit and cursor/offset query parameters
  • Worker fetches events in batches (configurable via POLL_BATCH_SIZE env var, default 100)
  • Worker continues fetching until the batch is smaller than POLL_BATCH_SIZE
  • POLL_BATCH_SIZE documented in xstreamroll-processing/.env.example

Files Affected: xstreamroll-processing/src/worker.ts, api/src/streams/streams.controller.ts

Difficulty: Medium | Effort: 1–2 days

Metadata

Metadata

Labels

GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardOfficial Campaign | FWC26Campaign: Official Campaign | FWC26performancePerformance improvementsprocessingRelated to xstreamroll-processing/ workerscalabilityHorizontal scaling, high-throughput scenarios

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions