Although I ran out of time, I was able to figure out the streaming endpoint would have been substantially faster than Claude's solution. It was a mistake to leave the initial API discovery to Claude, and investigate myself after Claude had started building a solution. Claude, having restrictions on unpublished/undocumented API's, would not use the /events/d4ta/x7k9/feed endpoint, and to write a solution from scratch would be tight, given I am editing this README about 16 minutes from the 3hr key expiration. Time permitting, I would get the stream-access token on any failed request, fetch events(feed) in batches of 5000 (discovered limit), and remove the logic for cooldown. I did NOT see a cooldown associated with the stream/feed API. Each batch of 5000 took approximately 4.09s, which in series would execute under ideal circumstances in 2454s/41m. (3000000/5000*4.09)
Given the instructions said under 30m was possible, I assume we can use concurrency - perhaps even with separate tokens? My submission only has 5000 ids purely to show the endpoint was reached manually. Claude's solution should work as expected, albeit slowly, via the regular events API, taking ~3 hours to execute.
sh run-ingestion.shThis starts PostgreSQL and the ingestion worker via Docker Compose. The script monitors progress and exits when all 3,000,000 events are ingested.
┌──────────────────────────────────────────────────────┐
│ Ingestion Worker │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────┐ │
│ │ Segment 0 │ │ Segment 1 │ │ Seg 2-9 │ │
│ │ ts: newest │ │ ts: mid-high │ │ ... │ │
│ │ cursor ──► │ │ cursor ──► │ │ ──► │ │
│ └──────┬──────┘ └──────┬───────┘ └────┬─────┘ │
│ │ │ │ │
│ └────────┬────────┴────────────────┘ │
│ ▼ │
│ ┌────────────────┐ │
│ │ Rate Limiter │ 10 req/min window │
│ │ (10 parallel) │ fire all, wait, repeat │
│ └───────┬────────┘ │
│ ▼ │
│ ┌────────────────┐ │
│ │ Batch Insert │ 500-row chunks │
│ │ ON CONFLICT │ dedup at boundary │
│ └───────┬────────┘ │
│ ▼ │
│ ┌────────────────┐ │
│ │ Progress DB │ cursor per segment │
│ └────────────────┘ │
└──────────────────────┬───────────────────────────────┘
▼
┌────────────────┐
│ PostgreSQL │
│ ingested_events│
│ (3,000,000) │
└────────────────┘
Core idea: Split the event timeline into 10 segments with synthetic cursors, then fire 10 parallel API requests per rate-limit window (one per segment), achieving the maximum possible throughput of ~50,000 events/minute.
| File | Purpose |
|---|---|
packages/ingestion/src/index.ts |
Main orchestrator — segments, rate-limit windows, progress loop |
packages/ingestion/src/api.ts |
HTTP client, synthetic cursor creation, error types |
packages/ingestion/src/db.ts |
PostgreSQL schema, batch inserts, progress tracking |
packages/ingestion/Dockerfile |
Multi-stage build (compile TS → slim runtime image) |
docker-compose.yml |
PostgreSQL + ingestion service |
- Each segment's cursor position is persisted to an
ingestion_progresstable after every page - On restart, completed segments are skipped and in-progress segments resume from their last cursor
ON CONFLICT (id) DO NOTHINGhandles deduplication at segment boundaries- Docker
restart: on-failureauto-restarts the worker after transient failures
Below is a chronological log of every discovery made while exploring the API.
GET /api/v1
→ 200: { endpoints: { events, sessions, metrics }, auth: { header: "X-API-Key" } }
Three endpoints confirmed: /events, /sessions, /metrics.
GET /api/v1/events
→ 200, 100 events returned
Headers:
X-RateLimit-Limit: 10
X-RateLimit-Remaining: 9
X-RateLimit-Reset: 60
X-Cache: MISS
X-Cache-TTL: 30
Body:
meta.total: 3,000,000
pagination: { limit: 100, hasMore: true, nextCursor: "eyJ...", cursorExpiresIn: 117 }
Findings:
- Default page size is 100
- Rate limit is 10 requests per 60 seconds (header-based auth)
- Cursor-based pagination; cursors expire in ~120s
- Response includes
X-Cacheheaders (caching layer present) - Timestamps are mixed format: some epoch milliseconds (
1769541612369), some ISO strings ("2026-01-27T19:19:13.629Z")
GET /api/v1/events?limit=1000 → 1,000 returned ✓
GET /api/v1/events?limit=5000 → 5,000 returned ✓ (Content-Length: 1,732,547)
GET /api/v1/events?limit=10000 → 5,000 returned (capped at 5,000)
Finding: Maximum page size is 5,000. At 10 req/min this gives 50,000 events/min baseline.
GET /api/v1/sessions?limit=1
→ X-RateLimit-Limit: 40 (separate bucket!)
→ meta.total: 60,000 sessions
→ Each session has ~50 events (3M / 60K)
Finding: Sessions have a separate rate limit (40/min) but don't embed event data. Individual session detail (GET /sessions/:id) returns metadata + eventCount but no event IDs.
GET /api/v1/metrics
→ X-RateLimit-Limit: 30
→ data: [] (empty), total: 0
No useful data here.
Base64 decode of nextCursor:
→ {"id":"af5c33c8-...","ts":1769540656330,"v":2,"exp":1772565391775}
Cursor fields:
id— last event UUIDts— last event timestamp (epoch ms)v— version (always 2)exp— cursor expiration (epoch ms)
Created a cursor with an arbitrary timestamp and far-future expiration:
{"id":"00000000-0000-0000-0000-000000000000","ts":1769200000000,"v":2,"exp":1872565391775}
→ Base64 encode → use as ?cursor= parameter
→ 200 OK! Returns events starting from that timestamp.
This enables starting pagination at any point in the timeline, unlocking parallel fetching.
Binary-searched the event timestamps:
Cursor at ts=1766900000000 → hasMore: false (below oldest event)
Cursor at ts=1766950000000 → hasMore: true (events exist)
Cursor at ts=1769542000000 → first event: ts=1769541612369 (newest)
Event range: ~1766950000000 to ~1769542000000 (epoch ms, spanning ~30 days)
Fired 12 rapid requests:
Requests 1-10: 200 OK
Requests 11-12: 429 Too Many Requests
Retry-After: 48
X-RateLimit-Remaining: 0
Confirmed: Strictly enforced, 10 per 60-second sliding window. Retry-After header gives exact wait time.
GET /api/v1/events?api_key=... (query param)
→ X-RateLimit-Limit: 5 (worse!)
→ Shares the same rate limit bucket as header auth
Finding: Header auth gives 10/min; query param gives 5/min. They share a bucket. The .env.example hint was correct: "Use header-based auth for best rate limits."
Request 1: X-Cache: MISS, X-RateLimit-Remaining: 9
Request 2: X-Cache: HIT, X-RateLimit-Remaining: 8
Finding: Cache hits still consume rate limit tokens. No shortcut here.
After exhausting 10/10 events rate limit:
GET /api/v1/sessions → 200 OK, X-RateLimit-Remaining: 39
Confirmed: Each endpoint has its own independent rate limit bucket.
Step 13 — Hidden endpoints (dashboard JS analysis)
Extracted API calls from the minified dashboard JavaScript (/assets/index-1KlV7jRD.js):
/api/v1/events — list events
/api/v1/events/${id} — single event detail
/api/v1/events/bulk — POST with { ids: [...] }, rate limit: 20/min
/internal/dashboard/stream-access — POST, returns streaming token
/internal/stats — GET, returns aggregate counts
/internal/health — GET, returns health status
POST /api/v1/events/bulk { ids: ["..."] }
→ X-RateLimit-Limit: 20 (separate bucket)
→ Requires knowing event IDs upfront
Useful for re-fetching specific events but not for initial discovery.
GET /internal/stats
→ 200: { counts: { users: 3000, sessions: 60000, events: 3000000, metrics: 0 } }
→ Includes event type distribution and device type breakdown
POST /internal/dashboard/stream-access
→ 403: "This endpoint requires dashboard access"
The dashboard JS shows this returns a streamAccess object with { endpoint, tokenHeader, token, expiresIn } for high-throughput streaming. This endpoint supports cursor, since, and until query params. However, it requires a dashboard-tier API key that my key didn't have.
| Parameter | Value |
|---|---|
| Rate limit | 10 requests / 60 seconds |
| Max page size | 5,000 events |
| Events per window | 10 × 5,000 = 50,000 |
| Total events | 3,000,000 |
| Windows needed | 3,000,000 / 50,000 = 60 |
| Time per window | ~61 seconds |
| Estimated total | ~61 minutes |
- Stream endpoint access — The
/internal/dashboard/stream-accessendpoint likely enables much higher throughput. With more time I'd investigate how to obtain dashboard-level access. - Multi-endpoint parallelism — Sessions (40/min) and bulk (20/min) have separate rate limits. A pipeline that discovers event IDs through one path and fetches full data through another could achieve higher aggregate throughput.
- Adaptive segment sizing — Instead of equal time-range segments, analyze event density and allocate more segments to dense time periods.
- COPY protocol — PostgreSQL's
COPY FROMis faster than multi-row INSERT for bulk loading. Would reduce DB insert time. - Compression — The API doesn't appear to support gzip responses, but monitoring for this could reduce network transfer time.
- Unit & integration tests — Mock the API layer and test segment management, cursor expiry recovery, and progress resumption.
- Try deflate vs GZIP to test any performance improvement.