Skip to content

refactor(backend): DB as source of truth — all iOS reads from DB #17

@Jing-yilin

Description

@Jing-yilin

Problem

Every iOS request to /api/campaigns and /api/campaigns/search hits ScrapingBee directly, consuming 10 credits per call. The DB is only used as a fallback or for sort=hot.

Current Data Flow

iOS → GET /api/campaigns         → ScrapingBee real-time fetch (10 credits/request)
iOS → GET /api/campaigns/search  → ScrapingBee real-time fetch (10 credits/request)
Cron 02:00 UTC                   → ScrapingBee batch crawl → write to RDS

Target Architecture

ScrapingBee ──(cron only)──→ RDS (source of truth)
                                     ↓
iOS → GET /api/campaigns         → query DB
iOS → GET /api/campaigns/search  → query DB (ILIKE)

Changes

handler/campaigns.go

ListCampaigns — remove ScrapingBee call, read from DB only. Sort mapping:

sort param DB ORDER BY
trending velocity_24h DESC, percent_funded DESC
newest first_seen_at DESC
ending deadline ASC (where deadline > now)
hot velocity_24h DESC

Pagination: cursor encoded as "offset:N", compatible with existing iOS String?.

SearchCampaigns — remove ScrapingBee call, replace with DB ILIKE:

WHERE (name ILIKE '%q%' OR blurb ILIKE '%q%')
  AND state = 'live'
  [AND category_id = ?]
ORDER BY percent_funded DESC

Both handlers no longer need *service.KickstarterScrapingService param.

cmd/api/main.go

Remove scrapingService param from ListCampaigns and SearchCampaigns handler registrations.

No changes

  • cron.go — continues nightly crawl via ScrapingBee → DB
  • scrapingbee_client.go / kickstarter_scraping.go — unchanged, cron-only
  • ListCategories — already DB-first, unchanged

Trade-offs

  • Search is limited to crawled campaigns (~15 categories × 10 pages). Campaigns not yet crawled will not appear in search results.
  • ScrapingBee credit usage drops to cron-only: ~750 credits/day instead of per iOS request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions