-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
backendBackend relatedBackend relatedenhancementNew feature or requestNew feature or requestpriority:mediumMedium priorityMedium priority
Description
Problem
Every iOS request to /api/campaigns and /api/campaigns/search hits ScrapingBee directly, consuming 10 credits per call. The DB is only used as a fallback or for sort=hot.
Current Data Flow
iOS → GET /api/campaigns → ScrapingBee real-time fetch (10 credits/request)
iOS → GET /api/campaigns/search → ScrapingBee real-time fetch (10 credits/request)
Cron 02:00 UTC → ScrapingBee batch crawl → write to RDS
Target Architecture
ScrapingBee ──(cron only)──→ RDS (source of truth)
↓
iOS → GET /api/campaigns → query DB
iOS → GET /api/campaigns/search → query DB (ILIKE)
Changes
handler/campaigns.go
ListCampaigns — remove ScrapingBee call, read from DB only. Sort mapping:
| sort param | DB ORDER BY |
|---|---|
trending |
velocity_24h DESC, percent_funded DESC |
newest |
first_seen_at DESC |
ending |
deadline ASC (where deadline > now) |
hot |
velocity_24h DESC |
Pagination: cursor encoded as "offset:N", compatible with existing iOS String?.
SearchCampaigns — remove ScrapingBee call, replace with DB ILIKE:
WHERE (name ILIKE '%q%' OR blurb ILIKE '%q%')
AND state = 'live'
[AND category_id = ?]
ORDER BY percent_funded DESCBoth handlers no longer need *service.KickstarterScrapingService param.
cmd/api/main.go
Remove scrapingService param from ListCampaigns and SearchCampaigns handler registrations.
No changes
cron.go— continues nightly crawl via ScrapingBee → DBscrapingbee_client.go/kickstarter_scraping.go— unchanged, cron-onlyListCategories— already DB-first, unchanged
Trade-offs
- Search is limited to crawled campaigns (~15 categories × 10 pages). Campaigns not yet crawled will not appear in search results.
- ScrapingBee credit usage drops to cron-only: ~750 credits/day instead of per iOS request.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
backendBackend relatedBackend relatedenhancementNew feature or requestNew feature or requestpriority:mediumMedium priorityMedium priority