Home

Welcome to the chive-show wiki!

Here I'm leaving some notes to my future self... and maybe others to understand what the heck is going on here. It'll probably not be very useful, but some breadcrumbs never hurt anyone.

Legacy Miner

stateDiagram-v2
direction LR
[*] --> /cron/parse: cron\n6 hrs
/cron/parse --> /cron/batch: via task-queue
/cron/batch --> [*]: Store data

Loading

• /cron/parse - parse RSS feed into batches
• /cron/batch - process batch

New Endpoints

New endpoints might take longer to run, but overall they are designed for incremental updates. Overall, the design could use some domain modeling to assist with the state of the world.

1. Rebuild Everything /cron/rebuild
   Rebuild entire storage from scratch (expensive... but useful for setting up the project)
2. Incremental Crawl /cron/crawl
   Fetch first page of RSS feed and mine metadata (an incremental update)
3. Mine Metadata /cron/mine
   Helper endpoint to mine the contents of a single post
4. Migrate data /cron/migrate Used to update data in datastore when mining algorithms change/improve

Updating GO versions

See https://github.com/bign8/chive-show/pull/57 for all the fun places that need to be updated when updating go versions. Would be nice to have this happen auto-magically every 6 months when a new "minor" version of go is released.