An example case of building a distributed and resilient system.
- Persisted (on disk for now) write ahead logs
- Distributed crawling layer with resilient resurrection
- Work requestor service (in crawler) for pulling work when able
- Full integration tests for failure cases
- "Slow" website which delays responses often