Architecture

Architecture Overview

Architecture should consist of 2 moving parts: the web layer, which serves pages and accepts connections from WS clients and broadcasts articles, and the crawler, which is the group of libraries for crawling sources.

The crawler adds to the database, and is a separate program altogether. That's so we follow the worker model, and we stay capable of scaling the crawlers individually.