DarkDarkGo is a scalable search engine for the dark web. The goal of the project is to build a simple, robust search engine from scratch that can search through the dark web content.
It is a work in progress. The last time we tested (December 2017), the dark web did not give us enough meaningful results to serve a query. However, each component was working and tested separately. Hence, this project can be used as a foundation to add more interesting features and build a more complex product.
Components
- Crawler: Crawls the onion websites
- Index Builder: Builds index chunk
- Index Server: Stores replicated index chunk
- MGMT: Serves as single master, schedules jobs for other servers and executes heartbeats messages
- Front-End: Serves a website with Google-like interface
- Back-End: Express server with caching and aggregation logic
API
Each component has its own Dockerfile that is ready to be built and tested.