Swarm-Crawler

This little project is created to teach myself something about swarm applications. I created a master app which has the full control and manage everything. Also a slave app is created which contains the worker units. The master says which site is to crawl and one slave will do it. If he found a new link to another site, he informs the master and the new site will be crawed also. The benefit of having multiple worker units is that you have more power (more cores) and your network card will not limit you.