Permalink
Browse files

Addind widow-core project

  • Loading branch information...
ScottMansfield committed Jun 24, 2014
1 parent 02d7f2a commit 83ac311c1f66fd3f38fb98490fd0fededcaae81a
Showing with 3 additions and 1 deletion.
  1. +3 −1 README.md
  2. 0 widow-core/.gitkeep
@@ -1,10 +1,12 @@
Widow - the extensible crawler for your website
==========

Widow is meant to be a crawler to index only the domains you specify. Instead of crawling the entire world, Widow will crawl your website to create your own search metadata. From this, you can see the average page load time, asset size, etc.

Widow has several parts:
* The Core, which contains machinery to pull messages and process them in a multi-threaded environment
* The Fetcher, which pulls pages down from the internet
* The Parser, which parses pages in an extensible way
* The Indexer, which pushes metadata into a search index
* The analyzer, which gives interesting data about the pages fetched
* The Analyzer, which gives interesting data about the pages fetched

No changes.

0 comments on commit 83ac311

Please sign in to comment.