web scale fetching in clojure
Switch branches/tags
Nothing to show
Pull request Compare This branch is 115 commits ahead of neotyk:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src/fetcher
test/fetcher
.gitignore
README.me
project.clj

README.me

##Fetcher

A production scale HTTP fetcher.

A large scale fetcher should be limited by the speed of fetching results and streaming them to disk.  We don't want other communication or processing happening in the fetcher.  Rather, other services listen on the fetcher's results and pick up work by polling disk or listening on some channel.