Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Browse files

remove request step-by-step from README again

  • Loading branch information...
commit 3753986b9c091f1d08865aa20b18d6c7bd670ab3 1 parent bf04ff3
@havocp havocp authored
Showing with 0 additions and 34 deletions.
  1. +0 −34
@@ -54,40 +54,6 @@ The major technologies in the app, in brief:
[Binary JSON aka BSON][bson] documents. On Heroku, one way to
use it is with the [MongoHQ][mongohq] addon.
-## A request step-by-step
-If you follow an incoming request to Web Words, here's what the app
-shows you:
- - an **embedded Jetty** HTTP server receives requests to spider sites
- -
- - requests are forwarded to **Akka HTTP**, which uses Jetty Continuations
- to keep requests from tying up threads
- -
- - the web process checks for previously-spidered info in a
- **MongoDB capped collection** which acts as a cache.
- This uses the **Heroku MongoHQ addon**.
- -
- -
- - if the spider results are not cached, the web process
- sends a spider request to an indexer process using
- the **RabbitMQ AMQP addon**
- -
- -
- - the app talks to RabbitMQ using **Akka AMQP**
- -
- - the indexer process receives a request from AMQP and shallow-spiders
- the site using an Akka actor that encapsulates **AsyncHttpClient**
- -
- - the indexer uses Akka, **Scala parallel collections**, and **JSoup** to
- grind through the downloaded HTML taking advantage of multiple CPU cores
- -
- -
- - the indexer stores its output back in the MongoDB cache and sends
- an AMQP message back to the web process
- - the web process loads the now-cached data from MongoDB
- - the web process unsuspends the Jetty request and writes out the results
## Setup
If you want to understand this app and/or try running it, here's what you'll
Please sign in to comment.
Something went wrong with that request. Please try again.