Web Words, an example Akka/Scala application ready to run on Heroku.
Created by Typesafe, http://typesafe.com/
See it live!
A live demo of the Web Words app may be running at
If it's down, try again later, or deploy the app yourself! (See below for instructions.)
Unlike the usual "Hello, World" this example shows off quite a few different features of the Akka and Heroku platforms. There's a fair amount of code here, but of course you can zoom in and look at the part you care about.
The "point" of the app (if you can call it that) is to do a shallow spider of a web site and compute some information about it, namely word counts and a list of links found on the site.
The app shows:
- how to use Akka to write a highly parallel application in functional
style, as an alternative to
- how to deploy it on Heroku including how to use addons and multiple processes
You can run the app using free price tiers from Heroku, MongoHQ, and RabbitMQ. You can also run it locally, of course.
A request step-by-step
If you follow an incoming request to the app, here's what the app shows you:
- an embedded Jetty HTTP server receives requests to spider sites
- requests are forwarded to Akka HTTP, which uses Jetty Continuations to keep requests from tying up threads
- the web process checks for previously-spidered info in a MongoDB capped collection which acts as a cache. This uses the Heroku MongoHQ addon.
- if the spider results are not cached, the web process sends a spider request to an indexer process using the RabbitMQ AMQP addon
- the app talks to RabbitMQ using Akka AMQP
- the indexer process receives a request from AMQP and shallow-spiders the site using an Akka actor that encapsulates AsyncHttpClient
- the indexer uses Akka, Scala parallel collections, and JSoup to grind through the downloaded HTML taking advantage of multiple CPU cores
- the indexer stores its output back in the MongoDB cache and sends an AMQP message back to the web process
- the web process loads the now-cached data from MongoDB
- the web process unsuspends the Jetty request and writes out the results
Build and deploy
The build for the app illustrates:
- SBT 0.11
- xsbt-start-script-plugin (useful for Heroku deployment)
- testing with ScalaTest
How to run it locally
sbt stageto stage the app (sbt must be sbt 0.10, not 0.7)
- Install and start up MongoDB
- Install and start up RabbitMQ Server
- Launch the app as specified in
Procfileif you have Heroku tools installed
foreman start --port 8080
- OR launch the app manually
indexer/target/startto run the indexer process
- in another terminal, run
web/target/startto run the web process
- Now open
http://localhost:8080in a browser
How to run it on Heroku
- Install the Heroku tools; be sure
herokuis on your path
- Type these commands inside the application's git clone:
heroku create --stack cedar
heroku addons:add mongohq
heroku addons:add rabbitmq
git push heroku master
heroku scale web=1 indexer=1
Learn more about the Scala and Akka stack at http://typesafe.com/ !