An example Akka/Scala application ready to run on Heroku.
Created by Typesafe, http://typesafe.com/
Unlike the usual "Hello, World" this example shows off quite a few different features of the Akka and Heroku platforms. There's a fair amount of code here, but of course you can zoom in and look at the part you care about.
The "point" of the app (if you can call it that) is to do a shallow spider of a web site and compute some information about it, namely word counts and a list of links found on the site.
The app shows:
- how to use Akka to write a highly parallel application in functional
style, as an alternative to
- how to deploy it on Heroku including how to use addons and multiple processes
You can run the app using free price tiers from Heroku, MongoHQ, and RabbitMQ. You can also run it locally, of course.
A request step-by-step
If you follow an incoming request to the app, here's what the app shows you:
- an embedded Jetty HTTP server receives requests to spider sites
- requests are forwarded to Akka HTTP, which uses Jetty Continuations to keep requests from tying up threads
- the web process checks for previously-spidered info in a MongoDB capped collection which acts as a cache. This uses the Heroku MongoHQ addon.
- if the spider results are not cached, the web process sends a spider request to an indexer process using the RabbitMQ AMQP addon
- the app talks to RabbitMQ using Akka AMQP
- the indexer process receives a request from AMQP and shallow-spiders the site using an Akka actor that encapsulates AsyncHttpClient
- the indexer uses Akka, Scala parallel collections, and JSoup to grind through the downloaded HTML taking advantage of multiple CPU cores
- the indexer stores its output back in the MongoDB cache and sends an AMQP message back to the web process
- the web process loads the now-cached data from MongoDB
- the web process unsuspends the Jetty request and writes out the results
Build and deploy
The build for the app illustrates:
- SBT 0.10
- xsbt-start-script-plugin (useful for Heroku deployment)
- testing with ScalaTest
How to run it locally
How to run it on Heroku
Learn more about the Scala and Akka stack at http://typesafe.com/ !