distributed twitter search engine
JavaScript Scala
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
build renamed sbt Feb 20, 2011
config cleanup and fixed an indexing issue May 3, 2011
lib moved configgy to lib Jul 27, 2011
project moved configgy to lib Jul 27, 2011
src cleanup and fixed an indexing issue May 3, 2011
README.md added more stuff to readme Feb 22, 2011
sbt renamed sbt Feb 20, 2011


What is Chirper?


Chirper is a real time tweet search engine (written in Scala) using open source technology built by the LinkedIn SNA team:

Although the amount of code is minimal, the system can scale horizontally infinitely by leveraging the distributed systems above. See this blog post on design considerations.

Build and run the system:

From the top level directory, e.g. ~/chirper

  1. sbt update (do this once to setup the probject)
  2. Make sure your twitter username/password is set in the config file
  3. sbt run

You will see a list of classes that ca be run:

Multiple main classes detected, select one to run:

[1] com.linkedin.chirper.streamer.ChirpStream
[2] com.linkedin.chirper.services.ZookeeperRunner
[3] com.linkedin.chirper.services.KafkaRunner
[4] com.linkedin.chirper.services.VoldemortRunner
[5] com.linkedin.chirper.search.ChirpSearchNode

Enter number:

Select a class to run (you'll need to run one per console window)

The order of class to be run are:

  1. [2] - Zookeeper
  2. [3] - Kafka
  3. [4] - Voldemort
  4. [1] - Streamer
  5. [5] - Search Node

Each of the components are pre-configured, details see configs.

The last thing to run is a Restful servlet that interacts with the system (Comes with a beautiful UI):

chirper$ sbt
[info] Building project Chirper 1.0 against Scala 2.8.0
[info]    using ChirperStreamerProject with sbt 0.7.4 and Scala 2.7.7
> jetty-run

Now you can point to: