Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

distributed twitter search engine

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 build
Octocat-spinner-32 config
Octocat-spinner-32 lib
Octocat-spinner-32 project
Octocat-spinner-32 src
Octocat-spinner-32 README.md
Octocat-spinner-32 sbt
README.md

What is Chirper?

(http://javasoze.github.com/chirper/)

Chirper is a real time tweet search engine (written in Scala) using open source technology built by the LinkedIn SNA team:

Although the amount of code is minimal, the system can scale horizontally infinitely by leveraging the distributed systems above. See this blog post on design considerations.

Build and run the system:

From the top level directory, e.g. ~/chirper

  1. sbt update (do this once to setup the probject)
  2. Make sure your twitter username/password is set in the config file
  3. sbt run

You will see a list of classes that ca be run:

Multiple main classes detected, select one to run:

[1] com.linkedin.chirper.streamer.ChirpStream
[2] com.linkedin.chirper.services.ZookeeperRunner
[3] com.linkedin.chirper.services.KafkaRunner
[4] com.linkedin.chirper.services.VoldemortRunner
[5] com.linkedin.chirper.search.ChirpSearchNode

Enter number:

Select a class to run (you'll need to run one per console window)

The order of class to be run are:

  1. [2] - Zookeeper
  2. [3] - Kafka
  3. [4] - Voldemort
  4. [1] - Streamer
  5. [5] - Search Node

Each of the components are pre-configured, details see configs.

The last thing to run is a Restful servlet that interacts with the system (Comes with a beautiful UI):

chirper$ sbt
[info] Building project Chirper 1.0 against Scala 2.8.0
[info]    using ChirperStreamerProject with sbt 0.7.4 and Scala 2.7.7
> jetty-run

Now you can point to:

Something went wrong with that request. Please try again.