Skip to content

jpzk/twitterstream

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

Processing Tweets with Kafka Streams in Scala

The example application consists of two services written in Scala, an ingestion service (code) and an aggregation service (code). The ingestion service subscribes to the Twitter Streaming API and receives fresh tweets filtered by a list of terms. Any raw tweet is sent to the Kafka topic 'tweets' in JSON. The aggregation service retrieves raw tweets, parses tweets, and aggregates word counts in tumbling time windows, see the code here. Kafka Streams uses an embedded RocksDB for maintaining a local state. Any change to the aggregate will be propagated to the topic 'aggregate'.

Both services share the same SBT project, and will be located in the same fat jar including all dependencies. Which allows us to easily share code in this small example project. Both applications access the application.conf in runtime via the Settings object, see code. I wrote a small build script to compile the services, building the Docker images and running the containers.

Read the full article

Twitter Hosebird Client: References

Kafka Streams: References

Official Documentation

Other Code Examples

Articles

About

Twitter Streaming API Example with Kafka Streams in Scala

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published