Skip to content
Repository for the slides and code samples of the talk "Functional Stream Processing with Scala" at ScalaItaly2016
JavaScript CSS Scala HTML
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.


Type Name Latest commit message Commit time
Failed to load latest commit information.

Functional Stream Processing with Scala

This repository contains the artefacts of the talk Functional Stream Processing with Scala at ScalaItaly 2016.

Primary objective of this talk was to introduce Scalaz-Stream and discuss its key goals, design philosophy and core building block, Process and its semantics.

Slides of the talk is available here. Credits for the images used in slides: Mario Sixtus.

Code: Sentiment Analysis of Tweets

To get started the examples demonstrated during the talk.

$ git clone
$ cd scalaitaly-functional-stream-processing

Please update the Twitter's access and consumer keys (which can be generated from at src/main/resources/application.conf and then run the application follows:

$ sbt 
> restart

It will start running the Demo Application (a Http4s based service) in http://localhost:8080.

In this talk, we presented two examples.

Example 1

In the first example (see Routes.scala), in every N seconds, we are getting tweets from Twitter.

val query = ...
val source = awakeEvery(5 seconds) |> buildTwitterQuery(query) through queryChannel flatMap {
  Process emitAll _

Then, several transformations have been performed on source, such as sentiment analysis.

source. map(status 
        author = Author(status.getUser.getScreenName),
        retweetCount = status.getRetweetCount,
        body = status.getText)
    ) through analysisChannel map (_.toString)

Finally, we are connecting the stream to two sinks: A websocket endpoint built using Http4s and also a io.stdOutLines sink.

val query = new Query("#spark")
val src1: Process[Task, Text] = twitterSource(query)
        .observe(io.stdOutLines) // sink1 

WS(Exchange(src1, Process.halt)) // sink2 

If we run the application and initiate the websocket request, we can see the live stream from twitter:

Stream from Example1

Example 2

In the first example, we have created the source from scratch. But in real world cases, there may be an existing stream of data that we have use as a source.

Hence, in the second example (see Routes.scala), instead of building a source from scratch, we have used an existing stream with async.boundedQueue, via Twitter’s Streaming API and similarly performed the sentiment analysis and connected it to the websocket endpoint.

import TwitterStreamClient._
val tweetsQueue: Queue[Status] = async.boundedQueue[Status](size)

  .runAsync { _  println("All input data was written") } 
    author = Author(status.getUser.getScreenName),
    retweetCount = status.getRetweetCount,
    body = status.getText)
) through analysisChannel map (_.toString)

Disclaimer: Note that these examples are for the demonstration purpose only. Using them in production system would require some careful consideration.

References and Further Readings

  1. Scalaz-Stream Masterclass by Runar at NE Scala 2016.
  2. Scalaz Task - the missing documentation
  3. scalaz-stream User Notes
  4. Comparing akka-stream and scalaz-stream with code examples
You can’t perform that action at this time.