Skip to content


Folders and files

Last commit message
Last commit date

Latest commit



32 Commits

Repository files navigation

Real-Time Burst Detection and Summarization in Twitter Streams

This code uses Spark Streaming to identify interesting tokens related to some given set of topics. Tweets containing these tokens are then saved to a file.

Operates by first splitting Twitter stream by topic using Lucene and then identifying bursty tokens in those topic streams.


This package has a number of dependencies spelled out in pom.xml but also expects to be executed via Apache Spark, v2.1.1+.

This code also requires Java 1.8 and will fail with newer versions of Java.


To build this code, you need only run mvn package, which will produce an "uber" jar with all relevant dependencies.


Primary configuration for this package is set in the conf/ file, which specifies several configurable options, including window size, match thresholds, the minimum number of tokens in a candidate tweet, etc.

Configuring Topic Streams

This software connects to a filtered Twitter stream, where we search for tweets that include a set of keywords or search terms present in a filter_keywords.txt file you provide to spark-submit.

Within that Twitter stream, you can also subdivide the stream into separate topics using the topics.json configuration file. This file contains a JSON array of dictionaries, where each element contains the following:

    "topid": "cv19",
    "query": "covid19 OR vaccine OR vacine OR vax OR vaxx OR cv19",
    "description": "Possible description of this topic",
    "narrative": "Test of CV19"

The topid field is an ID for this topic, used in logging. The description and narrative fields are informational, and their values are ignored.

The query field, however, defines a Lucene query that can be applied to a tweet's body to connect a particular tweet to this particular topic. This query corresponds to the MATCH_THRESHOLD field in the properties file, as the higher this threshold, the more a given tweet must match this query to be considered part of this topic. Alternatively, you can set MATCH_THRESHOLD to 0.0, and all tweets will be considered relevant to all topics.

Configuring Twitter Stream

To connect to a Twitter stream for real-time processing, the developer must also provide a file (or related configuration for Twitter4j), which specifies OAuth tokens. This properties file is expected to be in the current working directory.

Executing Stream Processing

To execute this streaming system, you need to submit this code via Spark's spark-submit command. An example command is as follows:

spark-submit \
    --master 'local[*]' \
    --conf spark.ui.port=6886 \
    --executor-memory 2g \
    --conf spark.executor.memoryOverhead=16g \
    --conf spark.driver.maxResultSize=16g \
    --class \
    target/BurstyTwitterStream-2.0-jar-with-dependencies.jar \
    <> \
    <topics.json> \
    <log_file.log> \


Code backing the TREC runs for the Microblag Track.







No packages published