Skip to content



Repository files navigation

Sherlock: An Interactive Summarization of Large Text Collections.

In this demo paper, we present a new system for interactive text summarization called Sherlock. The task of automatically producing textual summaries is an important step to understand a collection of multiple topic-related documents. It has many real-world applications in journalism, medicine, and many more. However, none of the existing summarization systems allow users to provide feedback at interactive speed. We therefore integrate a new approximate summarization model into Sherlock that can guarantee interactive speeds even for large text collections to keep the user engaged in the process.

If you reuse this software, please use the following citation:

  author = {P.V.S., Avinesh and Hättasch, Benjamin and Özyurt, Orkan and Binnig, Carsten and Meyer, Christian M.}, 
  title = {{Sherlock: A System for Interactive Summarization of Large Text Collections}}, 
  booktitle = {Proceedings of the VLDB Endowment}, 
  pages = {1902--1905}, 
  volume = {11}, 
  number = {12}, 
  month = {August}, 
  year = {2018}, 
  location = {Rio de Janeiro, Brazil}, 
  language = {English}, 
  doi = {10.14778/3229863.3236220}, 
  pdf = {}, 
  url = {} 

Contact person:

Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.


  • python >= 2.7 (tested with 2.7.6)
  • jdk 8
  • JAVA_HOME env variable has to be set

Setting up the Sherlock UI

  1. Install Anaconda

  2. Install the requirements

    pip install -r requirements.txt
  3. Install GLPK and CPLEX for PULP (Python Integer Linear Programming Package)

    • Install GLPK
      sudo apt-get install libglpk-dev
    • Install CPLEX for Pulp
      cd ukpsummarizer-be/cplex/cplex/python/2.7/x86-64_linux/
      python install 
  4. Perl dependencies for ROUGE:

    • LOCAL::LIB
      sudo apt-get install liblocal-lib-perl
    • XML::DOM
      sudo apt-get install libxml-dom-perl
    • libexpat :
      sudo apt-get install libexpat-dev
    • libparser
      sudo apt-get install libxml-parser-perl
  5. Bash as the default (source to work):

    sudo dpkg-reconfigure dash

Build and Run

Setup the sample data on your system.

cp -r data ~/.ukpsummarizer/datasets

The result of the build produces dist/ukpsummarizer-dist-bin.tar file which should be a standalone bundle.

./mvnw clean install
./mvnw -pl ukpsummarizer-server spring-boot:run


tar -xvf dist/ukpsummarizer-dist-bin.tar
java -jar ukpsummarizer-server.jar

Preparing Data Directory:

  1. Create a io directory, perefably in ~/.ukpsummarizer, which has the following layout:

     |  +--+raw/
     |  +--+processed/
     |     +--+DUC2006/
     |     |  +--+D0601A/
     |     |  |  +--+docs/
     |     |  |  +--+docs.parsed/
     |     |  |  +--+summaries/
     |     |  |  +--+summaries.parsed/
     |     |  |  +--+summaries.upperbound/
     |     |  +--+task.json
     |     |  +--+...
     |     +--+ ...
     |  +--+english/
     |  |  +--+GoogleNews-vectors-negative300.bin
     |  |     +data/
     |  +--+german/
     |     +2014_tudarmstadt_german_50mincount.vec
  2. Download and add the word embeddings into the ~/.ukpsummarizer/embeddings directory

    Download the Google embeddings (English) from here

      >> mkdir -p summarizer/data/embeddings/english
      >> mv GoogleNews-vectors-negative300.bin.gz ~/.ukpsummarizer/embeddings/english

    Download the News, Wikipedia embeddings (German) from here

      >> mkdir -p summarizer/data/embeddings/german
      >> mv 2014_tudarmstadt_german_50mincount.vec ~/.ukpsummarizer/embeddings/german

    Download and install the GloVe embeddings from here

      >> mkdir -p summarizer/data/embeddings/english/glove
      >> mv *.txt.w2v ~/.ukpsummarizer/embeddings/english/glove
  3. Make sure that you have the raw datasets available. Each raw dataset needs to be extracted and follow the following directory structure:

        |  +-+D0601A
        |    +-+ many files
        |  +-+D0650E
        |  +-+ many files
  4. Before running the pipeline, you have to preprocess the raw datasets using the script.

    python ukpsummarizer-be/data_processer/ -d DUC2006  -p ~/.ukpsummarizer/datasets/raw  -a parse -l english
    python ukpsummarizer-be/data_processer/ -d DUC2004  -p ~/.ukpsummarizer/datasets/raw  -a parse -l english
    python ukpsummarizer-be/data_processer/ -d TEST     -p ~/.ukpsummarizer/datasets/raw  -a parse -l english
    python ukpsummarizer-be/data_processer/ -d DBS      -p ~/.ukpsummarizer/datasets/raw  -a parse -l german

    The results should then be copied into a directory. We recommend using the --iobasedir argument to set the directory

     |  +--+raw/
     |  +--+processed/
     |     +--+DUC2006/
     |     |  +--+D0601A/
     |     |  |  +--+docs/
     |     |  |  +--+docs.parsed/
     |     |  |  +--+summaries/
     |     |  |  +--+summaries.parsed/
     |     |  |  +--+summaries.upperbound/
     |     |  |  +--+task.json
     |     |  +--+...
     |     +--+ ...

Windows setup

Verified by one (1) user.

  1. Download and install anaconda2 python 2.7.12 64bit from , e.g.

    • take care that it is NOT python 2.7.13, as that version contains a regression bug which breaks pulp

    TypeError: LoadLibrary() argument 1 must be string, not unicode


  2. Download + install strawberry perl 64bit. In my case, Strawberry Perl (

  3. download + install eclipse neon.2

  4. download + instlal eclipse pydev

  5. install perl module XML::DOM

  6. install python modules

        pip install -r requirements.txt


No description, website, or topics provided.







No releases published


No packages published


  • Python 36.2%
  • Jupyter Notebook 25.1%
  • C++ 18.7%
  • C 8.1%
  • Java 3.9%
  • HTML 3.4%
  • Other 4.6%