Skip to content


Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Interactive Multi-document Summarization Using Joint Optimization and Active Learning for Content Selection Grounded in User Feedback

In this project, we develop a general framework for Interactive Multi-Document Summarization. We propose an extractive multi-document summarization (MDS) system using joint optimization and active learning for content selection grounded in user feedback.

If you reuse this software, please use the following citation:

    title = {Joint Optimization of User-desired Content in Multi-document Summaries by Learning from User Feedback},
    author = {P.V.S., Avinesh and Meyer, Christian M.},
    publisher = {Association for Computational Linguistics},
    volume = {Volume 1: Long Paper},
    booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017)},
    pages = {(to appear)},
    month = aug,
    year = {2017},
    location = {Vancouver, Canada},

Abstract: In this paper, we propose an extractive multi-document summarization (MDS) system using joint optimization and active learning for content selection grounded in user feedback. Our method interactively obtains user feedback to gradually improve the results of a state-of-the-art integer linear programming (ILP) framework for MDS. Our methods complement fully automatic methods in producing high-quality summaries with a minimum number of iterations and feedbacks. We conduct multiple simulation-based experiments and analyze the effect of feedback-based concept selection in the ILP setup in order to maximize the user-desired content in the summary.

Contact person:

Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.


  • python >= 2.7 (tested with 2.7.6)


  1. Download ROUGE package from the link and place it in the rouge directory

    mv RELEASE-1.5.5 rouge/
  2. Install required python packages.

    pip install -r requirements.txt
  3. Download the Standford Parser models and jars from the link

    mv englishPCFG.ser.gz germanPCFG.ser.gz jars/
    mv stanford-parser.jar stanford-parser-3.6.0-models.jar jars/		
  4. [Optional] To run the system for active learning models

    Download the Google embeddings (English) from the link

    mkdir -p summarizer/data/embeddings/english
    mv GoogleNews-vectors-negative300.bin.gz summarizer/data/embeddings/english

    Download the News, Wikipedia embeddings (German) from the link

    mkdir -p summarizer/data/embeddings/german
    mv 2014_tudarmstadt_german_50mincount.vec summarizer/data/embeddings/german
  5. [Optional] To solve ILPs using CPLEX (faster), which can be obtained from IBM here: link. Install the cplex python package.

    cd cplex_installation_dir/python
    python setup install

To Run

  1. Make sure that you have the raw datasets available. Each raw dataset needs to be extracted and follow the following directory structure:

     |  |
     |  +-+d3103t
     |    | 
     |    +-+ many files
     |  |
     |  +-+d31001t
     |  |
     |  +-+ many files
  2. Before running the pipeline, you have to preprocess the raw datasets using the script. Replace the DUC_TEST with appropriate dataset and run the same command.

       python summarizer/data_processer/ -d DUC_TEST -p summarizer/data/raw  -a parse -l english

    The results should then be copied into a directory. We recommend using the --iobasedir argument to set the directory

     |  |
     |  +--+raw/
     |     |
     |     +--+DUC_TEST/
     |     |  |
     |     |  +--+d31013t/	 
     |     |  |
     |     |  +--+docs/
     |     |  |
     |     |  +--+models/
     |     |  |
     |  +--+processed/
     |     |
     |     +--+DUC_TEST/
     |     |  |
     |     |  +--+d31013t/
     |     |  |  |
     |     |  |  +--+docs/
     |     |  |  |
     |     |  |  +--+docs.parsed/
     |     |  |  |
     |     |  |  +--+summaries/
     |     |  |  |
     |     |  |  +--+summaries.parsed/
     |     |  |  |
     |     |  |  +--+summaries.upperbound/
     |     |  |  |
     |     |  |  +--+task.json
     |     |  |
     |     |  +--+...
     |     |
     |     +--+ ...
     |  |
     |  +-+GoogleNews-vectors-negative300.bin
     |  |
     |  +-+data/
  3. python --help for more details

        python --summary_size=100 --oracle_type=accept_reject --data_set=DUC_TEST --summarizer_type=feedback --language=english
        pyhton --summary_size=100 --oracle_type=accept_reject --data_set=DUC_TEST --summarizer_type=baselines --language=english --rouge=rouge/RELEASE-1.5.5/ --iobasedir=outputs/
  4. Bash file for the experiments in the paper and sample outputs of the system for DBS corpus

        ls outputs/DBS

Dataset notes

  • In DUC2004, task 5, topic d151h, document APW20000104.0268 produces and

    xml.etree.ElementTree.ParseError: mismatched tag: line 78, column 2

    The reason is a missing opening tag <P> in row 72.

  • In DUC2006, topic D0614E, Model Summary B D0614.M.250.E.B. To fix it, Chrétien has to be replaced by Chretien. (Two times)

Windows setup

Verified by one (1) user.

  1. download + install anaconda2 python 2.7.12 64bit from , e.g.

    • take care that it is NOT python 2.7.13, as that version contains a regression bug which breaks pulp

    TypeError: LoadLibrary() argument 1 must be string, not unicode


  2. download + install strawberry perl 64bit. In my case, Strawberry Perl (

  3. download + install eclipse neon.2

  4. download + instlal eclipse pydev

  5. install perl module XML::DOM

  6. install python modules

    	pip install -r requirements.txt
  7. configure eclipse pydev run configuration as set up here:

    --summary_size=100 --oracle_type=accept_reject --data_set=TEST --summarizer_type=feedback --language=english

  8. Create a directory "tmp" on your root, e.g. "C:\tmp"!



A general framework for Interactive Multi-Document Summarization







No releases published


No packages published