Skip to content
ROUGE summarization evaluation metric, enhanced with use of Word Embeddings
Perl Python Roff
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


ROUGE summarization evaluation metric, enhanced with use of Word Embeddings as described in:

Better Summarization Evaluation with Word Embeddings for ROUGE
Jun-Ping Ng and Viktoria Abrecht
in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015

For any questions, you may get in touch with me via email: Jun-Ping Ng


This program helps compute the ROUGE-WE scores of summaries. ROUGE-WE builds on top of ROUGE (


You will need the following:

  1. word2vec pre-trained vectors


  1. Start up the word2vec query server:

? python -m

  1. Test the word2vec query server by sending a HTTP Post request to: http://localhost:8888/word2vecdiff

You can do this in a variety of ways, such as by using curl. The following examples work in OS X:

? curl -X POST --data "word1=king&word2=queen" http://localhost:8888/word2vecdiff
{"status": 1, "word2vec_sim": 0.651095648143}

? curl -X POST --data "word1=raining heavily&word2=snowing badly" http://localhost:8888/word2vecdiff
{"status": 1, "word2vec_sim": 0.293822419014}

  1. Run ROUGE-WE in the same way you would with

NOTE: First create the necessary config file. A sample has been pre-created. Some sample data is found in rouge_1.5.5_data and the XML required is given in sample-config.xml
? ./ -x -n 2 -U -2 4 -e rouge_1.5.5_data/ -c 95 -a sample-config.xml

Sample output:

1 ROUGE-1 Average_R: 0.23145 ( 0.23145 - 0.23145)
1 ROUGE-1 Average_P: 0.27279 ( 0.27279 - 0.27279)
1 ROUGE-1 Average_F: 0.25043 ( 0.25043 - 0.25043)
1 ROUGE-2 Average_R: 0.05782 ( 0.05782 - 0.05782)
1 ROUGE-2 Average_P: 0.06894 ( 0.06894 - 0.06894)
1 ROUGE-2 Average_F: 0.06289 ( 0.06289 - 0.06289) ...... ......


An update is under development, which does away with the Python based web server, and directly loads the word2vec vectors inside Perl instead. I hope to get this done real soon.

You can’t perform that action at this time.