Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Code for the paper: Corpora Evaluation and System Bias Detection in Multi-document Summarization

CORPUS: For Pyramid evaluation through PyrEval: 1 -> Write a python script to copy and paste files from your respective source directories to the appropriate folders in PyrEval(Documents - Raw/model and Ground truth summaries - Raw/peer). This script needs to run in a loop for every topic in the corpus to get the final set of CSVs. 2 -> In file set the below : command to '0' at, remove the infinte loop at, add clear method in the end of autorun at 3 -> Remove any steps of deleting csvs if present in above mentioned clear() method 4 -> Write a python code to run through all the CSVs for each topic and calculate average

For Inv-Pyramid: 1 -> Since we are calculating the SCUs on a per document basis, write a python script to copy a single document 4 times in the Raw/model folder. 2 -> Each SCU generated will could be present multiple times, pick the unique ones among them. 3 -> Document SCUs can be found easily by putting the candidate set in the Raw/model folder. 4 -> Due to limited semantic understanding of Neural nets, SCUs may present in documents may not exactly match the SCU in reference. 5 -> Use any similarity alogorithm to match the SCU's, in our cases after manual evaluation we setted for Cosine similarity above 0.4.


Code for the paper: Corpora Evaluation and System Bias Detection in Multi-document Summarization, Accepted in EMNLP Findings 2020







No releases published


No packages published