Skip to content

Human Generated Questions in ''Shortcomings of Question Answering Based Factuality Frameworks for Error Localization'' (EACL2023)

Notifications You must be signed in to change notification settings

ryokamoi/QA-metrics-human-annotation

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Human Generated Questions in ''Shortcomings of Question Answering Based Factuality Frameworks for Error Localization''

Human generated questions used in the analysis in the paper Shortcomings of Question Answering Based Factuality Frameworks for Error Localization (Section 6)

Ryo Kamoi, Tanya Goyal and Greg Durrett, EACL 2023 (main conference).

Overview of this Paper

Our paper analyzes Question Answering based factuality evaluation frameworks such as QuestEval and QAFactEval. Although QA-based frameworks are motivated by their interpretability, we show that this is not true and QA-based frameworks cannot localize errors in summaries.

QA-based franeworks consist of question answering (QA) and question generation (QG) models. In Section 6 of our paper, we evaluate the upporbound performance of QA-based framework by replacing QG model with human generated questions.

Overview of this Repository

This repository includes human generated questions used in Section 6 of the paper.

cliff_cnndm and cliff_xsum directories include human generated questions for CNN/DM and XSum split of CLIFF dataset.

  • cliff_cnndm_{split}.json: Subset of CLIFF dataset corresponding to the human annotated questions
  • human_generated_{question_type}_{split}.json: human annotated questions

About

Human Generated Questions in ''Shortcomings of Question Answering Based Factuality Frameworks for Error Localization'' (EACL2023)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published