Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Align question answering tasks with sub-domains #2371

Closed
lewtun opened this issue May 18, 2021 · 1 comment
Closed

Align question answering tasks with sub-domains #2371

lewtun opened this issue May 18, 2021 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@lewtun
Copy link
Member

lewtun commented May 18, 2021

As pointed out by @thomwolf in #2255 we should consider breaking with the pipeline taxonomy of transformers to account for the various types of question-answering domains:

question-answering exists in two forms: abstractive and extractive question answering.

we can keep a generic question-answering but then it will probably mean diferrent schema of input/output for both (abstractive will have text for both while extractive can use spans indication as well as text).

Or we can also propose to use abstractive-question-answering and extractive-question-answering for instance.
Maybe we could have question-answering-abstractive and question-answering-extractive if somehow we can use a for a completion or search in the future (detail).
Actually I see that people are more organizing in terms of general and sub-tasks, for instance on paperwithcode: https://paperswithcode.com/area/natural-language-processing and on nlpprogress: https://github.com/sebastianruder/NLP-progress/blob/master/english/question_answering.md#squad

Probably the best is to align with one of these in terms of denomination, PaperWithCode is probably the most active and maintained and we work with them as well.
Maybe you want to check with a few QA datasets that this schema make sense. Typically NaturalQuestions, TriviaQA and can be good second datasets to compare to and be sure of the generality of the schema.

A good recent list of QA datasets to compare the schemas among, is for instance in the UnitedQA paper: https://arxiv.org/abs/2101.00178

Investigate which grouping of QA is best suited for datasets and adapt / extend the QA task template accordingly.

@mariosasko
Copy link
Collaborator

Closing this issue as the task_templates API has been deprecated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants