Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Seq2Seq QA example script to use SQuAD metric. #14212

Conversation

karthikrangasai
Copy link
Contributor

What does this PR do?

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @NielsRogge

Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good switch to the SQUAD metric! I realize however that the processing is not completely adapted to what we do: the answer is generated from the first feature created by each example, so there is really no point keeping all the features.

We either need to figure out a score for a generated answer, so that we can pick the best answer corresponding to all the features generated by one example, or we can just simplify the preprocessing and only keep one feature per example.

@karthikrangasai
Copy link
Contributor Author

In the extractive method, the logits are used to score the outputs but for generative there isn't any value we can define to rank it because the sentence is already generated. So I suppose we can just use one feature per example.

@sgugger
Copy link
Collaborator

sgugger commented Nov 8, 2021

It looks like GitHub does not like your rebase and is showing a diff with ~200 files touched. Could you close this PR and create a fresh one?

@karthikrangasai
Copy link
Contributor Author

karthikrangasai commented Nov 9, 2021

Sorry for the late reply.
I didn't notice the error while pushing the code.
I have created the new PR here at #14335 .
I will close this one now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants