Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong usage of Label encoding for STSBenchmark task #83

Open
marco-digio opened this issue Apr 29, 2021 · 0 comments
Open

Wrong usage of Label encoding for STSBenchmark task #83

marco-digio opened this issue Apr 29, 2021 · 0 comments

Comments

@marco-digio
Copy link

The encode_labels() function in senteval/sick.py from this article is meant for labels [1, ..., K] (see section 4.2 of the paper).
STSBenchmarkEval class inherits SICKRelatednessEval, so it inherits its encode_labels() function. However, STSBenchmark task has labels from 0 to 5.
Thus by constructions, a model trained in this way will never predict correctly data with label in [0, 1].

It is easy to check this issue by running examples/bow.py on STSBenchmark task and printing min(results['STSBenchmark']['yhat']) that will be always greater than 1!

An easy way to fix this could be shifting the original labels in senteval/sts.py:
sick_data['y'] = [float(s)+1 for s in sick_data['y']]
and then fix the ranges in the rest of the code. However, that will probably mess the code for SICK task, that is currently correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant