-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add bias-shades #37
Add bias-shades #37
Conversation
623a862
to
d0d731b
Compare
Removed language subsets that have no prompts yet.
This PR depends on the French and English prompts to be added in promptsource. |
…p#143) * [feature] Add task constructor arg support to CLI * Add tests * Fix bad text separator arg test * Loosen CLI parsing `strip` and `split` constraints * Update CLI arg string task creation unit tests * Update `main.py` arg help for `--task_args` * Add `task_args` to results metadata * Add arg string parsing edge case tests * Add misc. symbol checks to test
Removed language subsets that have no prompts yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@oskarvanderwal Thanks for putting this together! I've added bias-shades to the task registry and did a little refactoring to match similar tasks.
I ran into one minor issue - for some reason, the Spanish convert_to_stereotype template does not appear to be working properly
text, _ = self.prompt_template.apply(doc)
ValueError: not enough values to unpack (expected 2, got 1)
These errors almost always stem from the promptsource
side. If you can confirm this, then we can go ahead and merge this.
@jon-tow, thanks for your work! I have asked the author of the Spanish prompts to review the problematic template. I'm pretty sure the problem is coming from the promptsource side, so those prompts should be checked. |
This task implements the bias-shades dataset for French and English. I'll add the other languages, when the respective prompts are available.