New prompts for Shades FR/EN

[aneveol]

This is a test PR for the BigScience eval hackathon. The goals is to use these three new prompts for crowS-pairs to the following end:
1/ test process/pipeline (I think that works)
2/ get feedback on prompts format: are they acceptable?
3/ how will the metric "other" be defined? Here, we essentially need a count of answers, to convert into the proportion of "sentence A" answers.

(Our goal this week is to get the bias eval done on the revised/upgraded crowS-pairs_multilingual dataset).

[KhalidAlt]

The template with id:493b3361-f28d-4489-87b3-b42ae90e5e3a does not have a target or output. You need to add the output as you did for the reset : ||| {{answer_choices[stereo_antistereo]}}

[aneveol]

Thank you for the suggestion. I made the change. However, I am not sure I understand whether this is going to have the desired effect for the metric planned ("other").

[oskarvanderwal]

@aneveol, answer_choices[stereo_antistereo] wouldn't work as the target, since stereo_antistereo is not an integer label of which choice is the most stereotypical answer, but a string indicating whether it is about a stereotypical choice or an anti-stereotypical choice. I think we should always suggest the sent_more (more stereotypical choice) as the target.

I have fixed this and also added your prompts to my PR.

[jzf2101]

@oskarvanderwal we have merge conflicts that have to be resolved - also I don't see the dataset listed on promptsource. I think this PR needs to be cleaned up before merging. For some reason the PR did not build.

[oskarvanderwal]

@jzf2101 Sorry, I don't have access to this repository (yet). My PR for CrowS-Pairs can be found here. This PR is about the bias-shades dataset, but I agree we need to clean some things up.

@aneveol Maybe you could rename this PR to "Adding prompts for French and English bias-shades"? Then also

• remove the CrowS-Pairs prompts,
• add BigScienceEvalWG as user, and
• make sure your repository is up-to-date with the original.
  I can help out with this if you add me as contributor to this repo :)

[oskarvanderwal]

@jzf2101 I have cleaned this PR and fixed the prompt answer choices. Let me know if there is anything else required for this PR!

[jzf2101]

@oskarvanderwal there are merge conflicts, could you resolve?

[StellaAthena]

@jzf2101 Are there additional changes that need to be made to this PR or can it be merged?

[oskarvanderwal]

@jzf2101, I have created a new PR for adding French and English prompts without any blocking merge conflicts here: #837

This PR can be closed.

[jzf2101]

Closed for #837