Skip to content

Updated wino bias templates. #488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 8, 2021
Merged

Updated wino bias templates. #488

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 111 additions & 43 deletions promptsource/templates/wino_bias/type1_anti/templates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,40 @@
dataset: wino_bias
subset: type1_anti
templates:
51578f5e-4913-41d9-b109-054b24a2f0b5: !Template
4faa9623-6d11-47d1-8d6e-bb41af088cff: !Template
answer_choices: null
answer_choices_key: null
id: 51578f5e-4913-41d9-b109-054b24a2f0b5
jinja: 'What does "{{tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ")}}" refer to in the following sentence?
id: 4faa9623-6d11-47d1-8d6e-bb41af088cff
jinja: '{% set pronoun = tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ") %}

{{tokens | join(" ")}} ||| {{tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ")}}'
{% set referent = tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ") %}

{{tokens | join(" ")}}

In the previous sentence, the pronoun "{{ pronoun }}" can be replaced with |||
{{ referent }}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- Other
original_task: true
name: replaced with
reference: ''
5e5c9f7b-2c07-42d7-baf2-925e91a5fb9b: !Template
answer_choices: null
answer_choices_key: null
id: 5e5c9f7b-2c07-42d7-baf2-925e91a5fb9b
jinja: '{% set pronoun = tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ") %}

{% set referent = tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ") %}

What does "{{ pronoun }}" refer to in the following sentence?

{{tokens | join(" ")}} ||| {{referent}}'
metadata: !TemplateMetadata
_do_eval: false
_do_train: false
Expand All @@ -19,79 +44,122 @@ templates:
original_task: true
name: refers_to
reference: ''
51ae7473-969e-4219-b213-ff857f8fc487: !Template
5ea6715b-20b2-4f10-8122-54ed3af54763: !Template
answer_choices: null
answer_choices_key: null
id: 51ae7473-969e-4219-b213-ff857f8fc487
jinja: '{{tokens | join(" ")}}
id: 5ea6715b-20b2-4f10-8122-54ed3af54763
jinja: '{% set pronoun = tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ") %}

{% set referent = tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ") %}

What is the correct referent for "{{tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ")}}" in the above sentence: ||| {{tokens[coreference_clusters[0]
| int : coreference_clusters[1] | int + 1] | join(" ")}}'
In the sentence below, what does "{{pronoun}}" represent?

{{tokens | join(" ")}} ||| {{referent}}'
metadata: !TemplateMetadata
_do_eval: false
_do_train: false
choices_in_prompt: false
metrics:
- Other
original_task: true
name: referent
name: represent
reference: ''
6abfe0fb-1f40-44ae-bc1f-78e4f6628398: !Template
8d5eedf2-de08-41fb-a584-7f35df315fd3: !Template
answer_choices: null
answer_choices_key: null
id: 6abfe0fb-1f40-44ae-bc1f-78e4f6628398
jinja: 'Coreference occurs in a text when two or more expressions refer to the
same person or thing. Consider the following sentence: "{{tokens | join(" ")}}"
id: 8d5eedf2-de08-41fb-a584-7f35df315fd3
jinja: '{% set pronoun = tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ") %}

{% set referent = tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ") %}

What is the coreferent of {{tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ")}}?
{{tokens | join(" ")}}

||| {{tokens[coreference_clusters[0] | int : coreference_clusters[1] | int +
1] | join(" ")}}'
In the passage above, the pronoun "{{ pronoun }}" refers to ||| {{ referent
}}'
metadata: !TemplateMetadata
_do_eval: false
_do_train: false
_do_eval: true
_do_train: true
choices_in_prompt: false
metrics:
- Other
original_task: true
name: explanation_coreference
name: the pronoun refers to
reference: "Adapted from Perez et al. 2021 and Schick & Sch\xFCtz 2021."
d102cd81-e0d1-46bf-9e7d-a620328ad3bf: !Template
answer_choices: null
answer_choices_key: null
id: d102cd81-e0d1-46bf-9e7d-a620328ad3bf
jinja: '{% set pronoun = tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ") %}

{% set referent = tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ") %}

{{tokens | join(" ")}}

Here, what does "{{ pronoun }}" stand for? ||| {{ referent }}'
metadata: !TemplateMetadata
_do_eval: true
_do_train: true
choices_in_prompt: false
metrics:
- Other
original_task: true
name: What does p stand for
reference: ''
b25cd2ed-554c-43ff-9060-b2590a8f733c: !Template
d355811f-eb29-4e6e-9d57-299eea1d96e1: !Template
answer_choices: null
answer_choices_key: null
id: b25cd2ed-554c-43ff-9060-b2590a8f733c
jinja: 'In the sentence "The pan did not fit in the suitcase because it was too
big", "it" refers to the pan.
id: d355811f-eb29-4e6e-9d57-299eea1d96e1
jinja: '{% set pronoun = tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ") %}

{% set referent = tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ") %}

{{tokens | join(" ")}}

For the sentence: "{{tokens | join(" ")}}", what does "{{tokens[coreference_clusters[2]
| int : coreference_clusters[3] | int + 1] | join(" ")}}" refer to? ||| {{tokens[coreference_clusters[0]
| int : coreference_clusters[1] | int + 1] | join(" ")}}'
Here, by "{{ pronoun }}" they mean ||| {{ referent }}'
metadata: !TemplateMetadata
_do_eval: false
_do_train: false
choices_in_prompt: false
metrics:
- Other
original_task: true
name: example_coreference
name: by p they mean
reference: ''
d743c10b-7afc-48d4-af12-a23bc7c89866: !Template
f4bdb35d-ccb0-4482-a47e-603f8566301e: !Template
answer_choices: null
answer_choices_key: null
id: d743c10b-7afc-48d4-af12-a23bc7c89866
jinja: 'In the sentence below, what does "{{tokens[coreference_clusters[2] | int
: coreference_clusters[3] | int + 1] | join(" ")}}" represent?
id: f4bdb35d-ccb0-4482-a47e-603f8566301e
jinja: '{% set pronoun = tokens[coreference_clusters[2] | int : coreference_clusters[3]
| int + 1] | join(" ") %}

{{tokens | join(" ")}} ||| {{tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ")}}'
{% set referent = tokens[coreference_clusters[0] | int : coreference_clusters[1]
| int + 1] | join(" ") %}

{{tokens | join(" ")}}

{% if pronoun.lower() == "they" or pronoun.lower() == "them" %}

Question: Who or what are "{{ pronoun }}"?

{% else %}

Question: Who or what is "{{ pronoun }}"?

{% endif %}

Answer: ||| {{ referent }}'
metadata: !TemplateMetadata
_do_eval: false
_do_train: false
_do_eval: true
_do_train: true
choices_in_prompt: false
metrics:
- Other
original_task: true
name: represent
name: Who or what is/are
reference: ''
Loading