Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corr2Cause #3

Merged
merged 8 commits into from
Nov 8, 2023
Merged

Corr2Cause #3

merged 8 commits into from
Nov 8, 2023

Conversation

msaroufim
Copy link
Member

@msaroufim msaroufim commented Oct 31, 2023

The only thing we might want to wordsmith more is the prompt in get_corr2cause_spec()

       
                You will be given a scenario which includes a premise and a hypothesis. It is your task to determine whether the hypothesis is correct given the premise.
                You will be presented with two choices yes or no for each scenario. Please select the correct choice.
                
            
            Scenario
            : Premise: Suppose there is a closed system of 6 variables, A, B, C, D, E and F. All the statistical relations among these 6 variables are as follows: A correlates with B. A correlates with D. A correlates with E. A correlates with F. B correlates with D. B correlates with E. B correlates with F. C correlates with D. C correlates with E. C correlates with F. D correlates with E. D correlates with F. E correlates with F. However, A is independent of C. A and C are independent given B. A and C are independent given B and D. A and D are independent given B. A and D are independent given B and C. B is independent of C. B and C are independent given A. B and E are independent given A, C and D. C and F are independent given A, B, D and E.
            Hypothesis: Some variable(s) cause(s) both F and C.
            A. No
            B. Yes
            Answer: A

@msaroufim
Copy link
Member Author

msaroufim commented Nov 6, 2023

Answer if hypothesis is correct given the premise

Again use the generation adapter and the text input is premise, hypothesis and then then 2 references is the yes or no

Comment on lines 1026 to 1027
You will be given a scenario which includes a premise and a hypothesis. It is your task to determine whether the hypothesis is correct given the premise.
You will be presented with two choices yes or no for each scenario. Please select the correct choice.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe: "Give a scenario with a premise and a hypothesis, determine if the hypothesis can be inferred from the premise."

The verb "infer" seems most faithful to the original paper... alternatives are "suggest" and "entail".

csv_reader = csv.reader(f)
next(csv_reader)
for label, question in csv_reader:
# formatted_input = f"{question}\nFirst Option: 0\nSecond Option: 1"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure the formatting is correct here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was an issue where answer was always a again so just fixed

@msaroufim msaroufim merged commit a210323 into neurips_eval Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants