higher accuracies for experiment two with modified prompt

[sradc]

Hi there, super interesting work.

Looks like the accuracy of gpt-4 might be higher with this prompt.

[lukasberglund]

Hi @sradc, thanks for pointing this out and glad you enjoyed our paper. What prompt did you use to achieve this accuracy?

[sradc]

Hey, the notebook is included in this MR, (and the predictions themselves). Will include below for convenience:

PROMPT_TEMPLATE = """
This is a quiz related to celebrities, and their families.
Here are some example question and answers:

Q: A parent of X is Fahimeh Rahim Nia. Who is X?
Golshifteh Farahani

Q: A parent of X is Timothy Christopher Mara. Who is X?
Kate Mara

Q: A parent of X is Samira Calle. Who is X?
Sasha Calle

Q: A parent of X is Fiona Biggar. Who is X?
Daniel Portman

Now answer (response with just the name):
Q: A parent of X is {parent}. Who is X?
""".strip()

(Note that the example is removed from the prompt, if it's for the celebrity being tested.)

Also running this on gpt3.5 currently.

Edit: also, used this for the system prompt:

You are a helpful assistant, being quizzed on celebrities. If you are not sure, you **must** guess a name.

[sradc]

gtp-3.5turbo seems to get around 45% accuracy with this prompt (included results in previous commit)

[sradc]

Pushed updates. Best results so far:

gpt-4 - 0.561 accuracy (not in latest commit)
gpt-3.5-turbo-0613 - 0.484 accuracy

[sradc]

Best results are now in the latest commit:

gpt-4: 0.565 accuracy
gpt-3.5-turbo-0613: 0.516 accuracy

(Probably going to stop now because it's expensive.)

[lukasberglund]

Thanks for pointing these out! I'm not going to merge for now, since your change doesn't really integrate with the existing codebase, but it's cool to see that there are better prompts out there.

[sradc]

No prob, this PR was just to share and track the work. Let me know if you might want to integrate the prompt stuff.