LLM Cognitive Judgements...

...Differ From Human

Work published as an arXiv pre-print

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International

Repository Structure

data

• data/{TASK}-xaxis.csv: The $t$ values that were used as prompt from ¹
• data/{TASK}-participants.csv: Participants' answers for $t_\text{total}$ from ¹
• data/{TASK}-model.csv: Model predictions for $t_\text{total}$ from ¹

data/prompts

• data/prompts/{TASK}: Prompts for each task with %d placeholder for the actual value from ¹

src

• src/utils.py: Helper utilities
• src/query-openai.py: Script to query OpenAI GPT-3 and ChatGPT with the tasks
• src/plot.py: Script to reproduce the plots and tables in the paper (needs texfig)

results

• results/gpt3.csv: GPT-3 Answers
• results/gpt35.csv: ChatGPT Answers

Footnotes

1. Griffiths, Thomas L., and Joshua B. Tenenbaum. "Optimal predictions in everyday cognition." Psychological science 17.9 (2006): 767-773. ↩ ↩² ↩³ ↩⁴