This repository contains raw data, experimental programs, and analysis code for the paper Integrating Machine Behavior into Human Subject Experiments: A User-friendly Toolkit and an Application to Framed Prisoner’s Dilemmas by Engel, Grossmann & Ockenfels (2025).
This replication package contains:
- Raw experimental data from GPT-3.5, GPT-4, and human participants playing repeated prisoner's dilemma games under various framing treatments
- An oTree application (
ego_human) used to collect experimental data with human participants interacting with GPT-4 - R scripts (
plots.Randtables.R) that generate all figures and tables in the paper - Library functions (
lib/) for data processing and statistical analysis - Pre-generated output files (plots in PDF format and regression tables in LaTeX format)
The analysis uses R with various packages. Running both scripts (plots.R and tables.R) should take approximately 10-20 minutes on a standard desktop machine.
All data used in this paper were generated by the authors through experimental studies. The data were collected in two types of experiments:
-
GPT-GPT Interactions: The authors generated data by having GPT-3.5 and GPT-4 play repeated prisoner's dilemma games with each other under various experimental framing treatments. These interactions were conducted using the OpenAI API.
-
GPT-Human Interactions: Human participants were recruited online to play repeated prisoner's dilemma games with GPT-4. This data was collected using an oTree web application included in this repository.
The experimental instructions and prompts are fully documented in the ego_human/prompts/ directory, which contains:
- System prompts for both AI (
system_c.txt) and human participants (system_h.txt) - Round-by-round instructions (
first_mover.txt,second_mover.txt) - Result feedback templates (
result.txt) - Three treatment variations (prompts001, prompts003, prompts004) corresponding to different experimental framings
- The authors of the manuscript have legitimate access to and permission to use the data used in this manuscript (the data were generated by the authors).
- The authors of the manuscript have documented permission to redistribute/publish the data contained within this replication package.
The data are licensed under CC0 1.0 Universal (Public Domain Dedication). See LICENSE file for details.
- All data are publicly available and included in this repository.
| Data Name | Data Files | Location | Provided |
|---|---|---|---|
| GPT-3.5 Experimental Data | data35_new.csv | data/ | YES |
| GPT-4 Experimental Data | data4_new.csv | data/ | YES |
| Human-GPT-4 Experimental Data (Aug 11) | all_apps_wide-2023-08-11.csv | data/ | YES |
| Human-GPT-4 Experimental Data (Aug 15) | all_apps_wide-2023-08-15.csv | data/ | YES |
| Human-GPT-4 Experimental Data (Aug 16) | all_apps_wide-2023-08-16.csv | data/ | YES |
| Human-GPT-4 Experimental Data (Aug 17) | all_apps_wide-2023-08-17.csv | data/ | YES |
All experimental data were collected by the authors and are provided in CSV format.
Files: data/data35_new.csv (40,000 observations + header) and data/data4_new.csv (40,000 observations + header)
These files contain data from experiments where GPT-3.5 or GPT-4 played repeated prisoner's dilemma games with each other under various experimental framing treatments. For each of the 10 experimental treatments, there were 200 groups of 2 GPT instances each, interacting over 10 rounds, yielding 40,000 total choice observations (10 treatments × 200 groups × 2 players × 10 rounds).
Variables:
experiment: UUID identifying the experimental sessionconvo: UUID identifying a conversation (pair of AI agents playing together)thread: UUID identifying a specific AI thread/agent instancethread_type: Type of AI thread (e.g., "GPTThread")treatment: Integer code for experimental treatment/framing condition:- 1 = "base" (baseline)
- 2 = "project"
- 3 = "enemy"
- 4 = "competition"
- 5 = "war"
- 6 = "labour"
- 7 = "stats"
- 8 = "couple"
- 9 = "procurement"
- 10 = "discretion"
round_: Round number (1-10)choice: Player's choice in this round (1 = cooperate, 2 = defect)min_time: Timestamp of the first message in the conversation (Unix timestamp)
Files:
data/all_apps_wide-2023-08-11.csv(50 participants + header)data/all_apps_wide-2023-08-15.csv(73 participants + header)data/all_apps_wide-2023-08-16.csv(163 participants + header)data/all_apps_wide-2023-08-17.csv(100 participants + header)
These files contain data from experiments where human participants played repeated prisoner's dilemma games with GPT-4, conducted in August 2023 at the Cologne Laboratory for Economic Research. The raw files contain 386 total participant records. Of these, 82 participants (21%) dropped out before completing all 10 rounds and are excluded from the analysis, leaving 304 complete human-GPT-4 pairs (96 in base condition, 106 in enemy condition, 102 in competition condition). The filtering is performed in lib/data.R line 65 by excluding records where ego_human.10.player.other_choice is missing. Data is stored in wide format with separate columns for each round.
Key Variables:
participant.label: Unique identifier for the human participantego_human.N.player.treatment: Treatment/framing condition for round N (values 1, 3, or 4 corresponding to base, enemy, or competition)ego_human.N.player.choice: Human's choice in round N (1 = cooperate, 2 = defect)ego_human.N.player.other_choice: GPT-4's choice in round N (1 = cooperate, 2 = defect)
Where N ranges from 1 to 10 representing the round number.
All data files are provided in CSV format (open, non-proprietary format readable by R, Python, Excel, and other software).
| Data file | Source | Notes | Provided |
|---|---|---|---|
data/data35_new.csv |
Authors (GPT-3.5 experiments) | 40,000 observations of GPT-3.5 playing prisoner's dilemma | Yes |
data/data4_new.csv |
Authors (GPT-4 experiments) | 40,000 observations of GPT-4 playing prisoner's dilemma | Yes |
data/all_apps_wide-2023-08-11.csv |
Authors (human experiments) | 50 participants (Aug 11 session), wide format | Yes |
data/all_apps_wide-2023-08-15.csv |
Authors (human experiments) | 73 participants (Aug 15 session), wide format | Yes |
data/all_apps_wide-2023-08-16.csv |
Authors (human experiments) | 163 participants (Aug 16 session), wide format | Yes |
data/all_apps_wide-2023-08-17.csv |
Authors (human experiments) | 100 participants (Aug 17 session), wide format | Yes |
Data integrity verification: The file data/SHA256SUMS.asc contains SHA-256 checksums for all data files, digitally signed with GnuPG. This allows replicators to verify that the data files have not been corrupted or modified. To verify data integrity on Linux/macOS, run:
cd data && gpg --quiet --decrypt SHA256SUMS.asc | sha256sum -cWindows users can use equivalent tools.
- The replication package contains one or more programs to install all dependencies and set up the necessary directory structure.
The analysis code requires R and several packages. The code was last run successfully with R version 4.5.0.
Required R packages (with versions used):
dplyr(1.1.4): Data manipulationtidyr(1.3.1): Data reshaping (pivot functions)ggplot2(4.0.1): PlottingCairo(1.7.0): PDF output for plotspatchwork(1.3.2): Combining multiple plotslmtest(0.9.40): Testing linear regression modelssandwich(3.1.1): Robust covariance matrix estimationtexreg(1.39.3): Regression table formatting for LaTeX (custom fork required, see installation below)plm(2.6.7): Panel data modelsremotes(2.5.0): Required for installing texreg from GitHub
Installation:
To install all required packages automatically, run the provided installation script:
source("install_dependencies.R")Note: The installation script installs the latest versions from CRAN, which may be newer than the versions listed above. The versions listed above are those used by the authors and are known to work correctly. For exact reproducibility, you may need to install the specific versions listed. A complete R session information file (sessionInfo.txt) is provided in the repository for reference.
Alternatively, you can install packages manually:
# Install from CRAN
install.packages(c("dplyr", "tidyr", "ggplot2", "Cairo", "patchwork",
"lmtest", "sandwich", "plm"))
# Install texreg from custom fork (required)
install.packages("remotes")
remotes::install_github("mrpg/texreg_fork", ref = "6268c7f")Additional software (optional):
- The
ego_human/directory contains an oTree application used to collect the human-GPT-4 experimental data. This is provided for documentation purposes and is not needed to replicate the analysis. Running this application would require:- Python 3.x
- oTree package
- The alter_ego package for AI integration
- No pseudo-random number generator is used in the analysis code (plots.R and tables.R).
Note: The data generation process (GPT-GPT experiments and oTree experiments) involved randomness, but the data files provided are fixed and the analysis code processes these fixed datasets deterministically.
Approximate time needed to reproduce the analyses on a standard (2025) desktop machine:
- 10-60 minutes
Running plots.R takes approximately 5-15 minutes (the script processes ~86,000 rows and includes a progress counter).
Running tables.R takes approximately 2-5 minutes.
Approximate storage space needed:
- < 25 MBytes
The entire repository is approximately 12 MB (including data files of ~11 MB).
The code was last run successfully on:
- R version: 4.5.0 (2025-04-11)
- Platform:
x86_64-pc-linux-gnu - Operating System: Debian GNU/Linux 13 (trixie)
- BLAS/LAPACK: OpenBLAS 0.3.29
The code can run on any standard desktop or laptop machine with R installed. No special hardware requirements.
Minimum requirements:
- 2 GB RAM
- Any modern CPU
- ~25 MB disk space
- Any operating system that runs R (Windows, macOS, Linux)
plots.R
- Generates all plots/figures for the paper
- Sources helper libraries from
lib/preload.Randlib/data.R - Prints Table 3 summary statistics to console (Mean Percentage of Cooperative Choices per Platform and Frame)
- Creates the following plots in
output/:plot1.pdf: Cooperation conditional on platform and round (Figure 3 in paper)plot2.pdf: Cooperation conditional on platform, round, and identity of the player (Figure 4 in paper)plotA1.pdf: Cooperation conditional on platform, round, identity of the player and second mover's choice in first round (Figure A1 in appendix)plotA2.pdf: GPT-3.5 across all treatments (Figure A2 in appendix)plotA3.pdf: GPT-4 across all treatments (Figure A3 in appendix)
tables.R
- Generates all regression tables for the paper
- Sources helper libraries from
lib/preload.Randlib/data.R - Creates the following LaTeX tables in
output/:rt1.tex: Framing effects in GPT-GPT interaction (Table A2)rt2.tex: Effect of second mover defection on first mover cooperation (Table A3)rt3.tex: Framing effects in GPT-human interaction (Table A4)rt4.tex: Framing effects of additional frames in GPT-GPT interaction (Table A5)rt5.tex: Effect of competitive frame in GPT-human interaction (Table A6)rt6.tex: Cooperation by treatment and dyad composition (Table A7)
tables2.R
- Computes lower and upper bounds of cooperation rates that binomial tests reject at the 5% level
- Sources helper libraries from
lib/preload.Randlib/data.R - Defines a helper function
compute_bounds()to systematically compute rejection bounds - Prints results to console for Table A1 (bounds per platform, frame, and round):
- GPT-3.5 bounds (base, enemy, competition treatments)
- GPT-4 bounds (base, enemy, competition treatments)
- Human bounds (base, enemy, competition treatments)
- Uses binomial tests with alternatives "greater" (for lower bounds) and "less" (for upper bounds)
- For each platform-frame-round combination, reports:
- Lower bound: lowest cooperation rate that cannot be rejected
- Upper bound: highest cooperation rate that cannot be rejected
- Sample size and observed cooperation rate
lib/data.R
- Loads and processes all data files
- Defines key functions:
augment(): Adds timing information to conversationstreatments(): Converts treatment codes to factor labelsto_long(): Converts wide-format human data to long format
- Computes derived variables:
coop: Binary cooperation indicator (choice == 1)second: Indicator for second mover in conversationldef: Whether opponent last defectedlsdef: Sum of opponent defections so far
- Combines all data sources into unified datasets
lib/preload.R
- Loads required packages
- Defines helper functions for regression analysis:
tt(): Formats regression tables for LaTeX outputsave_to(): Saves multiple objects to filesrobustly(): Computes robust standard errors (HC3)standard(): Extracts standard errorsclustered(): Computes cluster-robust standard errorsci(): Calculates confidence intervals using proportion tests
ego_human/
- oTree application used to collect human-GPT-4 experimental data
- Contains:
__init__.py: Main oTree app logic (treatment assignment, game flow, payoff calculations)- HTML templates for participant interface
prompts/: Experimental instructions and prompts for different treatments
- This is provided for documentation purposes; it is not needed to replicate the analysis
Unless otherwise specified below, all files in this repository are licensed under GNU General Public License v3.0 (GPL-3.0) due to GPL-licensed R package dependencies. See LICENSE-CODE for full license text.
The following files are dual-licensed under GPL-3.0 AND CC0 1.0 (you may choose either license):
*.csv,*.md,*.txt,*.pdf,*.tex,*.ascego_human/*.py,ego_human/*.html
See LICENSE (CC0) and LICENSE-CODE (GPL-3.0) for full legal texts.
-
Install R and required packages:
- Install R (version 4.x or later recommended)
- Run the installation script to install all required packages:
source("install_dependencies.R") - Alternatively, see "Software Requirements" section above for manual installation commands
-
Clone or download this repository:
- All data files are included in the
data/directory - No additional data download is necessary
- All data files are included in the
-
Set working directory:
- In R, set your working directory to the root of this repository:
setwd("/path/to/ego_paper_data")
- In R, set your working directory to the root of this repository:
-
Generate all figures:
- Run the plots script:
source("plots.R") - This will take 5-15 minutes and create PDF files in the
output/directory - Progress will be displayed as the script processes approximately 86,000 rows
- Run the plots script:
-
Generate all tables:
- Run the main tables script:
source("tables.R") - This will take 2-5 minutes and create LaTeX table files in the
output/directory - Optionally, run the bounds analysis script:
source("tables2.R") - This will take 1-2 minutes and print Table A1 (cooperation rate bounds) to the console
- Run the main tables script:
-
Check output:
- All figures (PDF format) should appear in
output/: plot1.pdf, plot2.pdf, plotA1.pdf, plotA2.pdf, plotA3.pdf - All tables (LaTeX format) should appear in
output/: rt1.tex through rt6.tex - Table A1 bounds will be printed to the console when running tables2.R
- All figures (PDF format) should appear in
- The scripts can be run in any order (plots.R, tables.R, and tables2.R are independent)
- All scripts will display progress messages during execution
- The
lib/data.Rscript displays "Please wait until the counter reaches ~86000" while computing derived variables plots.Rprints Table 3 summary statistics to the console during executiontables2.Rprints Table A1 bounds analysis to the console- Pre-computed output files are already included in the
output/directory for comparison
The provided code reproduces:
- All tables and figures in the paper
| Figure/Table # | Program | Line Range | Output File | Notes |
|---|---|---|---|---|
| Table 3 | plots.R | 18-26 | Console output | Mean percentage of cooperative choices per platform and frame |
| Figure 3 | plots.R | 28-37 | output/plot1.pdf | Cooperation conditional on platform and round |
| Figure 4 | plots.R | 42-94 | output/plot2.pdf | Cooperation conditional on platform, round, and identity of player |
| Figure A1 | plots.R | 96-161 | output/plotA1.pdf | Cooperation conditional on platform, round, identity and initial choice |
| Figure A2 | plots.R | 164-181 | output/plotA2.pdf | GPT-3.5 across all treatments |
| Figure A3 | plots.R | 184-201 | output/plotA3.pdf | GPT-4 across all treatments |
| Table # | Program | Line Range | Output File | Notes |
|---|---|---|---|---|
| Table A1 | tables2.R | Full script | Console output | Lower and upper bounds of cooperation rates per platform, frame, and round |
| Table A2 | tables.R | 6-40 | output/rt1.tex | Framing effect in GPT-GPT interaction |
| Table A3 | tables.R | 43-85 | output/rt2.tex | Effect of second mover defection on first mover cooperation |
| Table A4 | tables.R | 87-118 | output/rt3.tex | Framing effect in GPT-human interaction |
| Table A5 | tables.R | 120-156 | output/rt4.tex | Framing effects of additional frames in GPT-GPT interaction |
| Table A6 | tables.R | 159-185 | output/rt5.tex | Effect of competitive frame in GPT-human interaction |
| Table A7 | tables.R | 187-217 | output/rt6.tex | Cooperation by treatment and dyad composition |
No external data sources or references are cited in this README. All data were generated by the authors as described above.
ego_paper_data/
├── README.md # This file
├── LICENSE # CC0 1.0 Universal license (for data/documentation)
├── LICENSE-CODE # GPL-3.0 license (for R code)
├── install_dependencies.R # Script to install all required R packages
├── sessionInfo.txt # R session information with package versions
├── data/ # Raw experimental data (CSV format)
│ ├── data35_new.csv # GPT-3.5 experimental data
│ ├── data4_new.csv # GPT-4 experimental data
│ ├── all_apps_wide-2023-08-11.csv # Human-GPT-4 data (session 1)
│ ├── all_apps_wide-2023-08-15.csv # Human-GPT-4 data (session 2)
│ ├── all_apps_wide-2023-08-16.csv # Human-GPT-4 data (session 3)
│ ├── all_apps_wide-2023-08-17.csv # Human-GPT-4 data (session 4)
│ └── SHA256SUMS.asc # Checksums for data files
├── lib/ # Library functions for analysis
│ ├── data.R # Data loading and processing
│ └── preload.R # Helper functions and package loading
├── plots.R # Script to generate all figures
├── tables.R # Script to generate all tables
├── tables2.R # Script to compute cooperation rate bounds (Table A1)
├── output/ # Generated output files
│ ├── plot1.pdf # Figure 3
│ ├── plot2.pdf # Figure 4
│ ├── plotA1.pdf # Appendix Figure A1
│ ├── plotA2.pdf # Appendix Figure A2
│ ├── plotA3.pdf # Appendix Figure A3
│ ├── rt1.tex # Table A2
│ ├── rt2.tex # Table A3
│ ├── rt3.tex # Table A4
│ ├── rt4.tex # Table A5
│ ├── rt5.tex # Table A6
│ └── rt6.tex # Table A7
└── ego_human/ # oTree experimental application (documentation)
├── __init__.py # Main oTree app logic
├── *.html # Participant interface templates
└── prompts/ # Experimental instructions and prompts
├── prompts001/ # Treatment 1 (base)
├── prompts003/ # Treatment 3 (enemy)
└── prompts004/ # Treatment 4 (competition)
This README template is based on the template provided by the Social Science Data Editors, adapted for this specific replication package.