Mirror

Simulating individual humans and partisan crowds, evaluated honestly.

Mirror asks a simple question: can large language model persona agents actually simulate a specific real person's survey answers better than a good statistical model? Across GSS panel data and a replication of the Becker, Porter, and Centola (2019) partisan crowds experiment, the answer is nuanced. A compact amortized person embedding model beats LLM personas at predicting individuals (accuracy .623 vs .553 to .591 for personas), while LLMs win in two specific regimes: answering zero shot questions that have no training data, and propagating social influence inside a crowd. Everything is measured against pre registered bars with panel retest ceilings, and negative results are kept.

Headline results

Individual response prediction (EVAL leaderboard, 19,836 pairs, 400 persons; retest ceiling .657)

method	acc [95% CI]	norm.	lift vs B0r	NLL	ECE
Mirror (K=5, e30, d64)	.623 [.616,.631]	.950	+.077	.838	.006
B1 demographics logistic	.589 [.582,.596]	.897	+.042	.909	.015
B2 kNN CF	.586 [.577,.593]	.892	+.039	.908	.014
B3 matrix factorization	.584 [.576,.591]	.889	+.037	.935	.008
B0r marginals (2010+)	.547 [.540,.554]	.833	ref	.972	.016

The amortized person model clearly beats every statistical baseline and is near perfectly calibrated (ECE .006). Mirror vs B1 McNemar p=3e-31.

LLM persona agents on the same pairs

method	acc	vs same-pairs Mirror	ECE
L1 haiku	.553	−.073	.075
L1m haiku (+population marginals in prompt)	.579	−.047	.041
L1 sonnet-5 (premium)	.591	−.032	.040
L2 hybrid Mirror+L1m (blend w=.10, DEV-fit)	.629	+.003 (n.s.)	.017

Persona agents never beat the statistical person model on identical information: haiku scale personas score below recent population marginals, and premium sonnet-5 only reaches the level of a ten field demographic regression. LLM value in human simulation is question understanding, not person understanding. Full detail, including where LLMs do win (zero data questions, and propagating social influence in crowds where they beat a fitted DeGroot model), is in FINDINGS.md.

Repo layout

• src/ : data pipeline, eval harness, models (matrix factorization, Mirror), persona prompts, crowds analysis and simulation, DeGroot baselines, active learning.
• scripts/ : runnable experiment entry points, one per protocol stage.
• results/ : run artifacts (JSON plus per pair parquet per run; every number in FINDINGS.md cites one), figures under results/figs/, and the frozen holdout spec results/holdout_spec.json.
• FINDINGS.md : final results, the scientific record.
• EVALS.md : the frozen, pre registered evaluation protocol.
• data/ : fetch script and small item metadata only. Raw GSS files and large processed tables are not distributed (see Data below).

Setup

Requires Python 3.12.

uv venv --python 3.12 .venv
uv pip install -r requirements.txt

Plain pip install -r requirements.txt inside a 3.12 virtual environment works too. Neural training runs on CUDA, MPS, or CPU automatically (src/model.py).

Data

None of the survey data is redistributed here. You need two sources.

1. GSS data from NORC at gss.norc.org. Download the cumulative cross sectional file gss7224_r3a.dta and the panel file gss2020panel_r1a.dta, and place both under data/. The helper bash data/download.sh automates fetching the NORC Stata archives.
2. The Becker, Porter, and Centola (2019) "Wisdom of Partisan Crowds" replication data (the trial level CSV) from the authors' OSF repository. The same data/download.sh clones their public replication repo.

After the raw files are in place, build the processed tables:

.venv/bin/python src/gss_data.py     # parse raw -> data/processed/

Reproduce

.venv/bin/python scripts/freeze_holdout.py                 # frozen eval spec (seeded)
.venv/bin/python scripts/run_baselines.py b0|b1|b2 {dev|eval}
.venv/bin/python scripts/run_b3.py {dev|eval}              # matrix factorization
.venv/bin/python scripts/run_mirror.py {dev|eval} 5 30 64  # Mirror ensemble
.venv/bin/python scripts/run_l1.py eval <model> <n> [chunk] [marg]   # LLM personas
.venv/bin/python scripts/run_l2_hybrid.py {dev|eval} <tag>
.venv/bin/python scripts/run_t1b_unseen.py                 # unseen-question experiment
.venv/bin/python scripts/run_task2.py {degroot|gen|dyn} [model]      # crowds
.venv/bin/python scripts/run_task3.py {fixedsel|curves}    # active learning
.venv/bin/python scripts/make_tables.py
.venv/bin/python scripts/make_figs.py

Run scripts/freeze_holdout.py first: it fixes the seeded evaluation split that every later stage reads. LLM calls go through the claude CLI and are cached in results/cache/llm keyed by prompt hash, with per call cost recorded.

Known issues and portability notes

• The open weight crowd runs (Qwen3-32B via vLLM) were executed on a 4xA100 SLURM cluster. scripts/leonardo_job.sh and scripts/leonardo_*.py contain cluster specific paths and an allocation account. They are included as a record of the exact run and will need editing for another environment.
• The exact neural checkpoints behind the reported numbers live under results/cache/ (gitignored). Regenerate them with scripts/run_mirror.py.

Paper

See paper/mirror.pdf.

Citation

The paper is archived as a preprint on Zenodo, 10.5281/zenodo.21500169 (CC-BY-4.0). The code repository has its own DOI, 10.5281/zenodo.21499001 (MIT).

@misc{aslam2026mirror,
  author = {Aslam, Umar},
  title  = {Mirror: What LLM Personas Can and Cannot Simulate},
  year   = {2026},
  doi    = {10.5281/zenodo.21500169},
  url    = {https://doi.org/10.5281/zenodo.21500169},
  note   = {Code: https://github.com/Umaraslam66/Mirror}
}

License

MIT. See LICENSE.