Jiarui Zhang^*, Muzi Tao^*, Shangshang Wang^*, Ollie Liu, Xuezhe Ma, Willie Neiswanger
University of Southern California
^*Equal contribution

ActiveVision is an benchmark with 17 tasks for iterative visual reasoning — frontier models score ~10% versus 96.1% for humans.

Paper · Website · Dataset

---

Example

Repository layout

ActiveVision/
  README.md
  requirements.txt
  assets/                   # figures used in this README
  data/                     # place the dataset here (see data/README.md)
  eval/
    README.md               # evaluation docs
    eval_claude.py          # main evaluation
    eval_gpt.py
    eval_gemini.py
    lib/                    # shared scoring / dataset / runner
      scoring.py
      dataset.py
      runner.py
      cli.py
      agent.py              # helpers for the optional agent ablation
    agent/                  # application: agent tool-use ablation
      claude_code/
        run_claude_code.py
        README.md
      codex/
        run_codex.py
        README.md

Setup

1. Install the Python dependencies:

   pip install -r requirements.txt

   You only need the SDK for the provider(s) you evaluate.

2. Download the dataset into data/ (see data/README.md for the layout and download link).

3. Set the API key for each provider you use:

   export ANTHROPIC_API_KEY=sk-ant-...   # Claude (CoT and Claude Code agent)
   export OPENAI_API_KEY=sk-...          # GPT (CoT) and Codex agent
   export GEMINI_API_KEY=...             # Gemini (CoT)

Running an evaluation

Main evaluation

This is the official ActiveVision protocol reported in the paper (no tools):

python eval/eval_claude.py --model claude-fable-5         --effort max
python eval/eval_claude.py --model claude-opus-4-8        --effort max
python eval/eval_gpt.py    --model gpt-5.5                --reasoning-effort xhigh
python eval/eval_gemini.py --model gemini-3.1-pro-preview --thinking-level high
python eval/eval_gemini.py --model gemini-3.5-flash       --thinking-level high

Model	Flag	none	minimal	low	medium	high	xhigh	max	Default
Claude Fable 5	--effort			✓	✓	✓	✓	✓	xhigh
Claude Opus 4.8	--effort			✓	✓	✓	✓	✓	max
GPT 5.5	--reasoning-effort	✓		✓	✓	✓	✓		xhigh
Gemini 3.1 Pro	--thinking-level			✓	✓	✓			high
Gemini 3.5 Flash	--thinking-level		✓	✓	✓	✓			high

See eval/README.md for all options (task selection, concurrency, resuming, quick tests).

Application: agent tool use

Optional ablation: give the model a coding agent that can inspect the image with code. Requires the corresponding CLI installed and authenticated:

# Claude Code  (https://docs.claude.com/en/docs/claude-code)
python eval/agent/claude_code/run_claude_code.py --model claude-fable-5 --effort xhigh

# Codex CLI    (https://developers.openai.com/codex/cli)
python eval/agent/codex/run_codex.py --model gpt-5.5 --effort xhigh

See the READMEs in each agent directory for details.

Model ids. The --model flags above are examples. Pass any model id your API key can access. CoT defaults: Claude / GPT xhigh, Gemini high. All CoT runners require the matching API key env var.

Results

Pure chain-of-thought accuracy vs. API cost per item (log scale) on the 85-item split; marker size encodes the reasoning-effort tier. Every model stays in a narrow low band at every effort level — more reasoning moves a model along the cost axis, not toward the human band.

Agent evaluation: the strongest tool-using coding agent reaches 50.6%.

Output

Every run writes results under the chosen --out-dir (default results/<model>):

results/<model>/
  <task>/items.jsonl     # one graded record per item
  <task>/summary.json    # per-task accuracy
  rollup.json            # all tasks combined

Each record in items.jsonl contains the item id, task, ground truth, the extracted prediction, a correct flag, the model's answer text, token usage (CoT only), elapsed time, and any error. A final accuracy table is printed at the end of each run.

Runs are resumable: items already present in items.jsonl are skipped on a re-run. Pass --redo to start over.

Scoring

Grading lives in eval/lib/scoring.py and is identical for every track:

1. Extract the model's final answer — the contents of the last <answer>...</answer> block (every question instructs the model to use these tags).
2. Normalise separator-insensitively: upper-case and keep only alphanumerics, so "C, A, G", "C A G" and "CAG" are equal.
3. Compare by exact match, with an integer fallback ("07" matches 7).

Citation

@article{zhang2026exam,
  title={An exam for active observers},
  author={Zhang, Jiarui and Tao, Muzi and Wang, Shangshang and Liu, Ollie and Ma, Xuezhe and Neiswanger, Willie},
  journal={arXiv preprint arXiv:2607.16165},
  year={2026}
}