Code as Text vs Dense Images

Measures provider-reported input tokens when source code is sent as plain text or as PNG pages rendered by pxpipe-proxy. The checked-in dataset covers five languages, nine snippet sizes (20–2,000 lines), and three API providers.

Current dataset

• Anthropic: 360/360 successful calls across four models.
• OpenAI: 540/540 successful calls across six models.
• Gemini: 450/450 successful calls across five models.

The CSV column image_ratio is image_input_tokens / text_input_tokens. A ratio below 1 means the image used fewer input tokens. Aggregate savings must be calculated as:

1 - sum(image_input_tokens) / sum(text_input_tokens)

The recorded weighted savings are 86.5% for Anthropic, 80.6% for OpenAI, and 75.8% for Gemini. These are input-token measurements, not price or task-quality measurements. Gemini images cost more than text below 200 lines, and its image accounting changes sharply when a render is split across multiple pages.

Important limitation

The compact representation replaces indentation with markers and enables renderer reflow. Token savings therefore do not prove that a model can recover, understand, edit, or execute the code equally well. Run the separate readable image fidelity study before making coding-harness or research-paper claims. Each language/size is also one nested prefix from one source file (n=1), so the current data is a pilot rather than a representative benchmark.

Requirements

• Python 3.10+ and uv
• Node.js 18+
• API keys for the providers being measured: ANTHROPIC_API_KEY, OPENAI_API_KEY, and/or a Vertex AI express-mode GEMINI_API_KEY

Setup

uv venv
source .venv/bin/activate
uv pip sync requirements.txt
npm ci
cp .env.example .env
# Replace only the keys for providers you intend to run.

Scripts read keys from the process environment first and then from the local, git-ignored .env file.

Run

python scripts/fetch_corpus.py
python scripts/run_rendering.py
python scripts/run_token_measurement.py --resume
python scripts/run_openai_token_measurement.py --resume
python scripts/run_gemini_token_measurement.py --resume

Corpus URLs in config.py are pinned to the revisions used for the checked-in measurements. Rendering replaces each snippet's existing PNG pages. In compact images, each leading > represents four spaces and ^N represents an irregular run of N spaces. Provider scripts checkpoint every result; --resume skips successful calls and retries transient failures. Existing results are never replaced unless --force is passed.

Run local pairing checks without credentials or API calls:

python scripts/check_results.py
python scripts/run_token_measurement.py --self-check
python scripts/run_openai_token_measurement.py --self-check
python scripts/run_gemini_token_measurement.py --self-check
python scripts/run_fidelity_study.py --self-check

Readable-image fidelity study

python scripts/run_fidelity_study.py --prepare-only
python scripts/run_fidelity_study.py

The full study makes 30 paid image calls: three 40-line windows per language with Haiku and Sonnet. It writes raw transcriptions and exact, character, line, identifier-token recall, and aligned-line indentation accuracy to fidelity/results/. Responses that are empty or hit the output-token limit are marked incomplete and excluded from the summary.

Sources

• Python: CPython Lib/asyncio/base_events.py (PSF-2.0)
• JavaScript: Lodash lodash.js (MIT)
• Rust: rustc parser expr.rs (Apache-2.0 OR MIT)
• Go: standard library net/http/server.go (BSD-3-Clause)
• Java: Guava LocalCache.java (Apache-2.0)

Exact source revisions and URLs are recorded in config.py.

License

Project code is licensed under Apache-2.0. Corpus files retain the upstream licenses listed above and are excluded from Git; fetch_corpus.py downloads them directly from the pinned upstream revisions.

Outputs

• corpus/<language>/snippet_<lines>.<ext>: generated, Git-ignored source prefixes
• renders/<language>/snippet_<lines>/page_N.png: generated, Git-ignored PNG pages
• results/<provider>/token_results.json: raw API usage records
• results/<provider>/token_results.csv: complete text/image pairs
• results/<provider>/RESULTS.md: per-language and per-model ratios
• fidelity/results/: optional readable-image fidelity results