MathGen: Revealing the Illusion of Mathematical Competence through Text-to-Image Generation

⭐ Introduction

Text-to-image models often produce images that look visually plausible while violating precise mathematical constraints. MathGen evaluates this gap by asking models to generate images for prompt-level mathematical tasks, then checking each output with deterministic, prompt-conditioned verification scripts.

The benchmark covers 420 problems across seven core mathematical domains: counting, angle, fraction, function, plane geometry, set, and solid geometry. It includes a 350-problem Clean-Scene set and a 70-problem Open-Scene set with paired mathematical constraints. Each evaluator checks the mathematical relation specified by the prompt, such as exact object counts, angle measures, set membership, geometric relations, filled fractions, or functional shape behavior.

📊 Benchmark Overview

MathGen is organized around seven clean-scene mathematical domains, with an additional open-scene split for evaluating the same kinds of reasoning under more realistic visual conditions.

⚙️ Installation

git clone https://github.com/Liuruiyao/mathgen.git
cd mathgen

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

🛠️ Usage

1. Generate Images

Dry-run a generation job without calling an external API:

python generate.py --backend dry-run --topics angle --model gpt-image-1 --limit 5

Generate images with OpenAI:

export OPENAI_API_KEY="..."
python generate.py --backend openai --model gpt-image-1 --topics angle fraction

Generate images with Replicate:

export REPLICATE_API_TOKEN="..."
python generate.py \
  --backend replicate \
  --model seedream-4 \
  --replicate-model bytedance/seedream-4 \
  --topics angle

By default, MathGen appends shared prompt constraints from utils/math_global_config.py. Use --no-prompt-constraints to send only the raw benchmark prompt.

2. Run Evaluation

Evaluate generated images from one or more models:

python evaluate.py \
  --topics angle fraction set \
  --models gpt-image-1 \
  --workers 8 \
  --timeout 20

Results are written to results/current/evaluation/:

• per_question_model_results.csv
• per_question_model_results.jsonl
• model_topic_summary.csv
• topic_summary.csv

The lower-level evaluator runner is also available:

python utils/run_evaluation.py --topics angle --models gpt-image-1

3. Run Generation and Evaluation

run_benchmark.py provides a single entrypoint for the full benchmark loop: it first generates images into data/generated_img/<model>/<topic>/, then runs the deterministic evaluators and writes summaries to results/current/evaluation/.

For a quick smoke test without calling an external image-generation API:

python run_benchmark.py \
  --backend dry-run \
  --model test-dry-run \
  --topics angle \
  --limit 5 \
  --workers 4

For a real generation-and-evaluation run:

python run_benchmark.py \
  --backend openai \
  --model gpt-image-1 \
  --topics angle \
  --workers 8

If images have already been generated, skip the generation stage and evaluate the existing files:

python run_benchmark.py \
  --skip-generate \
  --model gpt-image-1 \
  --topics angle fraction set \
  --workers 8

🏆 Leaderboard

Main results on the 350-problem Clean-Scene set of MathGen. We report accuracy across seven mathematical domains, with 50 problems per domain. The best result in each column is shown in bold, and the second-best result is shown with underline.

Model	Counting	Angle	Fraction	Function	Plane	Set	Solid	Overall
Diffusion Models
SD-3-Medium	0.0	0.0	0.0	0.0	6.0	0.0	6.0	1.7
SD-3.5-Medium	4.0	0.0	0.0	0.0	12.0	2.0	6.0	3.4
SD-3.5-Large	12.0	0.0	2.0	0.0	12.0	2.0	4.0	4.6
FLUX-2	8.0	2.0	8.0	2.0	42.0	8.0	8.0	11.1
PixArt-Sigma	10.0	0.0	2.0	0.0	12.0	2.0	8.0	4.9
PixArt-XL-2	0.0	0.0	0.0	0.0	14.0	0.0	4.0	2.6
HiDream-I1	6.0	0.0	0.0	2.0	4.0	2.0	6.0	2.9
Qwen-Image	22.0	0.0	8.0	2.0	24.0	4.0	6.0	9.4
Z-Image-Turbo	8.0	0.0	8.0	0.0	16.0	2.0	14.0	6.9
Autoregressive Models
Infinity-8B	6.0	0.0	4.0	0.0	18.0	2.0	8.0	5.4
GoT-R1-7B	8.0	0.0	0.0	0.0	16.0	2.0	2.0	4.0
Unified Models
BAGEL	4.0	0.0	0.0	0.0	14.0	0.0	2.0	2.9
show-o2-1.5B	0.0	0.0	0.0	4.0	18.0	0.0	2.0	3.4
show-o2-7B	0.0	0.0	0.0	4.0	10.0	0.0	8.0	3.1
Janus-Pro-1B	0.0	0.0	0.0	0.0	14.0	0.0	2.0	2.3
Janus-Pro-7B	0.0	0.0	0.0	0.0	12.0	2.0	6.0	2.9
BLIP3o-4B	4.0	0.0	2.0	0.0	14.0	2.0	8.0	4.3
BLIP3o-8B	6.0	0.0	2.0	0.0	14.0	2.0	4.0	4.0
OmniGen2-7B	4.0	0.0	2.0	0.0	12.0	2.0	8.0	4.0
Closed-Source Models
FLUX-2-Pro	22.0	10.0	20.0	18.0	54.0	16.0	20.0	22.9
FLUX-Kontext-Pro	10.0	0.0	10.0	4.0	18.0	6.0	4.0	7.4
Seedream 3.0	14.0	0.0	2.0	0.0	24.0	6.0	8.0	7.7
Seedream 4.0	20.0	0.0	6.0	10.0	36.0	6.0	14.0	13.1
Ideogram v3 Turbo	10.0	2.0	0.0	0.0	20.0	2.0	8.0	6.0
Nano Banana	20.0	8.0	24.0	10.0	64.0	10.0	24.0	22.9
Nano Banana Pro	48.0	54.0	50.0	70.0	72.0	42.0	40.0	53.7
Imagen 4	12.0	2.0	2.0	2.0	12.0	0.0	12.0	6.0
Imagen 4 Ultra	20.0	6.0	16.0	4.0	42.0	8.0	16.0	16.0
GPT-Image-1	32.0	12.0	44.0	16.0	68.0	20.0	24.0	30.9
GPT-Image-1.5	56.0	24.0	54.0	22.0	70.0	20.0	28.0	39.1

🖼️ Examples

The following examples illustrate MathGen prompts and generations across mathematical domains. They show that visually plausible images can still fail precise mathematical constraints, motivating deterministic script-based evaluation.

📄 License

This project is released under the MIT License.

📝 Citation

If you find MathGen helpful, please consider citing our paper:

@misc{liu2026mathgen,
  title={MathGen: Revealing the Illusion of Mathematical Competence through Text-to-Image Generation},
  author={Liu, Ruiyao and Shen, Hui and Zhang, Ping and Hsieh, Yunta and Zhang, Yifan and Xu, Jing and Han, Qi and Li, Junchen and Lu, Jiawei and Ma, Jianing and Mo, Jiaqi and Chen, Sicheng and Zhang, Zhen and Wan, Zhongwei and Xiong, Jing and Wang, Xin and Liu, Ziyuan and Cao, Hangrui and Wong, Ngai},
  year={2026},
  eprint={2603.27959},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2603.27959}
}