A benchmark for visually conditioned action on complete multi-object 3D scenes, under a unified agent–environment loop.
- About
- Tasks
- Leaderboard
- Dataset
- Getting Started
- Repository Layout
- License
- Citation
- Acknowledgements
- Contributors
- [2026] SceneActBench is released — code, dataset, and an 11-configuration leaderboard are now public.
- [2026] The dataset is available on Hugging Face.
Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under-evaluated.
SceneActBench evaluates whether an agent can convert visual evidence into executable 3D outputs across five tasks under one fixed agent loop. Given PNG images or sampled video frames (and, where applicable, supplied 3D assets), the agent controls Blender in headless mode through a shared MCP tool interface and produces JSON or GLB outputs. Each final output is scored against hidden 3D ground truth with task-specific geometric metrics.
Contributions.
- We formulate visually conditioned 3D action as an executable evaluation problem, scoring final outputs against hidden 3D ground truth (not text).
- We introduce five tasks built from 210 source instances (yielding 520 task cases), each probing a distinct 3D capability, all under one fixed agent loop.
- We benchmark 11 proprietary VLM configurations and use case-level diagnostics to characterise where and how failures manifest beyond aggregate scores.
Each task targets one capability:
| Capability | Task |
|---|---|
| Spatial grounding | Layout |
| Egocentric spatial reasoning | Camera |
| Kinematic reasoning | Articulated |
| Shape imagination | Reconstruction |
| Dynamic reasoning | Dynamic |
Across the 11 configurations, Overall scores span 38.6–50.2 — no model performs consistently well across all tasks.
Five tasks, each testing a different visually-conditioned action.
T1 · Layout — place and orient standardised furniture to match room images. Metric: ADD-S (m) ↓
The reference (leftmost) and the layouts produced by different models.
T2 · Camera — recover a camera pose from reference views. Metric: position / angle error ↓
The reference view (leftmost) and the views rendered from each model's recovered camera.
T3 · Articulated — animate articulated parts (doors, drawers, ...). Metric: mean part error ↓
Reference open–close frames (top) vs. the agent's reproduction (bottom).
T4 · Reconstruction — rebuild a room scene from multi-view references. Metric: F@5% ↑
Reference multi-view images (top) vs. the agent's reconstruction rendered from the same views (bottom).
T5 · Dynamic — produce a dynamic animation of a scene. Metric: motion / layout error ↓
Reference animation frames (top) vs. the agent's animation over time (bottom).
Per-task Score (0–100, higher is better; a fixed normalised summary of each task's native metric) and the Overall average across the five tasks.
| # | Configuration | Layout | Camera | Articulated | Reconstruction | Dynamic | Overall |
|---|---|---|---|---|---|---|---|
| 🥇 | Doubao Seed 2.0 Pro High | 77.4 | 34.5 | 59.6 | 8.8 | 70.7 | 50.2 |
| 🥈 | Claude Opus 4.6 High | 73.5 | 34.0 | 63.7 | 9.8 | 63.2 | 48.9 |
| 🥉 | GPT 5.4 Medium | 72.7 | 29.6 | 62.3 | 10.4 | 68.5 | 48.7 |
| 3 | GPT 5.4 High | 84.1 | 26.4 | 73.8 | 12.3 | 46.7 | 48.7 |
| 5 | Qwen 3.7 Plus High | 76.2 | 21.7 | 58.0 | 9.0 | 66.0 | 46.2 |
| 6 | Gemini 3.1 Pro High | 65.4 | 33.9 | 56.5 | 7.1 | 63.9 | 45.4 |
| 7 | MiMo 2.5 High | 79.4 | 25.0 | 49.8 | 9.1 | 43.7 | 41.4 |
| 8 | Kimi K2.6 Reason | 70.9 | 24.6 | 57.3 | 8.5 | 44.8 | 41.2 |
| 9 | Step 3.7 Flash High | 77.5 | 13.2 | 48.3 | 8.6 | 57.9 | 41.1 |
| 10 | Claude Sonnet 5 High | 51.9 | 27.3 | 57.8 | 10.5 | 49.9 | 39.5 |
| 11 | MiniMax M3 High | 58.1 | 25.6 | 58.4 | 9.6 | 41.4 | 38.6 |
Notation. Overall is a fixed normalised summary for compact comparison, not evidence that ranks are statistically distinct; the paper reports native metrics alongside it. Native metrics per task: ADD-S (surface distance, m) for Layout; PE/AE (position error m / angle error °) for Camera; MPE (mean part error) for Articulated; F@5% (fraction of surface within 5% tolerance) for Reconstruction; MME/LE (motion-matching error / layout error) for Dynamic.
Takeaways. Reconstruction is hard for every model (best score 12.3). No configuration leads on all tasks: GPT 5.4 High tops Layout / Articulated / Reconstruction but drops on Dynamic, while Doubao Seed 2.0 Pro High wins Dynamic and Overall. Reasoning-heavy configurations do not translate into a uniform advantage.
Assembled from three public sources, each standardised into a shared format (assets are centred / re-oriented / anonymised so the answer is not leaked in coordinates or names):
| Source | Contribution |
|---|---|
| 3D-FRONT / M3DLayout | 100 furnished rooms (3–7 objects, 27 categories), 11 rendered views each |
| S2O ACD | 100 articulated objects, 32-frame open–close video + per-frame GT mesh |
| Kenney (CC0) | 10 dynamic scenes, 144-frame reference video + animated GT scene |
huggingface-cli download FEInaldo/SceneActBench --repo-type dataset --local-dir ./dataThe dataset is organised as:
data/
benchmark_final/ # indoor scenes (Layout / Camera / Reconstruction)
benchmark_s2o_final/ # articulated objects (Articulated)
benchmark_t6_final/ # dynamic scenes (Dynamic)
Each task case is a JSON file with these fields:
| Field | Description |
|---|---|
id |
Unique task-case identifier |
task_type |
One of task1_layout / task3_camera / task4_anim / task5_recon / task6_anim |
scene_id |
Source scene / object identifier |
scene_dir |
Path to the scene's assets and ground truth |
system_prompt |
System prompt given to the agent |
prompt |
Task instruction |
setup_code |
Blender code that initialises the clean scene (imports assets) |
references |
Reference images / video frames the agent conditions on |
Ground truth is kept separate from the task file and never exposed to the agent.
pip install -r requirements.txt
cp .env.example .env # fill in API keys and DATA_ROOTModel credentials are read from environment variables referenced in
models.json (e.g. ANTHROPIC_API_KEY, gateway credentials); nothing secret is
stored in the repo. DATA_ROOT points to the downloaded dataset.
Run a single task, or every task in a directory:
# one task
python src/harness/run.py --models models.json --task tasks/<task>.json \
--headless --auto-port
# a whole task type
python src/harness/run.py --models models.json --all --tasks-dir tasks/t5 \
--headless --auto-port --checkpoint-every 10Outputs land under runs/<model>/<task_type>/<task_id>/:
steps.json # full agent trajectory (every tool call)
score.json # final score + headline metric
agent_scene.glb # exported scene (Layout / Reconstruction / Dynamic)
agent_camera.json # recovered camera (Camera)
curve_steps/ # per-checkpoint intermediate products + scores
runs/ is git-ignored.
src/harness/
run.py # entrypoint: load models + tasks, run each, score
agent.py # model-agnostic agent loop
adapters.py # provider adapters (OpenAI-compatible / Anthropic / Gemini)
metrics*.py # per-task scoring (ADD-S, camera pose, F@5%, ...)
run_io.py # output layout + summary rebuild
mcp_client.py # Blender MCP client
blender-mcp/ # Blender MCP server + addon
run_benchmark.sh # convenience wrapper
models.json # model configs (credentials via env vars)
The agent talks to Blender through the MCP server in blender-mcp/. Use
--headless --auto-port to let the harness launch headless Blender per worker.
This repository (the evaluation harness) is released under the MIT License. It bundles blender-mcp (also MIT). Dataset assets are distributed separately on Hugging Face under the licenses of their original sources (3D-FRONT, S2O ACD, and Kenney CC0 kits).
@misc{zhao2026sceneactbench,
title = {SceneActBench: Can Agents Act on the 3D Scenes They See?},
author = {Zhao, Yifei and Zhou, Xiangxin and Yang, Wenhao and Tang, Jiaqi
and Jian, Pu and Yao, Huanjin and Yao, Jiarui and Lin, Haowei
and Chen, Zhuo and Lyu, Wenkai and Ma, Jianzhu and Wang, Xueqian
and Zhu, Wenxi and Pang, Tianyu},
year = {2026},
note = {Tencent Hunyuan},
url = {https://github.com/Feinaldo2/SceneActBench}
}Built on data from 3D-FRONT / M3DLayout, S2O ACD, and Kenney asset kits, with photo-realistic dynamic references from NVIDIA Cosmos. Blender control uses the blender-mcp interface.
Yifei Zhao, Xiangxin Zhou, Wenhao Yang, Jiaqi Tang, Pu Jian, Huanjin Yao, Jiarui Yao, Haowei Lin, Zhuo Chen, Wenkai Lyu, Jianzhu Ma, Xueqian Wang, Wenxi Zhu, Tianyu Pang.
Tencent Hunyuan · Tsinghua University · Nanjing University · HKUST · UIUC · Peking University