Important
Benchmarking Track – Recognized for advancing AI agent evaluation and LLM benchmarking.
Vision-Language Models (VLMs) are known to struggle with spatial reasoning and visual alignment. To help overcome these limitations, we introduce iVISPAR, an interactive multi-modal benchmark designed to evaluate the spatial reasoning capabilities of VLMs acting as agents. iVISPAR is based on a variant of the sliding tile puzzle—a classic problem that demands logical planning, spatial awareness, and multi-step reasoning. The benchmark supports visual 2D, 3D, and text-based input modalities, enabling comprehensive assessments of VLMs' planning and reasoning skills.
We evaluate a broad suite of state-of-the-art open-source and closed-source VLMs, comparing their performance while also providing optimal path solutions and a human baseline to assess the task's complexity and feasibility for humans.
Results indicate that while some VLMs perform well on simple spatial tasks, they encounter difficulties with more complex configurations and problem properties. Notably, while VLMs generally perform better in 2D vision compared to 3D or text-based representations, they consistently fall short of human performance, illustrating the persistent challenge of visual alignment. This highlights critical gaps in current VLM capabilities, highlighting their limitations in achieving human-level cognition.
Find out more on our website.
Figure 1: GPT-4o interacting with the Sliding Geom Puzzle (SGP). Watch more examples on YouTube.
- Multi-modal evaluation: Assess visual-spatial reasoning in VLMs using both visual and text-based representations.
- VLM agent integration: Includes an VLM API to seamlessly integrate models as agents for interactive puzzle-solving tasks.
- AI agent integration: Provides benchmark comparisons with solutions from A* agents and random agents to establish performance baselines.
- Human Baseline: Incorporation of human performance data to define the problem's complexity ceiling and analyze the alignment of model-based spatial reasoning with human logic.
- Customizable and automated: Offers high customizability with automated tools for testing, evaluation, and comparative analysis.
Figure 2: Visualizations of VLMs’ observations for the active and goal states for all modalities: vision 3D (left), vision 2D (middle), text-base representation, provided to models as a string (right) on a 4x4 board at each step during the Sliding Geom Puzzle environment.
- Featuring two puzzle scenarios: Includes two distinct puzzle scenarios, Sliding Geom Puzzle (SGP) and Sliding Tile Puzzle (STP), with a prompt-based API, allowing VLM agents to interact with the environment through an action-perception loop.
- Custom puzzle generation: Generate interactive Sliding Tile Puzzle or Sliding Geom Puzzle datasets with full control over the game complexity.
- Interactive web app: A browser-based graphical user interface (GUI) for human participants to interact with the puzzles, enabling human baseline comparisons.
- Flexible deployment: Run the simulation locally or host it on a server for online access and remote experimentation.
- Move sequence visualization: Each puzzle configuration includes an animated optimal move sequence (computed using A*) to illustrate puzzle complexity.
Figure 3: Depiction of the interaction flow between VLM agents and the iVISPAR Simulator with a progression through an episode with the shortest path solution of 4 steps being solved by prompted actions from a VLM agent.
This is a 3D implementation of the generalized sliding tile puzzle (GSTP), where agents are tasked with moving geoms from a randomly sampled initial to a randomly sampled goal state. Agents can move tiles in the four cardinal directions (following the von Neumann neighborhood) using text prompts that reference their color and shape. Configuration options include board size, number of geoms, shapes and colors of geoms, camera angles, visibility of board labels, representation types (vision or text), complexity (minimal move sequence length), and more.
# Example action:
move blue sphere right
Figure 4: Visualization of Sliding Geom Puzzle (SGP) configuration examples of varying sizes solved by A*.
This is a 3D implementation of the classic Sliding Tile Puzzle (STP), commonly known as the 15-Puzzle or n-Puzzle. Agents are tasked with moving tiles from a randomly sampled initial configuration to a goal state where the numbered tiles are arranged in order. Agents can move tiles in the four cardinal directions (following the von Neumann neighborhood) using text prompts that reference the tile number. STP is a well-known NP-hard problem. Configuration options include board size, number of tiles, camera angles, visibility of board labels, representation types (vision or text), complexity (minimal move sequence length), and more.
# Example action:
move tile 5 down
Figure 5: Visualization of Sliding Tile Puzzle (STP) configuration examples of varying sizes solved by A*.
| Model | Completed Episodes | Step-Deviation from Optimal Path | ||||||
|---|---|---|---|---|---|---|---|---|
| All | 3D | 2D | Text | All | 3D | 2D | Text | |
| Sonnet-3.5 | 54.56 | 28.67 | 89.67 | 45.33 | 3.05 | 4.10 | 1.44 | 3.60 |
| Gemini-2.0-flash | 27.11 | 12.67 | 47.33 | 21.33 | 4.87 | 5.25 | 4.09 | 5.26 |
| GPT-4o | 17.56 | 9.33 | 37.33 | 6.00 | 5.30 | 5.45 | 4.15 | 6.30 |
| InternVL2.5-78B | 10.16 | 1.67 | 9.42 | 19.33 | 5.98 | 6.39 | 5.86 | 5.69 |
| LLaVA-OneVision-72B | 8.22 | 0.67 | 1.33 | 22.67 | 6.35 | 6.75 | 6.81 | 5.50 |
| Qwen2-72B | 5.89 | 0.67 | 1.67 | 15.33 | 6.37 | 6.66 | 6.54 | 5.90 |
Figure 6: VLM evaluation on 900 episodes per model across all three modalities, with 95% confidence intervals. Baseline comparisons for human performance and random moves are shown. VLMs’ success rates of episodes completed with higher values denoting better performance
Figure 7: VLMs’ average action counts per episode. Effective / ineffective actions respectively decrease / increase the path length to the goal state. Occupied destination and out-of-bounds are invalid moves, while illegal commands break the instructed action format, all of which leave the board state unchanged.
Figure 8: Cumulative heatmaps of VLMs’ minimal paths to the goal state after 20 steps across varying complexities, with the number of geoms and shortest path solutions ranging from 2 to 11. Evaluated over 5400 episodes across all modalities (vision 3D, vision 2D, text).
The repository is organized into four main directories:
Contains all the source code required to generate datasets, run experiments, and evaluate results.
- Configuration: Scripts for generating puzzle configuration datasets.
- Experiment: Code for running experiments based on the generated configurations.
- Evaluation: Tools for analyzing and evaluating experimental results.
- Utility: Independent utility scripts that add functions for the project.
- iVISPAR: The Unity project files for the iVISPAR web application.
Contains all data used or generated by the source code, including
- Configs: Contains puzzle configuration files used as input for experiments.
- Experiments: Raw data generated while running experiments, such as logs and execution traces.
- Instructions: Instructions or prompts used to guide AI agents or human participants.
- API keys: Contains the file to set your API keys (note: ensure this directory is properly secured and excluded from any public commits).
- Params: Parameter files used to configure and customize experiments and agent behavior.
- Results: Final output of the experiments, including performance metrics, logs, and summaries.
This directory contains the compiled version of the iVISPAR web app, ready to be launched in your browser. It requires no additional compilation or modification. However, it must connect to the experiment via Python to receive configuration files.
Contains metadata files and supporting resources for the project. This may include environment files, dependency lists, and documentation files.
iVISPAR has minimal dependencies and can be easily set up and run in just a few simple steps.
Clone the repository.
git clone https://github.com/SharkyBamboozle/iVISPAR.git
cd iVISPARiVISPAR runs on Python 3. You can find the list of required Python dependencies in Resources/environment.yml. We recommend using Anaconda to create a new conda environment.
conda env create -f Resources/environment.yml
conda activate conda_env_iVISPARYou can find an example of how to run the code in Source/main.py. First, the script generates a minimal dataset of configuration files. Next, iVISPAR is launched in your web browser and prompts you to copy the client ID from the web app into the Python console. Once connected, an AI agent, acting as a stand-in for your VLMs API connection, executes the optimal path. Finally, the results are evaluated and plotted automatically. For more details, see Quick Start.
cd Source
python main.pyThe project includes the compiled iVISPAR web app, which works out of the box on any common operating system with a web browser installed. The web app is built with C# using Unity, and its source code is available in Source/iVISPAR. We also provide a fully online working version for human experiments.
Note
No knowledge of C# or compilation of the Unity project is required to run experiments with iVISPAR.
Figure 9: Web app with UI for human experiments.
We have prepared a Getting Started Guide with step-by-step instructions to help you get started quickly.
Please find our publication on iVISPAR on arXiv. If you use iVISPAR in your work, we kindly ask you to cite our paper:
Mayer, J., Ballout, M., Jassim, S., Nosrat Nezami, F., & Bruni, E. (2025). iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs. arXiv. https://arxiv.org/abs/2502.03214
@misc{mayer2025ivispar,
title={iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs},
author={Julius Mayer and Mohamad Ballout and Serwan Jassim and Farbod Nosrat Nezami and Elia Bruni},
year={2025},
eprint={2502.03214},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.03214},
}