MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents

Ziming Wei^1*, Bingqian Lin^2*, Zijian Jiao^1*, Yunshuang Nie¹, Liang Ma³,
Yuecheng Liu⁴, Yuzheng Zhuang⁴, Xiaodan Liang^1✉

¹Shenzhen Campus of Sun Yat-Sen University,  ²Shanghai Jiao Tong University, 
³Mohamed bin Zayed University of Artificial Intelligence,  ⁴Huawei Noah’s Ark Lab
^* Equal contribution   ^✉ Corresponding author  

---

Spatial Planning is a crucial part in the field of spatial intelligence, which requires the understanding and planning about object arrangements in space perspective. AI agents with the spatial planning ability can better adapt to various real-world applications, including robotic manipulation, automatic assembly, urban planning etc. Recent works have attempted to construct benchmarks for evaluating the spatial intelligence of Multimodal Large Language Models (MLLMs). Nevertheless, these benchmarks primarily focus on spatial reasoning based on typical Visual Question-Answering (VQA) forms, which suffers from the gap between abstract spatial understanding and concrete task execution. In this work, we take a step further to build a comprehensive benchmark called MineAnyBuild, aiming to evaluate the spatial planning ability of open-world AI agents in the Minecraft game. Specifically, MineAnyBuild requires an agent to generate executable architecture building plans based on the given multi-modal human instructions. It involves 4,000 curated spatial planning tasks and also provides a paradigm for infinitely expandable data collection by utilizing rich player-generated content. MineAnyBuild evaluates spatial planning through four core supporting dimensions: spatial understanding, spatial reasoning, creativity, and spatial commonsense. Based on MineAnyBuild, we perform a comprehensive evaluation for existing MLLM-based agents, revealing the severe limitations but enormous potential in their spatial planning abilities. We believe our MineAnyBuild will open new avenues for the evaluation of spatial intelligence and help promote further development for open-world AI agents capable of spatial planning.

📰 Updates

• [05/2025] Arxiv paper released.

🚨 TODOs (Urgency, To-be-completed in ~June 2025)

• Remaining codes for evaluation of MLLM-based agents (5 tasks)
• Remaining codes for data curation (w/ example JSON files)
• Remaining codes for inference of MLLM-based agents (5 tasks)
• Icons&urls for HuggingFace datasets, project webpage.
• Upload files to Google Drive (e.g., pure map for evaluation).

✅ TODOs (Future)

• More detailed annotations
• Docs of instructions for Replay Mod
• Optimization of the use of visualization tools (timer in Replay Mod code, viewer)
• RL environment codes for Mineflayer
• Fix known bugs
• Debugging and provide adaptation to normal python codes (w/o Jupyter notebook)
• MineRL/MineDojo codes

---

Contents

• Contents
• Installation
  • Install Minecraft
  • Install packages
  • Create a Conda env and packages
• Running codes
  • Evaluation
    • Executable Spatial Plan Generation, Creativity and Spatial Understanding tasks
    • Spatial Reasoning task
    • Spatial Commonsense task
  • Data curation
  • Inference of MLLM-based agents
    • Proprietary MLLMs
    • Open-source MLLMs
• Citation
• Acknowledgement

Installation

Install Minecraft

Ensure you have a valid Microsoft account and Minecraft JAVA Edition.

1. Visit the Minecraft Official Website and navigate to the "Get Minecraft" section.
2. Select your platform (Windows, Linux or macOS) and choose JAVA Edition.
3. Download the launcher and install it, and then log in using your Microsoft account.
4. Start the launcher, select the release version (1.20.4) compatible with Mineflayer and click Play to load the game.
5. Load the test environment map we provide (recommanded) or create a new world with settings.
6. Open the Pause Menu, click the Open to LAN, and following the settings:
   • Take note of the Port Number displayed (in the lower versions of Minecraft) or enter a Port Number that is consistent with the code (in the latest versions of Minecraft).
   • Set the game mode to Creative.
   • [IMPORTANT] Enable Cheats to allow command usage required for MLLM-based agents.

For more detailed instructions, visit the Minecraft Help Center for help.

Install packages

1. Download and install the appropriate version of Node.js.
2. Run the following command to install Mineflayer on your terminal:

npm install mineflayer

Create a Conda env and packages

conda create -n mineflayer python=3.10
conda activate mineflayer
pip install -r requirements.txt

Running codes

Evaluation

You should run the Inference of MLLM-based agents first to generate the responses of MLLM-based agents.

For different tasks in our MineAnyBuild, please conduct evaluation according to the following steps respectively.

Executable Spatial Plan Generation, Creativity and Spatial Understanding tasks

1. Start the Minecraft game and record the video with Replay Mod (recommended currently).
   1) Start the recording in Pause Menu of Minecraft game.
   2) Run /mineanybuild/mineflayer.ipynb and specify the starting frame.
   3) Stop the recording when the notebook cells finish executing.
   4) Save and render the video in Replay Mod Menu.
   (For the concrete instructions, please refer to the Replay Mod documentation and Section B.3.2 in the Supplementary Material. We will provide a Doc for usage instructions of Replay Mod in several weeks.)

2. Video Segmentation Split the video into multiple frames and match each frame to its corresponding building structure. We provide an example of using this in frame_clipper() and select_frames() functions of /mineanybuild/utils.py. You can use it to match automatically or conduct manual matching selection (recommended for accuracy) if less data is to be tested.

3. Run the functions/scripts in /mineanybuild/evaluator.py following the below instructions.

   python /mineanybuild/evaluator.py --task [Spatial_Understanding|Creativity|Executable_Spatial_Plan_Generation]
   

4. Run the function process_json_critic_scores() in /mineanybuild/utils.py to process the JSON files generated by the previous step.

5. Run the functions calculate_scores_creativity(), calculate_scores_spatial_plan(), calculate_scores_spatial_understanding() to calculate the final scores of the three tasks respectively.

Spatial Reasoning task

1. Run the function calculate_scores_spatial_reasoning() in /mineanybuild/evaluator.py to calculate the Accuracy of the Spatial Reasoning task.

Spatial Commonsense task

1. Run the functions critic_spatial_commonsense() (or critic_spatial_commonsense_opensource()) in /mineanybuild/evaluator.py following the below instructions.

   python /mineanybuild/evaluator.py --task Spatial_Commonsense
   

2. Run the function calculate_scores_spatial_commonsense() in /mineanybuild/evaluator.py to calculate the scores of the Spatial Commonsense task.

Data curation

1. Run /data_curation/customize_data.ipynb to obtain the curated data.

2. Run the function in /data_curation/utils.py to process the data and format it in JSON.

3. Generate instructions of some tasks for the curated architecture data.

   python 
   

4. Calculate the difficulty factor of new data.

   python /data_curation/calculate_difficulty.py
   

Inference of MLLM-based agents

Specify your API keys for each models.

Run the following codes to perform inference for MLLM-based agents.

Proprietary MLLMs

1. Run /mineanybuild/prompter.py specifying the task of MineAnyBuild. Please specify the input and output file/directories for each task function.

python /mineanybuild/prompter.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]

2. Run the json_parser_blueprint() function in /mineanybuild/utils.py to transform the response of MLLM-based agents into blueprint 3D matrix in JSON format.

Open-source MLLMs

1. Run the following codes based on the type of open-source MLLMs you want to run.

python /mineanybuild/internvl.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]
python /mineanybuild/qwenvl.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]
python /mineanybuild/llavaov.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]

2. Run the json_parser_blueprint() function in /mineanybuild/utils.py to transform the response of MLLM-based agents into blueprint 3D matrix in JSON format.

Citation

If you find this work useful, please consider citing:

@article{wei2025mineanybuild,
  title={MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents},
  author={Wei, Ziming and Lin, Bingqian and Jiao, Zijian and Nie, Yunshuang and Ma, Liang and Liu, Yuecheng and Zhuang, Yuzheng and Liang, Xiaodan},
  journal={arXiv preprint arXiv:2505.20148},
  year={2025}
}

Acknowledgement

Some of the codes are built upon APT. Thanks them for their great works!