AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects

Danrui Li^1,4; Jiahao Zhang^2,4; Bernhard Egger^3,4; Moitreya Chatterjee⁴; Suhas Lohit⁴; Tim K. Marks⁴; Anoop Cherian⁴

¹Rutgers University ²Australian National University ³Friedrich-Alexander-Universität Erlangen-Nürnberg ⁴MERL

Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified scenarios, overlooking shape complexities and assembly trajectories in industrial assemblies.

We introduce AssemblyBench, a synthetic dataset of 2,789 industrial objects with multimodal instruction manuals, corresponding 3D part models, and physically plausible 6-DoF part assembly trajectories.

We also propose a transformer-based model, AssemblyDyno, which uses the instructional manual and the 3D shape of each part to jointly predict assembly order and part assembly trajectories. AssemblyDyno outperforms prior works in both assembly pose estimation and trajectory feasibility, where the latter is evaluated by our physics-based simulations.

Prerequisites

Installation

# clone project
git clone https://github.com/...(TODO: complete link here)

# install uv, https://docs.astral.sh/uv/getting-started/installation/
curl -LsSf https://astral.sh/uv/install.sh | sh

# install dependencies
uv sync

Dataset & Checkpoint Download

Download AssemblyBench dataset and checkpoint from Zenodo. (TODO: Attach links here)

Dataset and checkpoint usage notice

Our AssemblyBench dataset extends the Assemble-Them-All dataset, originally released under the MIT License. The original data remains under the MIT License. As the Assemble-Them-All dataset uses assets from the Fusion 360 Gallary Dataset, please refer to the Fusion 360 Gallery Dataset License for legal usage.

Please see our dataset page for the license information of our release.

Inferencing Using Pretrained Models

Standard Setting

source .venv/bin/activate
uv run src/main.py test \
  -c configs/data/assemblybench_synthetic_stepwise.yaml \
  -c configs/model/assembly_dyno.yaml \
  --data.order_pathname checkpoints/best_orders.json \
  --trainer.logger.name AssemblyDyno_Test \
  --ckpt_path checkpoints/best_model.ckpt

Use GT part orders

source .venv/bin/activate
uv run src/main.py test \
  -c configs/data/assemblybench_synthetic_stepwise.yaml \
  -c configs/model/assembly_dyno.yaml \
  --trainer.logger.name AssemblyDyno_Test_GTOrder \
  --ckpt_path checkpoints/best_model.ckpt

Training

Training Stage 1: Permutation Learning

Training

uv run src/main.py fit \
  -c configs/data/assemblybench_synthetic_stepwise.yaml \
  -c configs/model/assembly_dyno_order.yaml \
  --trainer.logger.name AssemblyDyno_Order

Inference

uv run src/main.py test \
  -c configs/data/assemblybench_synthetic_stepwise.yaml \
  -c configs/model/assembly_dyno_order.yaml \
  --trainer.logger.name AssemblyDyno_Order_Test \
  --ckpt_path logs/AssemblyDyno/<run_id>/checkpoints/<best>.ckpt

Training Stage 2: Part Pose Estimation

Training

uv run src/main.py fit \
  -c configs/data/assemblybench_synthetic_stepwise.yaml \
  -c configs/model/assembly_dyno.yaml \
  --trainer.logger.name AssemblyDyno

Inference

uv run src/main.py test \
  -c configs/data/assemblybench_synthetic_stepwise.yaml \
  -c configs/model/assembly_dyno.yaml \
  --trainer.logger.name AssemblyDyno_Test \
  --data.order_pathname logs/AssemblyDyno/<run_id>/pred_orders.json \
  --ckpt_path logs/AssemblyDyno/<run_id>/checkpoints/<best>.ckpt

Simulator-based Evaluation

Please see tools/evaluation/README.md for details.

Dataset Generation Pipeline

Please see tools/dataset_gen/README.md for details.

Citation

@inproceedings{Li2026AssemblyBench,
  author    = {Li, Danrui and Zhang, Jiahao and Egger, Bernhard and Chatterjee, Moitreya and Lohit, Suhas and Marks, Tim K. and Cherian, Anoop},
  title     = {AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026},
}

License

Released under AGPL-3.0-or-later license, as found in the LICENSE.md file.

All files, except as noted below:

Copyright (C) 2026 Mitsubishi Electric Research Laboratories (MERL)

SPDX-License-Identifier: AGPL-3.0-or-later

The following files in:

• src/models/manual_pa/transformer.py was modified from the PyTorch TransformerDecoder released under the BSD 3-Clause License (found in LICENSES/BSD-3-Clause.txt).

The following files in:

• tools/dataset_gen/external/patch_files/Assemble-Them-All/examples
• tools/dataset_gen/external/patch_files/Assemble-Them-All/assets

were modified from Assemble-Them-All released under the MIT License (license included in LICENSES/MIT.txt)).

The following file in::

• tools/dataset_gen/external/patch_files/ASAP/plan_sequence

were modified from ASAP released under the MIT License (license included in LICENSES/MIT.txt)).

The following files:

• tools/dataset_gen/utils/BaseBlenderConnection.py
• tools/dataset_gen/mcps/BlenderMCP/asap_import_server.py
• tools/dataset_gen/mcps/BlenderMCP/asmbl_import_server.py
• tools/dataset_gen/mcps/BlenderMCP/base_server.py
• tools/dataset_gen/mcps/BlenderMCP/manual_server.py

were adapted from here released under the MIT License (license included in LICENSES/MIT.txt)).

The files in the following folders

• src/*
• configs/* were adapted from here released under the MIT License (license included in LICENSES/MIT.txt)).