DexEvolve: Evolutionary Optimization for Robust and Diverse Dexterous Grasp Synthesis

René Zurbrügg, Andrei Cramariuc, Marco Hutter

Dexterous grasping relies on large, diverse, physically feasible grasp datasets that are costly to generate. Analytical grasp synthesis scales data collection, but its simplifying assumptions yield infeasible grasps that must be filtered in high-fidelity simulation — discarding most candidates and their diversity. DexEvolve is a scalable generate-and-refine pipeline: rather than using the high-fidelity simulator only to verify and filter, we use it as an optimization stage that continuously improves grasp quality without discarding precomputed candidates. We seed an evolutionary search with analytically generated, potentially suboptimal grasps and refine them directly in Isaac Sim with an asynchronous, gradient-free evolutionary algorithm, improving stability while maintaining diversity. The refinement can be guided toward human preferences and/or domain-specific quality metrics without a differentiable objective, and the refined distribution can be distilled into a diffusion model for real-world deployment. On a newly introduced Handles dataset and a DexGraspNet subset, DexEvolve yields 120+ distinct stable grasps per object (1.7–6× over unrefined analytical methods) and 46–60% higher unique-grasp coverage than diffusion-based alternatives.

This repository implements the generate and refine stages of that pipeline: create a dataset (the Handles dataset, via IkeaParser), generate the analytical seed grasps (via graspqp), and refine them with the asynchronous evolutionary search running on GPU physics in Isaac Lab. The downstream diffusion distillation is described in the paper and is not part of this repo.

 ┌──────────────┐     ┌──────────────────────┐     ┌──────────────────────────┐     ┌ ─ ─ ─ ─ ─ ─ ─ ┐
 │ 1. DATASET   │     │ 2. SEED (analytical) │     │ 3. REFINE (evolutionary) │       4. DISTILL
 │ IkeaParser   │ ──▶ │ graspqp  (fit.py)    │ ──▶ │ DexEvolveIsaacLab        │ ──▶ │ diffusion model │
 │ .glb → .usd  │     │ initial .dexgrasp.pt │     │ mine_object_grasp.py     │       (see paper)
 └──────────────┘     └──────────────────────┘     └──────────────────────────┘     └ ─ ─ ─ ─ ─ ─ ─ ┘
   Handles dataset      seed grasp candidates        refined + diverse grasps          (not in this repo)

Repository structure

DexEvolve/
├── IkeaParser/             # dataset tooling: download IKEA meshes + convert .glb → .usd
│   └── README.md           #   (package `ikea_assets`)
├── DexEvolveIsaacLab/      # the grasp-mining Isaac Lab extension + scripts
│   └── README.md           #   (package `dex_evolve`, tasks `Evol-…-Grasp-Mining-…`)
├── docker/                 # Dockerfile / compose to build & run the full stack
└── README.md               # ← you are here

Each subproject has its own README with the full detail; this file is the top-level walkthrough that strings them together.

Dependency: graspqp

The seed stage and the base grasp environments come from graspqp / graspqp_isaaclab (not part of this repo). The DexEvolve docker image is built on top of the graspqp_isaaclab image, which already provides Isaac Lab, graspqp, graspqp_isaaclab, pytorch3d and kaolin. Build that base first from a current graspqp checkout with graspqp's docker/build_isaaclab_docker.sh (which needs an isaac-lab-base image built from Isaac Lab 2.3 / Isaac Sim 5.1 — older Isaac Lab lacks APIs graspqp/DexEvolve rely on), then build DexEvolve on top. If the base is stale, the seed or mine stage fails — see Troubleshooting.

---

Setup (Docker)

# 0. Prerequisite: the graspqp_isaaclab base image must exist (see graspqp's README).

# 1. Point docker/.env.base at your host dataset directory:
#      DATA_DIR=/path/to/your/dataset           # mounted to /data in the container

# 2. Build the DexEvolve image (installs ikea_assets[crawler], dex_evolve and the
#    qpth/proxsuite solvers on top of the base).
cd docker && ./build.sh

# 3. Open an interactive GPU shell (repo is bind-mounted, edits are live).
docker compose --env-file .env.base -f docker-compose.yaml run --rm --gpus all dexevolve

Inside the container the dataset directory is mounted at /data, this repo at /workspace/DexEvolve, and Isaac Lab's launcher isaaclab.sh -p is on PATH (callable from any directory). All commands below are run inside this shell. Note that files written from the container (dataset, results) are owned by root on the host.

Native (non-Docker) installs are possible too — see the per-subproject READMEs. You need an Isaac Lab 2.3 (Isaac Sim 5.1) runtime with graspqp / graspqp_isaaclab importable.

---

Quickstart: one asset, end to end

The fastest way to validate your setup — takes a few minutes total on a single GPU and exercises every stage on one handle (ENERYDA, item 20347509):

# Stage 1: download + convert one handle
cd /workspace/DexEvolve/IkeaParser
IKEA_DATA_DIR="$PWD/static" python -c "
from ikea_assets.data_crawler import IkeaDataCrawler
IkeaDataCrawler('handles').download_item_numbers(['20347509'])"
isaaclab.sh -p scripts/process_asset.py --dataset handles --parser ikea-handles --headless
cp -r static/handles/.data/usd_assets/20347509 /data/20347509

# Stage 2: seed grasps (reduced iterations for a quick check; use 7000 for real quality)
cd /workspace/graspqp
isaaclab.sh -p scripts/fit.py --data_root_path /data --object_code_list 20347509 \
    --hand_name allegro --n_contact 12 --energy_type handle --energy_name handles \
    --grasp_type tips --batch_size 32 --n_iter 200

# Stage 3: mine (tiny smoke config; see below for the real settings)
cd /workspace/DexEvolve/DexEvolveIsaacLab
isaaclab.sh -p scripts/mine_object_grasp.py \
    --object_type Handle --hand_type allegro --n_contacts 12 \
    --energy_type handles --grasp_type tips --assets 20347509 \
    --data_path /data --num_envs 4 --total_steps 10 --eval_freq 5 --no_log --headless

Success looks like: stage 1 leaves ENERYDA_20347509_20347509.usd, ENERYDA_20347509_20347509_3dof_.usd and remeshed.obj in /data/20347509/; stage 2 prints ==> Exported to /data/20347509/grasp_predictions/allegro/12_contacts/handles/tips/20347509.dexgrasp.pt; stage 3 finishes with an evaluation table and *_mined*.pt / eval .csv files next to the seed. The smoke config evolves only 4 grasps from barely-optimized seeds for 10 steps, so the reported success rate is noise — it verifies the plumbing, not grasp quality. For paper-quality results use --n_iter 7000 (stage 2) and --num_envs 4096 --total_steps 10000 --eval_freq 2000 (stage 3), as in the full walkthrough below.

---

1. Create the asset dataset (IkeaParser)

Download the IKEA handle meshes and convert them to articulated USDs. The exact item numbers from the paper are in IkeaParser/handle_ids.txt (90 handles; only those whose 3D assets still exist on IKEA's CDN will download).

cd /workspace/DexEvolve/IkeaParser

# a) Download the handles by item number into static/  (no browser required; the
#    crawler dependencies are preinstalled in the docker image — on a native install
#    run `pip install -e ".[crawler]"` first)
IKEA_DATA_DIR="$PWD/static" python -c "
from ikea_assets.data_crawler import IkeaDataCrawler
ids = open('handle_ids.txt').read().split()
IkeaDataCrawler('handles').download_item_numbers(ids)"

# b) Convert .glb → .usd  (needs Isaac Sim). For each handle the 'ikea-handles' parser
#    adds a back panel behind the handle, connected by a prismatic joint, and additionally
#    writes the 3-DOF articulated variant (X/Y prismatic joints) the mining env requires.
isaaclab.sh -p scripts/process_asset.py --dataset handles --parser ikea-handles --headless

# c) Copy each processed item folder into the dataset root the later stages read from.
for d in static/handles/.data/usd_assets/*/; do cp -r "$d" /data/; done

Per item this produces, under static/handles/.data/usd_assets/<itemNo>/:

• <name>.usd — the processed handle (name pattern <NAME>_<itemNo>_<itemNo>.usd),
• <name>_3dof_.usd — the articulated variant loaded by the Handle mining environments,
• remeshed.obj — the watertight mesh the seed stage optimizes on,
• collision.usd, asset_info.json, textures/.

Full options (new datasets by category, color/finish variations, visualization, Streamlit browser): see IkeaParser/README.md.

Expected dataset layout

Both the seed and mine stages discover assets from a single dataset root (--data_path / --data_root_path). Each asset is one subfolder named by item number:

/data/
└── 20347509/                                    # one folder per asset (item number)
    ├── <name>.usd                                # processed USD (filename ends in <itemNo>.usd)
    ├── <name>_3dof_.usd                          # articulated variant (Handle only, REQUIRED by the miner)
    ├── remeshed.obj                              # mesh used by the seed stage (fit.py)
    └── grasp_predictions/                        # created by stage 2 (graspqp)
        └── allegro/                              # --hand_type / --hand_name
            └── 12_contacts/                      # --n_contacts / --n_contact
                └── handles/                      # --energy_type / --energy_name
                    └── tips/                     # --grasp_type
                        └── <itemNo>.dexgrasp.pt  # seed grasps the miner evolves

---

2. Seed the grasps with graspqp (fit.py)

The miner evolves an existing grasp population, so each asset first needs an initial set of grasps. Generate them with graspqp's fit.py, writing into the grasp_predictions/… tree the miner reads from. fit.py ships with graspqp (already in the image at /workspace/graspqp).

Default example (allegro hand):

cd /workspace/graspqp

isaaclab.sh -p scripts/fit.py \
    --data_root_path /data \                       # dataset root (same dir the miner uses)
    --object_code_file /workspace/DexEvolve/IkeaParser/handle_ids.txt \  # or --object_code_list <id> …
    --hand_name   allegro \                        # dexterous hand (graspqp default)
    --n_contact   12 \                             # contact points
    --energy_type handle \                         # handle grasp energy (adds pull-direction terms)
    --energy_name handles \                        # ← output folder name (must match the miner's --energy_type)
    --grasp_type  tips \                           # grasp style → output subfolder
    --batch_size  32 \                             # grasps optimized per object
    --n_iter      7000                             # optimization steps (7000 = paper setting)

Output (per asset): /data/<id>/grasp_predictions/allegro/12_contacts/handles/tips/<id>.dexgrasp.pt.

For another hand, swap --hand_name (e.g. shadow_hand); the output folder name changes to match, so use the same value for the miner's --hand_type in stage 3.

Folder-name mapping matters. The miner discovers seeds by path, so the names must line up: graspqp's --energy_name ↔ miner's --energy_type, --n_contact ↔ --n_contacts, --hand_name ↔ --hand_type, and --grasp_type is identical on both sides. The example above produces the …/allegro/12_contacts/handles/tips/ tree the mining command below expects.

---

3. Refine grasps — evolutionary mining (DexEvolveIsaacLab)

With seeds in place, run the asynchronous, gradient-free evolutionary search. It auto-discovers each asset's _3dof_ USD and seed .dexgrasp.pt, refines the population directly in Isaac Sim (selection / crossover / mutation on a fitness pool), periodically evaluates the best survivors and saves them back under grasp_predictions/… — improving stability while keeping the seed candidates and maintaining diversity rather than filtering them away.

Default example (allegro hand, paper settings):

cd /workspace/DexEvolve/DexEvolveIsaacLab

isaaclab.sh -p scripts/mine_object_grasp.py \
    --object_type Handle --hand_type allegro \     # task = Evol-Handle-Grasp-Mining-allegro-v0
    --n_contacts 12 --energy_type handles --grasp_type tips \  # ← match the seed folders
    --data_path /data \                            # dataset root (omit --assets to mine ALL ids)
    --num_envs 4096 --total_steps 10000 --eval_freq 2000 \
    --headless

Useful flags: --assets <id> [<id> …] to mine a subset (default: every folder under --data_path); GA knobs --n_parents, --tournament_size, --mutation_rate, --crossover_probability; --skip_eval to skip periodic in-sim evaluation; --no_log to disable wandb (otherwise pass --log_to_wandb).

Every --eval_freq steps the current population is evaluated in simulation (the object is pulled along each axis while the grasp must hold) and these files are written next to the seed .dexgrasp.pt:

• <id>.dexgrasp_step_<N>.handles.mined.pt — the full mined population at step N,
• <id>.dexgrasp_step_<N>.handles.mined_survivors.pt — the evaluated best survivors,
• succ_grasps__mined_step_<N>.pt / failed_grasps__mined_step_<N>.pt — the successful / failed grasps from that evaluation round,
• dexgrasp_eval_isaac_sim__mined_step_<N>.csv — per-asset success-rate statistics.

Hands registered for the tasks: allegro (default), ability_hand, panda, robotiq2, robotiq3, schunk2f, shadow_hand. (xhand is supported in code but its hand asset is not part of the public graspqp_isaaclab release, so its config is skipped during task registration — see dex_evolve/tasks/__init__.py.)

Guiding refinement (preferences / quality metrics)

Because the search is gradient-free, the objective need not be differentiable — refinement can be steered toward human preferences or domain-specific quality metrics. Pass a trained reward model with --reward_model_checkpoint <ckpt> (weighted by --w_e_reward); the energy weights --w_e_fc, --w_e_dis, --w_e_pen, --w_lifetime tune the analytical fitness terms.

---

4. Evaluate (optional)

Mining already evaluates survivors every --eval_freq steps. To re-evaluate a stored set:

cd /workspace/DexEvolve/DexEvolveIsaacLab
isaaclab.sh -p scripts/eval_object_grasp.py \
    --object_type Handle --hand_type allegro --n_contacts 12 \
    --energy_type handles --grasp_type tips --data_path /data --headless

---

At a glance: parameter consistency across stages

The same logical configuration appears under different flag names in each stage. Keep them aligned or the miner won't find the seeds:

Concept	IkeaParser	graspqp fit.py	miner mine_object_grasp.py
dataset root	IKEA_DATA_DIR/static → /data	--data_root_path /data	--data_path /data
asset ids	handle_ids.txt	--object_code_file / --object_code_list	--assets (or all)
hand	—	--hand_name allegro	--hand_type allegro
contacts	—	--n_contact 12	--n_contacts 12
energy folder	—	--energy_name handles	--energy_type handles
grasp style	—	--grasp_type tips	--grasp_type tips
object kind	--parser ikea-handles	--energy_type handle	--object_type Handle

Troubleshooting

• 'Trimesh' object has no attribute 'remove_degenerate_faces', a "backward through the graph a second time" error in fit.py, or module '….grasp.mdp' has no attribute 'joint_pos' when the mining task registers → your graspqp_isaaclab base image was built from an outdated graspqp checkout (before the Isaac Lab 2.x / trimesh 4 port). Rebuild it from current graspqp, then rebuild DexEvolve.
• qpth is required for SQPLsqSolver in fit.py → you are not in the DexEvolve image (which installs qpth/proxsuite on top of the base); run pip install qpth proxsuite.
• 3DOF handle file not found for asset … from the miner → the asset folder is missing its <name>_3dof_.usd. Re-run process_asset.py --parser ikea-handles (it writes the variant automatically) and copy the full item folder into /data/<itemNo>/.
• Isaac Sim refuses to start with an EULA message → the compose file sets ACCEPT_EULA=Y/PRIVACY_CONSENT=Y; if you docker run the image manually, pass those env vars (and --entrypoint bash — the base image's default entrypoint launches the Isaac Sim streaming app instead).
• Item <id> already exists. Skipping from the crawler → delete static/handles/.data/raw_assets/<id>/ to force a re-download.

Citation

If you use DexEvolve, the Handles dataset, or this code, please cite:

@inproceedings{zurbrugg2026dexevolve,
  title     = {DexEvolve: Evolutionary Optimization for Robust and Diverse Dexterous Grasp Synthesis},
  author    = {Zurbr{\"u}gg, Ren{\'e} and Cramariuc, Andrei and Hutter, Marco},
  booktitle = {Robotics: Science and Systems (RSS)},
  year      = {2026}
}

License and copyright

©2026 ETH Zurich, René Zurbrügg (Robotic Systems Lab). Released under the MIT License — see LICENSE and AUTHORS.

Files derived from the Isaac Lab project / extension template retain their BSD-3-Clause license (see NOTICE for the full third-party attribution). The IKEA 3D assets downloaded by IkeaParser are the property of Inter IKEA Systems B.V., are not part of this repository, and are not covered by its license.