- GraspMAS : swapped the GPT-4o backend for a locally-run Qwen2-VL-7B on AMD MI300X/ROCm, added directional 6-DoF language-conditioned grasping and a physics-aware force/mass-estimation pipeline
GraspMAS example gallery
| Pipeline / method | Directional-grasp concept | Direction success rates |
 |
 |
 |
| Baseline comparison | Real-robot experiment | Benchmark comparison |
 |
 |
 |
| Mustard — scene input | Mustard — top grasp | Orange — all 4 directions |
 |
 |
 |
| OCID-VLG — ball grasp | YCB — banana grasp | ManiSkill — pick execution |
 |
 |
 |
Experiments run in this fork
- Local Qwen2-VL backend swap
- Directional 6-DoF grasping
- Physics-aware force estimation
- Zero-shot YCB mass evaluation
- OCID-VLG benchmark with Qwen2-VL
- Agent runtime hardening (GMAS runtime)
- Full grasp example gallery
Future goal: move past the planner/observer/coder multi-agent split toward an OpenClaw-style VLM — a single end-to-end vision-language-action model that drives the claw directly from language + image input.
- ShapeGrasp — extended zero-shot task-oriented grasping to real-world images via GroundingDINO + SAM auto-segmentation, swapped in local Qwen2-VL-7B inference, and validated on ManiSkill YCB (61% success, 30/49 objects).
| ManiSkill — scissors pipeline (RGB → hulls → shapes → LLM grasp) | Real-world — bottle pipeline (detect → segment → shape → grasp) |
|---|---|
 |
 |
Success story
-
kinematics-flow (weights) — trained the multi-embodiment grasp model to convergence across 5 grippers (Panda, Shadow, Allegro, Dexee, VX300), releasing checkpoints from epoch 5 through 120 with mean success rates climbing to 86.1%.
Dexee gripper — successful grasp rollout on the final (epoch 120) checkpoint.
Debugging in progress
- geomatch — trying to train and reproduce geomatch and geomatch++ : a range of multiembodied grasping models from google deepmind
here is the hf link : https://huggingface.co/Dimios45/Graspmax
core training/eval pipeline still being debugged.
- 3DObjectTracking — forked DLR-RM's region/depth tracking suite (RBGT → M3T) as a reference base for object pose tracking; evaluating algorithms ahead of integrating one into the grasping pipeline.
- Benchmarked M3T against 2D point trackers (CoTracker2, CoTracker3, LocoTrack) on the OPT dataset in the
comparisonbranch — CoTracker3 wins on speed/accuracy tradeoff, matching LocoTrack's mean TSR (0.948 vs 0.949) at ~5x the speed (33ms vs 180ms/frame on MI300X).
CoTracker3 pear-tracking demo (from cotracker_eval/)
| Input frame | Mask overlay | Tracked (frame 0 / 100 / 250) |
|---|---|---|
 |
 |
   |
- WiLoR — forked the CVPR 2025 3D hand localization/reconstruction model to source hand-flow data for GenieMotion experiments.
| Flow overlay | Render overlay |
|---|---|
 |
 |
6dof + multifinger + rosclaw type architecture
single video demo to direct grasping + multi finger + dual robot / object flow
Full-res clip : assets/genie_motion.mp4