Challenges when deploying PI0 on DROID like Setup using Franka

[NeumannJulius]

Environment:

• Robot: Franka Emika (using either the Franka Gripper or OnRobot RG6)
• Controller: Joint Velocity Controller (with chained safety controller for limiting velocities and efforts)
• Vision System:
  • Wrist camera
  • External camera
    (Images are zero-padded and resized to match model input)

Images:

• Base Image:
  
• Wrist Image:
  

---

Issue Description:

• When deploying the pi0_droid_fast policy, the robot drives randomly with no clear task intent.

---

Questions & Points for Investigation:

1. DROID Inference without Robotiq Gripper:

   • Has anyone experienced issues running the DROID inference without using the Robotiq Gripper?

2. Normalization Statistics:

   • Could loading the default normalization statistics (not on my domain specific) be causing this behavior?

3. Camera Calibration:

   • Is it necessary to calibrate the cameras before deployment?

4. Action Clipping by Safety Controller:

   • The chained safety controller regularly clips the velocity actions.
   • Would a linear scaling of the velocity actions be a viable solution, or might that lead to unexpected behavior?

---

Additional Context:

• Not running the DROID RobotEnv.
• Using a Joint Velocity Controller (assuming no Cartesian velocity controller).

Any insights, suggestions, or experiences with similar setups would be greatly appreciated!

[kpertsch]

Hi!

It is important to note that the pi0_fast_droid policy can only be used out of the box on DROID setups, ie setups that use the DROID cameras (2x ZED), DROID gripper setup (Robotiq), and DROID controller setup (polymetis, joint velocity control, gripper absolute position control). The reason is that the model is fine-tuned on a dataset with data that was exclusively collected using this setup. So we do not expect it to generalize to a new setup out of the box.

Your options are (1) modify your setup to match the DROID setup specifications (see https://droid-dataset.github.io/droid/) or (2) collect some fine-tuning data in your setup and adapt the model to it. For best generalization results I would recommend (1), but I understand that there may be some hardware constraints!

Hope this helps! I will close this issue for now, but feel free to reopen if there are more things!

[akhilrkurup]

@NeumannJulius were you able to get Pi0 working on a non DROID Franka platform ?

[NeumannJulius]

@akhilrkurup We were only able to run pi0_droid partially and with some restrictions. While the model could not be executed out of the box, we successfully fine-tuned the pi0_base model.
Initial successes were achieved with as few as ten real-world demonstrations.
Key factors contributing to these results included correct synchronization of observations and actions, accurate gripper parameterization, recalculation of normalization statistics, and maintaining environmental consistency.
Performance on long-horizon tasks was less consistent.

[aatxag]

Hi @NeumannJulius , which values of accuracy are you achieving? Which kind of tasks and benchmarks did you work with?
Thank you :)

[NeumannJulius]

@akhilrkurup
We achieved around 50% success on a simple Pick Object task with 10 demonstrations, and close to 100% with 100 demonstrations, all within a restricted workspace.
In contrast, a two-phase Pick-and-Place task involving two distinct objectives reached a maximum success rate of 25%, which could be improved e.g. through data augmentation.

[aatxag]

Alright, thankyou so much for the info, thats very interesting @NeumannJulius . Did you publish something?

[Samsam050]

did you try without the robotiq gripper?