Clarifications on fine-tuning on different envs and embodiments

[RonPlusSign]

Hi everyone,
I’m currently working on fine-tuning SmolVLA and π₀ using RLBench. The robot setup is a Franka Emika Panda (7DoF + gripper), and I’ve already collected custom LeRobot datasets for a pick-and-place task (available on my Hugging Face) with 500 demo episodes.

I’ve successfully fine-tuned OpenVLA using its official repository, where the action space is defined as ΔEEF pose (Euler rotation) + gripper, and the state as ΔEEF pose (quaternion rotation) + gripper, using a single observation image (left shoulder), reaching around 22% success rate.

However, when trying to fine-tune SmolVLA, despite the training running without issues (loss converges and wandb plots look fine), the evaluation yields 0% success. I suspect I’m misunderstanding how to correctly define the state and action spaces for SmolVLA in this context.
Since RLBench is not one of the officially supported envs, I created an evaluation script (you can find it here), similar to the examples provided in Robot Learning: A Tutorial (thanks @fracapuano for the amazing work!).

For example, I started the finetuning using:

python src/lerobot/scripts/lerobot_train.py \
        --policy.path=HuggingFaceVLA/smolvla_libero \
        --policy.repo_id=RonPlusSign/smolvla_PutRubbishInBin \
        --dataset.repo_id=RonPlusSign/RLBench-LeRobot-v3-PutRubbishInBin \
        --batch_size=32  \
        --output_dir=outputs/train/smolvla_finetuned_rubbish \
        --policy.device=cuda \
        --wandb.enable=true \
        --save_freq=10000 \
        --steps=60000

I also tested smaller finetunings (e.g. 5k, 10k, 20k steps).

Here are some specific points I’d like to clarify:

1. What are the exact action and state spaces used in SmolVLA and π₀ pretraining? (ΔEEF pose, absolute EEF pose, joint positions, joint velocities, ... and angle representations e.g. quaternion or Euler).

2. Regarding camera inputs: does the naming or number of cameras affect the model performance? Should I stick to the exact names provided in the config.json file, such as observation.images.image and observation.images.image2 (front/wrist), similar to pretraining? Or is it fine to use different camera names and/or add extra views? Is there a way to override the existing input and output features or this means that the pretrain would be wasted?

3. The base model lerobot/smolvla_base is pretrained on the SO100/SO101 robot, so I assume it might not transfer well to Franka Panda tasks — is that correct?

4. Would it make more sense to start from a model trained on Franka, e.g. HuggingFaceVLA/smolvla_libero, or it's still a different type of embodiment (it seems with 6DoF+gripper, which is not my case)?

5. Are the datasets HuggingFaceVLA/libero and/or HuggingFaceVLA/smol-libero the ones used for pretraining HuggingFaceVLA/smolvla_libero?

6. In HuggingFaceVLA/smol-libero the actions have dimension 7, which doesn’t clearly map to 7 joint angles + gripper. Are these absolute joint positions, EEF poses, or something else? Does LIBERO use a 6DoF or 7DoF Franka setup? If 6DoF, which joint is excluded?

Any guidance on these points (or pointers to where this information is documented) would be very helpful — I’ve been trying to align my setup with the pretrained models but haven’t found clear references for these details.

Thanks a lot for your time and for maintaining this project!

[fracapuano]

ciao @RonPlusSign 👋 will get into this myself very soon. In the meantime, what do you think of adding RLBench to lerobot? You could contribute the environment similiarily to how we're currently supporting LIBERO and Meta-World. That would make the benchmark directly compatible with lerobot, so that you (and the community!) could test directly any policy and iteration on this benchmark without the need of having a script. Wdyt? Up for the challenge? 😊

If you think that's something you could do I'd suggest pinging myself and @jadechoghari (the simulation goat in the lerobot team!)

[RonPlusSign]

I can try to work on a PR that adds RLBench support for lerobot in my spare time!
In the meantime I'd love to understand how to finetune SmolVLA correctly on different input/output features, action spaces, and embodiments. Thanks again for your time 😊

[haomin1010]

@RonPlusSign Hi, i saw you asked this question:
"Are the datasets [HuggingFaceVLA/libero] and/or [HuggingFaceVLA/smol-libero] the ones used for pretraining [HuggingFaceVLA/smolvla_libero]"

Now whether you have answer? I want to train a model similar to HuggingFaceVLA/smolvla_libero but failed.

Look forward to your reply!

[RonPlusSign]

Hi @haomin1010, unfortunately I still don't have an answer to any of the questions I asked, I hope someone from the community will answer soon!

[haomin1010]

@RonPlusSign Thanks, I would update in this issue if i have more information! Lerobot SmolVLA is a little difficult to train hhh

[RonPlusSign]

@fracapuano hi, any updates on this?

[RonPlusSign]

@jadechoghari I started working on a PR to add RLBench simulator support on LeRobot (see my fork for details), but I'm still facing issues with graphics libraries and multithreading with CoppeliaSim, and currently I'm a bit stuck. Should I open a PR anyway?

[jadechoghari]

@RonPlusSign that's awesome, and yes feel free to open a PR anyways so we have a look 🤗

[RonPlusSign]

I'm particularly confused about input_features and output_features.
I trained a diffusion and a smolvla model on a dataset I collected, containing the following features:

• observation.state: dim=7, [x, y, z, roll, pitch, yaw, gripper_open], representing the current pose
• observation.state.joints: dim=7, [joint1, ..., joint7], representing the current joint positions
• observation.images.left_shoulder_rgb, observation.images.right_shoulder_rgb, observation.images.front_rgb, observation.images.wrist_rgb, observation.images.overhead_rgb: shape=(3,256,256), representing visual observations
• action: dim=7, [Δx, Δy, Δz, Δroll, Δpitch, Δyaw, gripper_open], which is the delta movement the end effector has to perform

I've run the training using the following commands:

python src/lerobot/scripts/lerobot_train.py \
        --policy.type=diffusion \
        --policy.repo_id="RonPlusSign/diffusion_PutRubbishInBin" \
        --dataset.repo_id=RonPlusSign/RLBench-LeRobot-v3-PutRubbishInBin \
        --batch_size=64  \
        --output_dir=outputs/train/diffusion_rubbish \
        --policy.n_action_steps=10 \
        --policy.device=cuda \
        --wandb.enable=true \
        --save_freq=10000 \
        --steps=100000

python src/lerobot/scripts/lerobot_train.py \
        --policy.type=smolvla \
        --policy.repo_id="RonPlusSign/smolvla" \
        --dataset.repo_id=RonPlusSign/RLBench-LeRobot-v3-PutRubbishInBin \
        --batch_size=32  \
        --output_dir=outputs/train/smolvla_fulltrain_rubbish \
        --policy.n_action_steps=10 \
        --policy.device=cuda \
        --wandb.enable=true \
        --save_freq=10000 \
        --steps=60000

so, without specifying a pretrained model (no --policy.path) (does this mean I'm training from scratch?). The trainings run smoothly:

But the resulting models (diffusion, smolvla) have different input_features from the ones specified in the dataset!

In particular analyzing their config.json, they don't seem to receive observation.state.joints as input, and the observation.state has shape 8 instead of 7.

// ...
"input_features": {
    "observation.state": {
        "type": "STATE",
        "shape": [ 8 ]     // Why not 7 as in the training dataset?
    },
    // ... No observation.state.joints
}
// ...

And during inference, the policies predict only one action at a time, even though I specified --policy.n_action_steps=10 during training.

[fracapuano]

Hey @RonPlusSign 👋

I've run the training using the following commands:

Why don't you try to re-run training (even a short training run, just to make sure everything works accordingly) following our examples on how to train a diffusion model on your own dataset? e.g., training diffusion

[RonPlusSign]

Hey @fracapuano, thanks for the answer!
I thought the official/correct way of training was using lerobot-train, as reported in all model cards and in the documentation: https://huggingface.co/docs/lerobot/il_robots#train-a-policy

How is the script you linked different from what the src/lerobot/scripts/lerobot_train.py does?
Also, I tried a diffusion model just for testing purposes, but I'd love to use SmolVLA and Pi0 models instead.