lerobot-eval run with exception:

[talregev]

System Info

- lerobot version: 0.3.4
- libero  version: 0.1.1
- Mujoco  version: 3.3.7
- Platform: ubuntu 24.04
- Python version: 3.12
- Huggingface Hub version: 0.35.3
- Datasets version: 4.1.1
- Numpy version: 2.2.6
- PyTorch version: 2.7.1+cpu
- Using GPU in script: No

Information

• One of the scripts in the examples/ folder of LeRobot
• My own task or dataset (give details below)

Reproduction

pip install git+https://github.com/huggingface/lerobot.git#egg=lerobot[libero] accelerate num2words --extra-index-url https://download.pytorch.org/whl/cpu

    lerobot-eval \
        --policy.path=lerobot/smolvla_base \
        --policy.device=cpu \
        --env.type=libero \
        --env.task=libero_object \
        --eval.n_episodes=1 \
        --eval.batch_size=1 \
        --output vla-results/smolvla_libero_cpu.json

Error

Traceback (most recent call last): File "/home/runner/.local/bin/lerobot-eval", line 8, in <module> sys.exit(main()) ^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/scripts/lerobot_eval.py", line 756, in main eval_main() File "/home/runner/.local/lib/python3.12/site-packages/lerobot/configs/parser.py", line 225, in wrapper_inner response = fn(cfg, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/scripts/lerobot_eval.py", line 514, in eval_main info = eval_policy_all( ^^^^^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/scripts/lerobot_eval.py", line 703, in eval_policy_all tg, tid, metrics = task_runner(task_group, task_id, env) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/scripts/lerobot_eval.py", line 615, in run_one metrics = eval_one( ^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/scripts/lerobot_eval.py", line 569, in eval_one task_result = eval_policy( ^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/scripts/lerobot_eval.py", line 319, in eval_policy rollout_data = rollout( ^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/scripts/lerobot_eval.py", line 170, in rollout action = policy.select_action(observation) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/policies/smolvla/modeling_smolvla.py", line 305, in select_action actions = self._get_action_chunk(batch, noise) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/policies/smolvla/modeling_smolvla.py", line 258, in _get_action_chunk images, img_masks = self.prepare_images(batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/.local/lib/python3.12/site-packages/lerobot/policies/smolvla/modeling_smolvla.py", line 354, in prepare_images raise ValueError( ValueError: All image features are missing from the batch. At least one expected. (batch: dict_keys(['action', 'next.reward', 'next.done', 'next.truncated', 'info', 'task', 'observation.images.image', 'observation.images.image2', 'observation.state', 'observation.language.tokens', 'observation.language.attention_mask'])) (image_features:{'observation.images.camera1': PolicyFeature(type=<FeatureType.VISUAL: 'VISUAL'>, shape=(3, 256, 256)), 'observation.images.camera2': PolicyFeature(type=<FeatureType.VISUAL: 'VISUAL'>, shape=(3, 256, 256)), 'observation.images.camera3': PolicyFeature(type=<FeatureType.VISUAL: 'VISUAL'>, shape=(3, 256, 256))})

Workaround

I solve this issue with this workaround, but I still think it should be fixed.
Not sure if it correct. I added these lines to the original command:

            --policy.input_features='{"observation.images.image":  {"type":"VISUAL","shape":[3,256,256]}, "observation.images.image2": {"type":"VISUAL","shape":[3,256,256]}}' \
            --policy.output_features='{"action":{"type":"ACTION","shape":[7]}}' \

Expected behavior

The script should run without exceptions

I verify that this PR causing this regression
#1771

Can you create tags on the lerobot-libero repo?
https://github.com/huggingface/lerobot-libero/tags

[jadechoghari]

Hello ! First note that this is not suitable for libero as it was not trained speicfically for libero tasks, you shoudl either fine tune this checkpoint to the libero dataset but make sure the input and output naming feature match! see https://huggingface.co/docs/lerobot/en/libero#training-with-libero ; smolvla_base has a different camera naming than the libero dataset! But processor easily fix this here: https://huggingface.co/docs/lerobot/en/introduction_processors#processor-examples. I recommend using this checkpoint as it was trained for libero: https://huggingface.co/HuggingFaceVLA/smolvla_libero
Our TODO is to maybe add an exemple where we fine tune smolvla_base on libero and add an exemple on how to rename with processor!

[talregev]

Did you even read my all issue? I already have the workaround.
Also this error didn't appear a week/2 weeks ago, so it regression.
Also other people have the same issue as well:
#2247

[talregev]

I think the lerobot-eval can easily detect the policy input and output as before without specify it. It already was before and it was broken. This is the issue all about.

[imstevenpmwork]

Hey @talregev

We recently merged: #1771 which touches specifically that part of the code. My spider sense tells me that your issue might be related. As a quick test, could you please try to revert the changes in this PR and run again your setup to see if works without it ?

Thanks !

[talregev]

@imstevenpmwork You have strong spider sense! 🕷️
I take latest lerobot and revert the #1771 PR, and it work in this setup without the workaround.

Now the question how we can fix it? :)

[talregev]

from #1771 PR:
Maybe these dictionaries is not empty, even when the user not set for them?

    if not cfg.output_features:
    if not cfg.input_features:

[talregev]

I create some prints. run latest lerobot without the revert without the workaround:
print(f"type of cfg.output_features: {type(cfg.output_features)}")
print(f"type of cfg.input_features: {type(cfg.input_features)}")
print(f"value of cfg.output_features: {cfg.output_features}")
print(f"value of cfg.input_features: {cfg.input_features}")

output:
type of cfg.output_features: <class 'dict'>
type of cfg.input_features: <class 'dict'>
value of cfg.output_features: {'action': PolicyFeature(type=<FeatureType.ACTION: 'ACTION'>, shape=(6,))}
value of cfg.input_features: {'observation.state': PolicyFeature(type=<FeatureType.STATE: 'STATE'>, shape=(6,)), 'observation.images.camera1': PolicyFeature(type=<FeatureType.VISUAL: 'VISUAL'>, shape=(3, 256, 256)), 'observation.images.camera2': PolicyFeature(type=<FeatureType.VISUAL: 'VISUAL'>, shape=(3, 256, 256)), 'observation.images.camera3': PolicyFeature(type=<FeatureType.VISUAL: 'VISUAL'>, shape=(3, 256, 256))}

Where are coming these values / defaults?

[imstevenpmwork]

I'm guessing is coming from here:

lerobot/src/lerobot/configs/eval.py

Line 45 in 9db6213

self.policy = PreTrainedConfig.from_pretrained(policy_path, cli_overrides=cli_overrides)

Meaning is loading these ones: https://huggingface.co/lerobot/smolvla_base/commit/4d2f2b37fa245361ef1efe6d91ce96b8bd4af511#d2h-025836

[talregev]

Can we have a flag to override policies input and output? For fresh start?

[imstevenpmwork]

With #2265 you can now do:

lerobot-eval \
  ... (your flags) \
  --rename_map={"observation.images.camera1": "observation.images.your_camera"} # remap the base model keys to your keys

[talregev]

Thanks for the PR! I will try it out and let you know.