Data size discrepancy between openvla/modified_libero_rlds and cleaned HDF5 converted with libero2lerobot

[aiming1998]

Hello,

Thank you for your great work. I have been comparing two versions of the Libero dataset:

The dataset from Hugging Face: openvla/modified_libero_rlds

A dataset I generated by first cleaning the original Libero HDF5 files, and then converting them using the libero2lerobot script you provided.

I noticed that the dataset obtained from your libero2lerobot conversion has more data than the one from Hugging Face.

Could you please clarify what causes this difference in dataset size?
Is it due to filtering/cleaning steps applied differently, or some preprocessing choices in the Hugging Face OpenVLA version vs. libero2lerobot?

Thanks a lot, and I’m looking forward to your reply.

[Tavish9]

Hi, @aiming1998

The script for conversion does not affect the size of the dataset. The only reason for the inconsistency is the filtering you applied.

You can check the difference between your filtered version in h5 format and the openvla's version in rlds format.

BTW, if you want to directly use openvla's rlds version, try openx2lerobot, you don't need to filter the dataset yourself.