finetuning on libero

[lzl2040]

When I try to finetune RDT on LIBERO, the loss curve is as follows:

Is this normal? My dataset preparation is as follows:

• complete the libero_vla_dataset.py based on hdf5_maniskill_dataset.py: Because the libero provides the actions and the states in the format of x, y, z, roll, pitch, yaw, gripper, I first transform it to 6D representation. Then I use the following state:

UNI_STATE_INDICES = [ STATE_VEC_IDX_MAPPING['eef_pos_x']
                            ] + [ STATE_VEC_IDX_MAPPING['eef_pos_y']
                            ] + [ STATE_VEC_IDX_MAPPING['eef_pos_z']
                    ] + [
                        STATE_VEC_IDX_MAPPING[f"eef_angle_{i}"] for i in range(6)
                    ] + [
                        STATE_VEC_IDX_MAPPING["right_gripper_open"]
                    ]

Is there any error?

[csuastt]

It is correct. How about the sampling error?

[Buming-O]

Hello! Thanks for your work. I'm also trying to finetuning LIBERO (using the original HDF5 version instead of the RLDS version provided by OpenVLA), and here are the current results. Can this be considered convergence?

[jasper0314]

Hi @lzl2040 and @Buming-O , could you please share the performance of RDT on LIBERO that you’ve tried? I’m considering using RDT, but I’m not sure how well it works in simulation environments like LIBERO or SimplerEnv. Thanks!