dataset

[Nimingez]

I want to use my own UR5 to collect data for wiping tables. Are there any data collection scripts and scripts to convert to RLDS available?

[kpertsch]

We don't have data collection scripts -- it probably depends a lot on your robot hardware.
A utility to convert your custom dataset to RLDS is here: https://github.com/kpertsch/rlds_dataset_builder

[Nimingez]

Test the installation by finetuning on the debug dataset, I just run python scripts/finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small --debug
It raise
ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.16; Detected an installation of version 2.15.0. Please upgrade TensorFlow to proceed.

[kpertsch]

Can you make sure that you are using the packages at the versions pinned in requirements.txt?
ie tensorflow_probability should be version 0.23.0
You can check your version via: pip freeze | grep tensorflow_probability

[Nimingez]

Do you know what 'action' represents? I actually didn't understand it.
Does it represent the difference between the xyzrpy of the robot arm end at this time step and the xyzrpy of the previous time step?
I printed out one of the trajectories.

[kpertsch]

Yes, the action should be the delta between the end-effector pose at this time step and the pose at the next time step.

[kpertsch]

(if the data is collected with non-blocking control this may not directly correspond to the actually achieved robot state at the next time step)

[Nimingez]

Do you know how I should get this action? In this sample dataset, subtracting two XYZ values does not yield the action.
this time step 2.45179296e-01, 1.25298291e-01, -6.78761229e-02,5.92707310e-01, 7.93969614e-01, -5.74835867e-03,-1.35193394e-01
next time step 2.45277494e-01, 1.25291064e-01, -6.78688809e-02,5.92718418e-01, 7.93973078e-01, -5.69880711e-03,-1.35126432e-01
action 0, 0.000228571429, 0, 0, 0, 0

[kpertsch]

The dataset already has a field called "action" and that contains the correct end-effector action.
The fact that the delta between consecutive positions is not equal to the action is expected and is what I meant with my comment above re non-blocking control. Non-blocking control means that the robot does not wait until the delta commanded by the action is achieved before executing the next action. Thus, the next robot position may not exactly achieved the commanded delta.

[Nimingez]

Thank you!
How many wiping table tasks should I collect?
After fine-tuning my own UR5 data, how should I deploy and evaluate it on the actual robot?

[kpertsch]

We have an example here for evaluating a finetuned model on the BridgeV2 WidowX robot, you can modify this for your UR5 setup: https://github.com/octo-models/octo/blob/main/examples/04_eval_finetuned_on_robot.py

Re number of demos: 50 is usually a good number for a single task, though you may want to collect more if your task is complex.

[Nimingez]

I have collected my ur dataset like this

when convert to rlds

where should i change?

File "/home/nmz/mgit/rlds_dataset_builder/UR5_Wipe/UR5_Wipe_dataset_builder.py", line 112, in _parse_example
for i, step in enumerate(data):
TypeError: iteration over a 0-d array

[kpertsch]

Looks like the data variable may be empty -- can you pdb into the parse_example function and check that data correctly loads your raw data?
You can follow the instructions for modifying the parse_example function in the README here: https://github.com/kpertsch/rlds_dataset_builder/blob/main/README.md

[Nimingez]

I have convert my dataset to rlds
when visualize dataset
The builder directory /home/nmz/tensorflow_datasets/u_r5__wipe doesn't contain any versions.
No builder could be found in the directory: /home/nmz/tensorflow_datasets for the builder: u_r5__wipe.
No registered data_dirs were found in:
- /home/nmz/tensorflow_datasets

[kpertsch]

Did you store your dataset into ~/tensorflow_datasets? If not, you need to add data_dir=path_to_data_root to the tfds.load() command. here the data root refers to the folder that contains u_r5__wipe

[Nimingez]

yes，  tfds build  store my dataset into ～/tensorflow_dataset发自我的 iPhone在 2024年5月27日，12:36，Karl Pertsch ***@***.***> 写道： Did you store your dataset into ~/tensorflow_datasets? If not, you need to add data_dir=path_to_data_root to the tfds.load() command. here the data root refers to the folder that contains u_r5__wipe —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[kpertsch]

What's the content of the directory ~/tensorflow_datasets/u_r5__wipe?

[Nimingez]

/home/nmz/tensorflow_datasets/u_r5__wipe_dataset
I can see your example dataset is also in the same location.

[kpertsch]

Can you copy the dataset one folder up, ie it should look like this: ~/tensorflow_datasets/<dataset_name>/1.0.0 and then you can load it with tfds.load(<dataset_name>)

[Nimingez]

Do you have plans to open source the RLDS build process for the UR5 berkeley_autolab_ur5?

[kpertsch]

No, unfortunately it will be hard to open-source that, but there is nothing UR5-specific in the conversion code -- you can just follow another example, eg here for Bridge data V2: https://github.com/kpertsch/bridge_rlds_builder/blob/main/bridge_dataset/bridge_dataset_dataset_builder.py (note that this repo is forked off of the multithreaded branch here: https://github.com/kpertsch/rlds_dataset_builder/tree/multithreaded)

[Nimingez]

Thank you for your patient response
When I initially did the installation test, there were no errors in fine-tuning, but now there are errors.
python finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 --debug
Traceback (most recent call last):
File "/home/nmz/mgit/octo/scripts/finetune.py", line 426, in
app.run(main)
File "/home/nmz/anaconda3/envs/octo-models/lib/python3.10/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/nmz/anaconda3/envs/octo-models/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/home/nmz/mgit/octo/scripts/finetune.py", line 130, in main
pretrained_model = OctoModel.load_pretrained(
File "/home/nmz/mgit/octo/octo/model/octo_model.py", line 259, in load_pretrained
module = OctoModule.create(**config["model"])
TypeError: OctoModule.create() got an unexpected keyword argument 'use_correct_attention'

[Nimingez]

Can you introduce the inputs and outputs in the UR5 wipe task? Is the input from one camera or two cameras?

[kpertsch]

It's single camera input and 7D end-effector action output

[Nimingez]

7D means x,y,z, qx,qy,qz,qw or x y z rx ry rz and gripper？

[kpertsch]

the latter

[Nimingez]

Does the seven dimensions represent the change of the end-effector relative to the previous frame? It's not the target position of the robotic arm?

[Nimingez]

my qq 670761972发自我的 iPhone在 2024年6月6日，19:05，liufeifei ***@***.***> 写道： Thank you for your patient response When I initially did the installation test, there were no errors in fine-tuning, but now there are errors. python finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 --debug Traceback (most recent call last): File "/home/nmz/mgit/octo/scripts/finetune.py", line 426, in app.run(main) File "/home/nmz/anaconda3/envs/octo-models/lib/python3.10/site-packages/absl/app.py", line 308, in run _run_main(main, args) File "/home/nmz/anaconda3/envs/octo-models/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/home/nmz/mgit/octo/scripts/finetune.py", line 130, in main pretrained_model = OctoModel.load_pretrained( File "/home/nmz/mgit/octo/octo/model/octo_model.py", line 259, in load_pretrained module = OctoModule.create(**config["model"]) TypeError: OctoModule.create() got an unexpected keyword argument 'use_correct_attention' Hi, I have the same problem. Have you solved it now? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you modified the open/close state.Message ID: ***@***.***>