Support for LeRobot

[koonpeng]

This PR adds the necessary tooling to run rosetta's bag_to_lerobot.py script.

• rosetta is only tested on humble, but I am able to run it on jazzy with a small patch fix episode_recorder in jazzy iblnkn/rosetta#3. Haven't tested, but the patch should work on kilted as well.
• The biggest challenge is mixing pypi packages and ros packages. aic_venv contains a custom build script that creates a venv and installs the required pypi packages on it (so far just lerobot). It exports aic_venv_python3 to run python scripts under the venv.
• aic_lerobot_tools/bag_to_lerobot is a wrapper to rosetta's bag_to_lerobot.py script. It is needed for a number of reasons.
  • bag_to_lerobot.py is not yet released so we need to run it from source.
  • it does not support dynamically loading decoders.
  • it needs lerobot which is only available on pypi.

So far, we can convert a rosbag2 recording of /joint_states into a lerobot dataset. Additional message types will require writing our own decoders.

[Yadunund]

@koonpeng this is great!

Regarding the virtual environment, i'm curious if we could leverage conda instead? In some previous testing, I also ran into the issue of dependency mismatch between pypi and apt. I resorted to creating a conda environment that worked well. Some instructions on what i did can be found here.

Do you think we can adopt a similar approach here? We expect participant nodes to be packaged within such virtual environments as well so it wouldn't just be useful for recording datasets.

[koonpeng]

The instructions linked installs ros inside the conda env. What we could also do is put the entire workspace inside a conda env and avoid apt altogether. Are we looking to support mac and windows users? If so, then I think we should go with conda. I can help test on windows if needed.

[koonpeng]

@Yadunund After testing out the conda setup, I think it will be very hard to use conda without using it on the whole workspace. The main problem is the dependencies conflict with the system packages outside the conda env. A package built inside conda env cannot be used outside the env and vice versa.

In the context of aic_lerobot_tools, that means that we will need to use a version of ros bag inside conda to record topics published by nodes running outside the conda env. There should be no issue for most standard msgs that are very stable. But for custom messages, we will need to build it inside the env as well. We can't share the colcon workspace so that means we now need to manage 2 different workspaces.

[Yadunund]

Thanks for trying that out. I agree having two separate workspaces isn't great. What are your suggestions for moving this forward? @codebot is working on an aic_model node that will likely run inside an conda environment and will need to subscribe and publish interface defined in this repo. So we'll need to build those interfaces within the conda workspace as well.

[koonpeng]

I think if we can build everything in conda then we should use that as the source of truth. If not then I think it depends on several factors. How are we releasing the sdk? Do participants build this repo from source, or we release compiled binaries or docker images? Or we only release parts of this repo?

---

On a separate note, I tried loading the recording in rerun.io but it only supports lerobot v2 dataset atm. Support for v3 should be landing "very soon" rerun-io/rerun#12071.

[codebot]

Yeah. I had assumed that we'd end up with two workspaces and two separate builds, but I haven't tried it yet.

1. a "participant model" workspace that must build inside conda due to needing many "non-ROS" dependencies which is challenging without conda. I think this would only have a handful of packages, like aic_model and a few custom interface packages.
2. a "world" or "workcell" workspace that builds GZ and the controllers, etc., (aic_bringup, etc.). This may, or may not, be built using conda; it's up to the user. If they are using a pre-built container image, they likely don't care either way.

I had assumed there would be a few packages, like those in the aic_interfaces repo, which are needed in both repos, but I think the number and size of overlapping packages would be small and not too time-consuming to build twice.

[Yadunund]

I had assumed there would be a few packages, like those in the aic_interfaces repo, which are needed in both repos, but I think the number and size of overlapping packages would be small and not too time-consuming to build twice.

This is a great point.

Let's move forward with the idea that:

1. For us developers (and those participants who wish to build from source), we will rely on two workspaces. One "native" workspace built from source using apt bianaries for the "world". And another conda workspace for the "participant model".
2. Most participants will work only in the conda workspace to integrate their models. They will run a docker image of the "world" workspace for their testing.

[koonpeng]

Moved aic_lerobot_tools to a separate workspace. Because we are attempting to represent 2 workspaces in one repo, the setup requires COLCON_IGNORE and symlinks.

---

I will probably be making another PR to integrate https://github.com/ycheng517/lerobot-ros. It seems like a easier way to work with lerobot. Instead of tools to convert between the 2 format, it creates a ros2 driver for lerobot.

[Yadunund]

I will probably be making another PR to integrate https://github.com/ycheng517/lerobot-ros. It seems like a easier way to work with lerobot. Instead of tools to convert between the 2 format, it creates a ros2 driver for lerobot

@koonpeng We can point users to such tooling but I'm not sure if we want to explicitly include it in our toolkit. Mostly because we're commanding robots through our own controllers here. What is valuable to have is a way to record rosbags, and export them to tabular formats for learning where one such format is Dataset V3.

[koonpeng]

@Yadunund It allows us to use all of lerobot's tools as well, including recording a dataset. So far I managed to control the robot through lerobot-teleoperate. Though cameras and the new parallel gripper controller is not supported yet.

[koonpeng]

I updated this to use pixi. pixi has better support for mixing pypi and "system" (ros) packages so we no longer need to setup a separate workspace. Things should just work with pixi run ....

[Yadunund]

Thanks for the updates, @koonpeng. I'll work with @grkw to test the new workflow.

[Yadunund]

This is looking great. Tested it out and apart from missing rmw_zenoh exported, things worked well. Left some suggestions.

[Yadunund]

Remove. No longer needed after #79

[koonpeng]

done

[Yadunund]

Suggested change

	<license>Apache License 2.0</license>
	<license>Apache-2.0</license>

[koonpeng]

done

[Yadunund]

Please expand on this note to highlight pixi run or pixi shell

[koonpeng]

This package is not included in the pixi workspace. I added a note to avoid using this due to the bugs present.

[Yadunund]

Add copyright header here and other files

[koonpeng]

done

[Yadunund]

I don't think we've setup the colcon graph correctly to build all GZ packages upto fizzgig_bringup. We should do that in a separate PR. For now, can we rely on --packages-ignore to ignore lerobot pkgs?

[koonpeng]

done

[Yadunund]

Add ros-kilted-rmw-zenoh-cpp

[koonpeng]

done

[Yadunund]

Add [activation] and [activation.env] blocks to 1) source RoboStack ROS and 2) set RMW_IMPLEMENTATION=rmw_zenoh_cpp

[koonpeng]

Added activation.env. We do not need to source robostack as it is installed in the environment.

[Yadunund]

hmm when I ran pixi shell and then ros2 , the environment couldn't find any ros2 cli tools 🤔

[koonpeng]

Yeah, that is "normal" since we didn't install any. I think pixi doesn't have a proper "dev dependencies" so we would have to add it to run dependencies even though it is technically not needed.

[Yadunund]

Suggested change

	rosdep install --from-paths src --ignore-src --rosdistro kilted -yr --skip-keys "gz-cmake3 DART libogre-dev libogre-next-2.3-dev"
	rosdep install --from-paths src --ignore-src --rosdistro kilted -yr --skip-keys "gz-cmake3 DART libogre-dev libogre-next-2.3-dev rosetta"

[koonpeng]

done

[Yadunund]

LGTM thanks for iterating.