Provides capabilities2 runners for fabric stack. These are required by the fabric stack to be able to execute the plans generated by the LLM.
| Runner | Feature Focus | Description |
|---|---|---|
| Completion runner | System Functionality | informs fabric server about execution completion |
| Set plan runner | Plan Management | adds a new execution plan to the queue of Fabric |
Following examples depend on CollaborativeRoboticsLab/capabilities2 and CollaborativeRoboticsLab/prompt_tools. The examples have been tested against turtlebot3 simulation using CollaborativeRoboticsLab/turtlebot3-docker. This is compatible with any robot that uses Nav2 Stack.
| Example | Description |
|---|---|
| generative_1.xml | In this example (2.0, -0.5) point is reachable. The LLM would need to generate a plan that moves the robot to the point. |
| generative_2.xml | In this example (2.0, -0.5), (0.0, 2.0), (-2.0, 0.0), (0.0, -2.0) points are reachable. The LLM would need to generate a plan that moves the robot through the points in order. |
| generative_3.xml | In this example, the robot is suppoed to go through (2.0, -0.5), (1.0, 3.0), (-2.0, 0.0), (0.0, -2.0) points. Out of these (1.0, 3.0) is unreachable. The LLM would need to generate a plan that moves the robot through the points in order considering the unreachable point |
| generative_4.xml | In this example, the robot is suppoed to go through (2.0, -0.5), (1.0, 3.0), (-3.0, -1.0), (0.0, -2.0) points. Out of these (1.0, 3.0) and (-3.0, -1.0) are unreachable. So instead of (1.0, 3.0) robot will move to (0.0, 2.0). Instead of (-3.0, -1.0) the robot will move to (-2.0, 0.0). The LLM would need to generate a plan that moves the robot through the points in order considering the unreachable point. |
| generative_5.xml | In this example, the robot is suppoed to go through (2.0, -0.5), (1.0, 3.0), (-3.0, -1.0), (0.0, -2.0) points. Out of these (1.0, 3.0) and (-3.0, -1.0) are unreachable. So instead of (1.0, 3.0) robot will move to (0.0, 2.0). Instead of (-3.0, -1.0) the robot will move to (-2.0, 0.0). (0.0, -2.0) has a recovery location but the position is reachable. The LLM would need to generate a plan that moves the robot through the points in order considering the unreachable point. |
Note: The examples are designed to work and has been tested with the turtlebot3 world in Gazebo, but on the assumption that the start position of the robot is (0.0, 0.5) in gazebo simulation. In ROS
humblethis is the default, but injazzyyou might need to change the initial position of the robot in the world file to match this. If you use CollaborativeRoboticsLab/turtlebot3-docker, it is already set to this position.
Above examples depend on the capabilities2 and prompt_tools packages. You can clone these packages in your workspace and build them using colcon build.
cd ~/colcon_ws/src
git clone https://github.com/CollaborativeRoboticsLab/capabilities2.git
git clone https://github.com/CollaborativeRoboticsLab/fabric.git
git clone https://github.com/CollaborativeRoboticsLab/prompt_tools.git
git clone https://github.com/CollaborativeRoboticsLab/prompt_capabilities.git
git clone https://github.com/CollaborativeRoboticsLab/nav2_capabilities.git
git clone https://github.com/CollaborativeRoboticsLab/fabric_capabilities.git
cd ~/colcon_ws
colcon build --symlink-installTo setup the simulation, you can use the turtlebot3 world in Gazebo. You can launch the simulation using the following command,
git clone https://github.com/CollaborativeRoboticsLab/turtlebot3-docker.git
cd turtlebot3-docker/docker
docker compose pull
xhost +local:root
docker compose upTo run the examples, first make sure that the simulation is running and then on different terminals run,
source install/setup.bash
ros2 launch capabilities2_server capabilities2_server.launch.pyReplace <your_openai_api_key> with your actual OpenAI API key in the command below.
export OPENAI_API_KEY=<your_openai_api_key>
source install/setup.bash
ros2 launch prompt_bridge prompt_bridge.launch.pysource install/setup.bash
ros2 launch fabric_capabilities system.launch.py filename:=generative_1.xmlChange filename:=generative_1.xml to match the other plans as shown in above table to run different examples.