This repository provides a ROS 2 package for generating explanations in autonomous robots based on log analysis using LLMs.
Explainable_ros uses the RAG method to filter the most relevant logs from those generated by the robot during the execution of its behavior.
To enhance the robot's internal data, a VLM is used to process the images captured by the onboard camera, describe them, and log them through rosout. This allows the combination of logs generated by the robot subsystems with images of the environment. The workflow of the system is illustrated in the following figure:
On the other hand, the high-level representation of the components that make up the developed system is shown in the following figure.
To run llama_ros with CUDA, first, you must install the CUDA Toolkit. Then, you can continue with the installation.
cd ~/ros2_ws/src
git clone https://github.com/mgonzs13/llama_ros.git
git clone https://github.com/Dsobh/explainable_ros.git
pip3 install -r llama_ros/requirements.txt
cd ~/ros2_ws
rosdep install --from-paths src --ignore-src -r -y
colcon build --cmake-args -DGGML_CUDA=ON # add this for CUDA- Run the launch file with the emebedding, reranker and LLM models:
ros2 launch explainable_ros_bringup explainable_ros.launch.pyThe explainable node subscribes to the rosout topic and process the logs to add them to the context of the LLM. You can play a rosbag file in order to generate logs and test the operation of the system.
- To request a explanation you should use the /question service:
ros2 service call /question explainable_ros_msgs/srv/Question "{'question': 'What is happening?'}"Or you can use the explainability client:
ros2 run explainable_ros explainability_client_node "What is happening?"For the examples shown in this section we use the following models:
- Embbeding model: bge-base-en-v1.5
- Reranker model: jina-reranker
- Base model: Qwen2.5-Coder-7B-Instruct
First of all, download the bags. Then, launch the explainable_ros bringup:
ros2 launch explainable_ros_bringup explainable_ros.launch.pyThen, play one of the bags, for example, bag 4:
ros2 bag play Prueba4Finally, wait till all logs are processed and stored and you can ask questions:
ros2 run explainable_ros_demos explainable_demo_nodeHere you have the results for each bag using specific questions from demo.
Click to expand
| User Question | LLM Answer |
|---|---|
| How many waypoints were received during the navigation task? | Based on the logs provided, the robot received a total of 3 waypoints during the navigation task. These waypoints were identified by their IDs: 9, 6, and 7. |
| What were the IDs of the waypoints received during the navigation task? | The IDs of the waypoints received during the navigation task were 9, 6, and 7. |
| Were all the waypoints received successfully reached? | Yes, all the waypoints received were successfully reached. The logs indicate that the robot received waypoints with IDs 9, 6, and 7, and after reaching each waypoint, it continued to wait for the next one. The final log entry states that "All the waypoints received have been reached. Navigation task completed," confirming that the robot successfully navigated to all the waypoints. |
| What happened during navigation to waypoint with ID 6? | During the navigation to waypoint with ID 6, the robot successfully reached the waypoint. However, there was an obstacle detected during the navigation process. The distance to the obstacle increased from 0.12 meters to 0.16 meters before the robot successfully navigated past it. |
| Why was the route replanned during navigation to waypoint with ID 6? | The route was not explicitly replanned during the navigation to waypoint with ID 6. However, the logs indicate that an obstacle was detected during the navigation, which caused the robot to stop and wait for the obstacle to clear before continuing. This is a common behavior for mobile robots equipped with obstacle detection sensors. The robot's navigation stack likely includes a mechanism to handle obstacles, such as pausing the navigation, adjusting the path, and retrying the navigation once the obstacle is no longer in the way. |
| Have any relevant events occurred during navigation? | Yes, relevant events have occurred during the navigation. Here are the key events: 1. Navigation to Waypoint 9: - Started navigation from (7.53, 5.17) to (1.55, 4.03). - Successfully reached waypoint 9. 2. Navigation to Waypoint 6: - Started navigation from (1.53, 4.16) to (4.89, 1.64). - Detected an obstacle during navigation, with the distance to the point increasing from 0.12 meters to 0.16 meters. - Successfully reached waypoint 6. 3. Navigation to Waypoint 7: - Detected an obstacle during navigation, with the distance to the point increasing from 0.03 meters to 0.09 meters. 4. Completion of Navigation Task: - All waypoints have been reached. - Navigation task completed successfully. These events indicate that the robot encountered obstacles during its navigation and successfully navigated to all waypoints. |
| What is the task that the robot had to perform? | The robot had to perform a navigation task. It received multiple waypoints and navigated to each one, reaching each goal before moving on to the next. The task was completed when all the waypoints were reached. |
| Did the robot avoid any obstacle during the navigation? | Yes, the robot avoided obstacles during the navigation. The logs indicate that the robot detected obstacles and adjusted its path accordingly. Specifically: 1. Obstacle Detected at Waypoint 6: - The robot detected an obstacle during the navigation to waypoint with ID: 6. - The distance to the obstacle increased from 0.12 meters to 0.16 meters, indicating that the robot moved away from the obstacle. - The robot successfully navigated to waypoint with ID: 6. 2. Obstacle Detected at Waypoint 7: - The robot detected an obstacle during the navigation to waypoint with ID: 7. - The distance to the obstacle increased from 0.03 meters to 0.09 meters, indicating that the robot moved away from the obstacle. - The robot passed a new path to the controller, suggesting that it adjusted its path to avoid the obstacle. In both cases, the robot successfully navigated past the obstacles, demonstrating its ability to avoid them during the navigation process. |
Click to expand
| User Question | LLM Answer |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | ----------------- | -------------- |
| How many waypoints were received during the navigation task? | Based on the logs provided, the robot received a total of 3 waypoints during the navigation task. These waypoints were identified by IDs: 9, 6, and 7. |
| What were the IDs of the waypoints received during the navigation task? | The IDs of the waypoints received during the navigation task were 9, 6, and 7. |
| Were all the waypoints received successfully reached? | Yes, all the waypoints received were successfully reached. The logs indicate that the robot received waypoints with IDs 9, 6, and 7, and after reaching each waypoint, it continued to wait for a new one. The final log entry states that "All the waypoints received have been reached. Navigation task completed," confirming that the navigation task was successfully completed. |
| What happened during navigation to waypoint with ID 6? | During the navigation to waypoint with ID 6, the robot successfully reached the waypoint. The logs indicate that the robot started navigating to waypoint 6 at 1699877457.3765593 and confirmed that the navigation was in progress at 1699877457.3810558. The navigation to waypoint 6 was completed successfully at 1699877503.3723066. |
| Why was the route replanned during navigation to waypoint with ID 6? | Based on the logs provided, there is no explicit mention of a route replanning during the navigation to waypoint with ID 6. The logs indicate that the robot successfully navigated to waypoint 6 without any replanning. The logs show that the robot received waypoint 6, started navigating to it, and successfully reached it. There is no indication of any issues or replanning events during this process. |
| Have any relevant events occurred during navigation? | Yes, relevant events have occurred during the navigation. Here are the key events: 1. Navigation to Waypoint 9: - Started: The robot began navigation to waypoint with ID: 9 at 1699877403.3216784. - Completed: The robot successfully reached waypoint with ID: 9 at 1699877446.3722014. 2. Navigation to Waypoint 6: - Started: The robot began navigation to waypoint with ID: 6 at 1699877457.3810558. - Completed: The robot successfully reached waypoint with ID: 6 at 1699877503.3723066. 3. Navigation to Waypoint 7: - Started: The robot began navigation to waypoint with ID: 7 at 1699877513.4374268. - Obstacle Detected: The robot detected an obstacle during navigation to waypoint with ID: 7. The distance to the point increased from 0.11 meters to 0.16 meters at 1699877542.402691. 4. Completion of Navigation Task: - All Waypoints Reached: The robot has reached all the waypoints it was assigned to, and the navigation task is completed at 1699877553.4754102. 5. Lifecycle Manager Shutdown: - The robot is in the process of shutting down the Nav2 LifecycleManager, LifecycleNode for waypoint_follower, and LifecycleNode for bt_navigator at 1699877578.810685 and 1699877579.2267847, respectively. These events provide a clear timeline of the robot's navigation activities, including successful waypoint completions and an obstacle detection during the navigation to waypoint 7. |
| What is the task that the robot had to perform? | Based on the logs provided, it appears that the robot had to perform two main tasks: 1. Navigation to a Goal: - The logs show that the robot received a goal and began computing control effort at timestamps 1699877403.3742614 and 1699877513.4678938. This indicates that the robot was navigating to a specific location. - The robot successfully reached the goal at 1699877446.3323631. 2. Cleaning Up: - After reaching the goal, the robot started cleaning up at 1699877579.2269316. The logs show multiple instances of the robot cleaning up, with the final completion at 1699877579.6445086. Therefore, the robot had to perform a navigation task to reach a goal and then perform a cleaning-up task after reaching that goal. |
| Did the robot avoid any obstacle during the navigation? | Yes, the robot avoided an obstacle during the navigation. The logs indicate that the robot detected an obstacle while navigating to waypoint with ID: 7. The distance to the point increased from 0.11 meters to 0.16 meters, suggesting that the robot adjusted its path to avoid the obstacle. After avoiding the obstacle, the robot successfully navigated to waypoint with ID: 7. | | User Question | LLM Answer |
| ------------------- | ---------------- |
| How many waypoints were received during the navigation task? | Based on the logs provided, the robot received a total of 3 waypoints during the navigation task. These waypoints were identified by IDs: 9, 6, and 7. |
| What were the IDs of the waypoints received during the navigation task? | The IDs of the waypoints received during the navigation task were 9, 6, and 7. |
| Were all the waypoints received successfully reached? | Yes, all the waypoints received were successfully reached. The logs indicate that the robot received waypoints with IDs 9, 6, and 7, and after reaching each waypoint, it continued to wait for a new one. The final log entry states that "All the waypoints received have been reached. Navigation task completed," confirming that the navigation task was successfully completed. |
| What happened during navigation to waypoint with ID 6? | During the navigation to waypoint with ID 6, the robot successfully reached the waypoint. The logs indicate that the robot started navigating to waypoint 6 at 1699877457.3765593 and confirmed that the navigation was in progress at 1699877457.3810558. The navigation to waypoint 6 was completed successfully at 1699877503.3723066. |
| Why was the route replanned during navigation to waypoint with ID 6? | Based on the logs provided, there is no explicit mention of a route replanning during the navigation to waypoint with ID 6. The logs indicate that the robot successfully navigated to waypoint 6 without any replanning. The logs show that the robot received waypoint 6, started navigating to it, and successfully reached it. There is no indication of any issues or replanning events during this process. |
| Have any relevant events occurred during navigation? | Yes, relevant events have occurred during the navigation. Here are the key events: 1. Navigation to Waypoint 9: - Started: The robot began navigation to waypoint with ID: 9 at 1699877403.3216784. - Completed: The robot successfully reached waypoint with ID: 9 at 1699877446.3722014. 2. Navigation to Waypoint 6: - Started: The robot began navigation to waypoint with ID: 6 at 1699877457.3810558. - Completed: The robot successfully reached waypoint with ID: 6 at 1699877503.3723066. 3. Navigation to Waypoint 7: - Started: The robot began navigation to waypoint with ID: 7 at 1699877513.4374268. - Obstacle Detected: The robot detected an obstacle during navigation to waypoint with ID: 7. The distance to the point increased from 0.11 meters to 0.16 meters at 1699877542.402691. 4. Completion of Navigation Task: - All Waypoints Reached: The robot has reached all the waypoints it was assigned to, and the navigation task is completed at 1699877553.4754102. 5. Lifecycle Manager Shutdown: - The robot is in the process of shutting down the Nav2 LifecycleManager, LifecycleNode for waypoint_follower, and LifecycleNode for bt_navigator at 1699877578.810685 and 1699877579.2267847, respectively. These events provide a clear timeline of the robot's navigation activities, including successful waypoint completions and an obstacle detection during the navigation to waypoint 7. |
| What is the task that the robot had to perform? | Based on the logs provided, it appears that the robot had to perform two main tasks: 1. Navigation to a Goal: - The logs show that the robot received a goal and began computing control effort at timestamps 1699877403.3742614 and 1699877513.4678938. This indicates that the robot was navigating to a specific location. - The robot successfully reached the goal at 1699877446.3323631. 2. Cleaning Up: - After reaching the goal, the robot started cleaning up at 1699877579.2269316. The logs show multiple instances of the robot cleaning up, with the final completion at 1699877579.6445086. Therefore, the robot had to perform a navigation task to reach a goal and then perform a cleaning-up task after reaching that goal. |
| Did the robot avoid any obstacle during the navigation? | Yes, the robot avoided an obstacle during the navigation. The logs indicate that the robot detected an obstacle while navigating to waypoint with ID: 7. The distance to the point increased from 0.11 meters to 0.16 meters, suggesting that the robot adjusted its path to avoid the obstacle. After avoiding the obstacle, the robot successfully navigated to waypoint with ID: 7. |
Click to expand
| User Question | LLM Answer |
|---|---|
| How many waypoints were received during the navigation task? | Based on the logs provided, the robot received a total of 3 waypoints during the navigation task. These waypoints were identified by their IDs: 9, 6, and 7. |
| What were the IDs of the waypoints received during the navigation task? | The IDs of the waypoints received during the navigation task were 9, 6, and 7. |
| Were all the waypoints received successfully reached? | Yes, all the waypoints received were successfully reached. The logs indicate that the robot received waypoints with IDs 9, 6, and 7, and after reaching each waypoint, it continued to wait for the next one. The final log entry states that "All the waypoints received have been reached. Navigation task completed," confirming that the robot successfully navigated to all the waypoints. |
| What happened during navigation to waypoint with ID 6? | During the navigation to waypoint with ID 6, the robot successfully reached the waypoint without encountering any obstacles. The logs indicate that the robot received the waypoint with ID 6 and immediately started navigating to it. There is no mention of any obstacles or issues during this navigation, suggesting that the path to waypoint 6 was clear and the robot was able to proceed smoothly to its destination. |
| Why was the route replanned during navigation to waypoint with ID 6? | The route was not replanned during the navigation to waypoint with ID 6. The logs indicate that the robot successfully navigated to waypoint 6 without any replanning. The logs show that after reaching waypoint 6, the robot was waiting for a new waypoint, as indicated by the message "Waiting for a new waypoint...". There is no mention of replanning in the logs provided. |
| Have any relevant events occurred during navigation? | Yes, relevant events have occurred during the navigation. Here are the key events: 1. Navigation Initiation: The robot began navigating from one location to another, as indicated by timestamps such as 1699878817.0609927 and 1699878870.1231549. 2. Waypoint Progress: The robot was navigating to specific waypoints, identified by their IDs. For example, it was navigating to waypoint with ID: 9 and then to waypoint with ID: 7. 3. Obstacle Detection: The robot detected obstacles during its navigation. Specifically: - At 1699878858.4495645, it detected an obstacle during the navigation to waypoint with ID: 9, with the distance to the point increasing from 0.23 meters to 0.30 meters. - At 1699878928.3215034, it detected an obstacle during the navigation to waypoint with ID: 7, with the distance to the point increasing from 0.20 meters to 5.22 meters. 4. Navigation Success: The robot successfully navigated to the waypoints with IDs 9 and 7, as indicated by timestamps such as 1699878859.4943073 and 1699878928.2011423. 5. Navigation Completion: The robot completed the navigation task after reaching all the waypoints, as indicated by the timestamp 1699878971.243703. These events provide a clear sequence of actions and outcomes during the robot's navigation task. |
| What is the task that the robot had to perform? | The robot had to perform a navigation task, which involved reaching a series of waypoints. The logs indicate that the robot received multiple goals, computed control efforts to move towards these goals, and successfully reached each waypoint. The task was completed when all the waypoints were reached. |
| Did the robot avoid any obstacle during the navigation? | Yes, the robot avoided an obstacle during the navigation. Here's the breakdown of the logs: 1. First Navigation Attempt: - The robot started navigating from (0.93, 8.58) to (7.50, 4.89). - It passed a new path to the controller. - An obstacle was detected during the navigation to waypoint with ID: 9. The distance to the point increased from 0.23 meters to 0.30 meters. - The robot passed a new path to the controller again. - The navigation to waypoint with ID: 9 succeeded. 2. Second Navigation Attempt: - The robot started navigating from (1.49, 4.22) to (4.89, 1.64). - It passed a new path to the controller. - An obstacle was detected during the navigation to waypoint with ID: 7. The distance to the point increased from 0.20 meters to 5.22 meters. - The navigation to waypoint with ID: 7 succeeded. In both cases, the robot successfully navigated to the waypoints despite encountering obstacles. The fact that the robot passed a new path to the controller after detecting an obstacle indicates that it adapted its path to avoid the obstacle and continue the navigation. |
Click to expand
| User Question | LLM Answer |
|---|---|
| How many waypoints were received during the navigation task? | Based on the logs provided, the robot received a total of 3 waypoints during the navigation task. These waypoints were identified by their IDs: 9, 6, and 7. |
| What were the IDs of the waypoints received during the navigation task? | The IDs of the waypoints received during the navigation task were 9, 6, and 7. |
| Were all the waypoints received successfully reached? | Yes, all the waypoints received were successfully reached. The logs indicate that the robot received waypoints with IDs 9, 6, and 7, and after reaching each waypoint, it continued to wait for a new one. The final log entry states that "All the waypoints received have been reached. Navigation task completed," confirming that the robot successfully navigated to all the waypoints. |
| What happened during navigation to waypoint with ID 6? | During the navigation to waypoint with ID 6, the robot encountered an obstacle. The distance to the point increased from 5.78 meters to 14.65 meters, indicating that the robot had to adjust its path to avoid the obstacle. Despite the obstacle, the robot successfully navigated to the waypoint with ID 6. |
| Why was the route replanned during navigation to waypoint with ID 6? | The route was replanned during navigation to waypoint with ID 6 because an obstacle was detected. The logs indicate that the robot initially started navigating to waypoint 6, but as it approached, it detected an obstacle, causing the distance to the point to increase significantly from 5.78 meters to 14.65 meters. This obstacle detection triggered the robot to replan its route to avoid the obstacle and continue its navigation to the intended waypoint. Once the obstacle was avoided, the robot successfully navigated to waypoint 6. |
| Have any relevant events occurred during navigation? | Yes, relevant events have occurred during navigation. Specifically, the robot encountered obstacles while navigating to certain waypoints. Here are the details: 1. Obstacle Detected during Navigation to Waypoint with ID: 6: - The robot started navigating from (7.51, 5.03) to (1.55, 4.03). - An obstacle was detected, causing the distance to the point to increase from 5.78 meters to 14.65 meters. - Despite the obstacle, the robot successfully navigated to the waypoint with ID: 6. 2. Obstacle Detected during Navigation to Waypoint with ID: 7: - The robot started navigating from (1.50, 3.82) to (4.89, 1.64). - An obstacle was detected, causing the distance to the point to increase from 0.15 meters to 0.21 meters. - Despite the obstacle, the robot successfully navigated to the waypoint with ID: 7. These events indicate that the robot's navigation system is functioning as intended, as it is able to detect obstacles and continue navigating to the next waypoint. |
| What is the task that the robot had to perform? | Based on the logs provided, it appears that the robot was performing a task involving multiple goal points or waypoints. The logs indicate that the robot received multiple goals, computed control effort for each, and successfully reached each goal. The robot also seems to be waiting for a new waypoint after reaching a goal, suggesting that it is following a sequence of waypoints or goals. |
| Did the robot avoid any obstacle during the navigation? | Yes, the robot avoided obstacles during the navigation. The logs indicate that the robot detected obstacles and successfully navigated around them. Specifically: 1. Waypoint ID: 6 - The robot detected an obstacle and increased the distance to the point from 5.78 meters to 14.65 meters before successfully reaching the waypoint. 2. Waypoint ID: 7 - The robot detected a smaller obstacle and increased the distance to the point from 0.15 meters to 0.21 meters before successfully reaching the waypoint. These observations show that the robot's navigation system was able to detect obstacles and adjust its path accordingly, ensuring that it avoided them during the navigation process. |
Click to expand
| User Question | LLM Answer |
|---|---|
| How many waypoints were received during the navigation task? | Based on the logs provided, the robot received a list of waypoints and then proceeded to navigate to each waypoint. The logs indicate that the waypoints received were: 1. Waypoint with ID: 9 2. Waypoint with ID: 6 3. Waypoint with ID: 7 The logs also show that the robot was in the process of navigating to waypoint ID 9 when an obstacle was detected. After the obstacle was detected, the robot continued to navigate to waypoint ID 9, but the logs do not indicate that the robot received any new waypoints after that. Therefore, the number of waypoints received during the navigation task is 3. |
| What were the IDs of the waypoints received during the navigation task? | The IDs of the waypoints received during the navigation task were 9, 6, and 7. |
| Were all the waypoints received successfully reached? | Based on the logs provided, it appears that the robot received and processed the waypoints successfully. Here's a breakdown of the relevant information: 1. The robot subscribed to the /waypoints and /waypoint_follower/transition_event topics. 2. At 1701166121.2646344, a list of waypoints was received, and the waypoints were listed as 9, 6, and 7. 3. The robot then processed each waypoint, confirming receipt of the waypoint with ID 9. 4. The logs indicate that the robot completed the spin and wait processes successfully. 5. At 1701166339.2856762, the robot is running the Nav2 LifecycleNode for the waypoint_follower. 6. The robot is then waiting for a new waypoint, and at 1701166341.146829, it is cleaning up the waypoint_follower. Given these logs, it seems that the robot successfully received and processed the waypoints. However, the logs do not explicitly state whether the robot reached each waypoint. To determine if all waypoints were reached successfully, you would need additional information, such as the robot's position relative to the waypoints or any error messages related to waypoint navigation. |
| What happened during navigation to waypoint with ID 6? | Based on the logs provided, there is no information about the navigation to waypoint with ID 6. The logs only mention the navigation to waypoint with ID 9 and the obstacle detected during that navigation. There is no mention of waypoint with ID 6 in the logs. |
| Why was the route replanned during navigation to waypoint with ID 6? | Based on the provided logs, there is no explicit mention of the route being replanned during the navigation to waypoint with ID 6. The logs indicate that the robot is navigating to waypoints 9, 6, and 7 in sequence. The only obstacle detection is related to waypoint 9, and there is no information about a replanning event for waypoint 6. If you have additional logs or context that might indicate a replanning event for waypoint 6, please provide them, and I can assist you further. Otherwise, based on the given information, there is no evidence of a route replanning during the navigation to waypoint with ID 6. |
| Have any relevant events occurred during navigation? | Yes, relevant events have occurred during the navigation to waypoint with ID: 9. Here is a summary of the events: 1. Navigation Start: The robot started navigating to waypoint with ID: 9 at 1701166121.270141. 2. Obstacle Detection: The robot detected obstacles multiple times during the navigation: - At 1701166147.5558825, the distance to the obstacle increased from 4.11 meters to 16.57 meters. - At 1701166201.8920684, the distance to the obstacle increased from 4.53 meters to 15.87 meters. - At 1701166256.8877797, the distance to the obstacle increased from 3.98 meters to 16.83 meters. - At 1701166312.9179695, the distance to the obstacle increased from 4.36 meters to 15.95 meters. 3. Path Update: The robot passed a new path to the controller at 1701166200.9238155. 4. Navigation Abortion: The navigation to waypoint with ID: 9 was aborted at 1701166339.3121033. 5. Lifecycle Manager Shutdown: The Nav2 LifecycleManager was running a preshutdown at 1701166339.491577. 6. State Change Failure: The state change for the node bt_navigator failed at 1701166340.4311814. 7. Plugin Cleanup: The GridBased plugin of type NavfnPlanner was cleaned up at 1701166341.2137623. These events indicate that the robot encountered obstacles during navigation and had to abort the mission. |
| What is the task that the robot had to perform? | Based on the logs provided, it appears that the robot was tasked with performing a series of actions related to goal processing and cleaning up. The logs indicate that the robot received multiple goals, each time beginning to compute control effort. However, the robot was eventually canceled and stopped, as indicated by the log entry "Goal was canceled. Stopping the robot." The subsequent logs show the robot cleaning up, which suggests that the task involved some form of movement or operation that needed to be completed before the robot could safely stop. |
| Did the robot avoid any obstacle during the navigation? | Based on the logs provided, it appears that the robot did not avoid any obstacle during the navigation to waypoint with ID: 9. The logs indicate that the robot detected obstacles multiple times, with the distance to the obstacle increasing from 4.11 meters to 16.83 meters in some cases. Additionally, the logs show that the robot was actively passing new paths to the controller in response to the obstacles, but it did not successfully navigate around them. The final log entry indicates that the goal was canceled and the robot stopped, suggesting that it was unable to reach the waypoint due to the obstacles. |
For the examples shown in this section we use the following models:
- Embbeding model: bge-base-en-v1.5
- Reranker model: jina-reranker
- Base model: Qwen2.5-VL-7B
First of all, download the bags. Then, launch the explainable_ros bringup:
ros2 launch explainable_ros_bringup vlm_explainable_ros.launch.pyThen, play one of the bags, for example, bag 4:
ros2 bag play Rosbag-VLM-ExplainabilityFinally, wait till all logs are processed and stored and you can ask questions:
ros2 run explainable_ros_demos vlm_explainable_demo_nodeClick to expand
| User Question | LLM Answer |
|---|---|
| Did the robot encounter any obstacles during navigation? What type of obstacles? | Based on the logs provided, the robot encountered an obstacle during navigation. The type of obstacle detected is a blue leg or part of a leg, which is positioned very close to the wall. This obstacle is in the foreground and appears to be in the way of the robot's path. The robot's path was adjusted to avoid this obstacle, as indicated by the change in the distance to the goal from 2.92 meters to 15.27 meters. |
- llama_ros → A repository that provides a set of ROS 2 packages to integrate llama.cpp into ROS 2.
A series of rosbags (ROS 2 Humble) published in Zenodo are listed below. This data can be used to test the explainability capabilities of the project.
-
Sobrín Hidalgo, D. (2024). Navigation Benchmark Rosbags Inspired by ERL Competition Test (1.0.0) [Data set]. Robotics Group. https://doi.org/10.5281/zenodo.10518775
-
Sobrín Hidalgo, D. (2024). Navigation Test in Simulated Environment Rosbag. Human obstacle detection. (1.0.0) [Data set]. Robotics Group. https://doi.org/10.5281/zenodo.10896141
- Sobrín-Hidalgo, D., González-Santamarta, M. A., Guerrero-Higueras, Á. M., Rodríguez-Lera, F. J., & Matellán-Olivera, V. (2024). Explaining Autonomy: Enhancing Human-Robot Interaction through Explanation Generation with Large Language Models. arXiv preprint arXiv:2402.04206.
- Sobrín-Hidalgo, D., González-Santamarta, M. Á., Guerrero-Higueras, Á. M., Rodríguez-Lera, F. J., & Matellán-Olivera, V. (2024). Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction. arXiv preprint arXiv:2404.09705.
@misc{sobrínhidalgo2024enhancingrobotexplanationcapabilities,
title={Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction},
author={David Sobrín-Hidalgo and Miguel Ángel González-Santamarta and Ángel Manuel Guerrero-Higueras and Francisco Javier Rodríguez-Lera and Vicente Matellán-Olivera},
year={2024},
eprint={2404.09705},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2404.09705},
}
@article{sobrin2024explaining,
title={Explaining Autonomy: Enhancing Human-Robot Interaction through Explanation Generation with Large Language Models},
author={Sobr{\'\i}n-Hidalgo, David and Gonz{\'a}lez-Santamarta, Miguel A and Guerrero-Higueras, {\'A}ngel M and Rodr{\'\i}guez-Lera, Francisco J and Matell{\'a}n-Olivera, Vicente},
journal={arXiv preprint arXiv:2402.04206},
year={2024}
}
This project has been partially funded by the Recovery, Transformation, and Resilience Plan, financed by the European Union (Next Generation) thanks to the TESCAC project (Traceability and Explainability in Autonomous Cystems for improved Cybersecurity) granted by INCIBE to the University of León, and by grant PID2021-126592OB-C21 funded by MCIN/AEI/10.13039/501100011033 EDMAR (Explainable Decision Making in Autonomous Robots) project, PID2021-126592OB-C21 funded by MCIN/AEI/10.13039/501100011033 and by ERDF ”A way of making Europe”.
