refactor: demos rewrite, rai_interfaces import guards #449
Conversation
This reverts commit 897338a.
fix: allow arbitrary types in toolkit definitions
maciejmajek
changed the title
|
@maciejmajek Could you please resolve confilicts? |
There was a problem hiding this comment.
@maciejmajek
This PR introduces major changes. I left some initial questions about the plan support rai features.
There was a problem hiding this comment.
Removing this file will break turtlebot_demo, so I think it should be refactored as well
| @@ -114,32 +81,31 @@ def main(allowlist: Optional[Path] = None): | |||
| (0.92, 1.01, 0.0) | |||
| """ | |||
|
|
|||
| node = RaiStateBasedLlmNode( | |||
There was a problem hiding this comment.
@maciejmajek
I can see that RaiStateBasedLlmNode has been replaced with a simpler agent.
Could you please clarify which of it's capabilities are planned to be added again and which will be dropped? Here is the overview of features:
robot state
- observing a ros2 topic and postprocessing/interpreting it's output. Adding the output of interpretation to the agent history.
- observing rosout withrai_state_logsnoderos2 action interface- this node had ros2 action interface, which could be used to send a mission to the robot. Then summary of langgraph agent has been posted as ros2 action feedbacks
There was a problem hiding this comment.
robot state
I think we can reintroduce it either by using a separate node in a graph, or by specifying topic name in connector sources.
ros2 action interface
The action interface can be reintroduced with support for actions in connectors. If you feel that this is important, we can add it to examples catalog showcasing ros2 usage using action server api from rclpy
There was a problem hiding this comment.
I think we can reintroduce it either by using a separate node in a graph, or by specifying topic name in connector sources.
I think it'll require both, because state should be received (connector) and then placed in the LLM context (node in the graph). In general, it sounds like a good idea, if you don't plan to refactor it in this PR, how about adding an Issue that has to be resolved before the next release to add it again?
There was a problem hiding this comment.
Could you please clarify which features of rai_whoami are planned to be still supported in rai?
- RAG for documentation
- "embodiment" in system prompt using documentation and images
- setting robot constitution
There was a problem hiding this comment.
I don't think that the system prompt propagation should be done via ros2 or any other communication protocol at the moment. Right now I find it easier to use text files, but in the future we could build a specific database with system prompt versioning.
There was a problem hiding this comment.
- I think the design can be changed to the easier one (for example text files instead of ros2 node).
However my question is more about removing/leaving functionalities: - automatic creation of the system prompt using documents and images (can be used to create a system prompt in a file)
- RAG for robot documentation (I think it might be too much for the
rai_core, so I thinkrai_whoamipackage might be a good place for it. ros interface is up to discussion, probably RAG can be queried directly in the tool call.)
Purpose
Using RAI as a standalone framework is unfeasible due to many custom ROS 2 dependencies. Some of the parts of RAI could already be used in a pip install rai installation.
Proposed Changes
This PR adds various changes including:
Issues
Testing