Skip to content

0.4.3

Choose a tag to compare

View all tags
@aleph-ra aleph-ra released this 07 Nov 09:46
· 375 commits to main since this release
0.4.3
4e5f96c

🚀 EmbodiedAgents v0.4.3

EmbodiedAgents v0.4.3 introduces a modernized, web-based fully dynamic UI experience, enhanced visual and multimodal interaction capabilities, and numerous stability improvements across LLM/VLM components and the vision pipeline.

✨ New Features

Dynamically generated UI for Agents

  • Extended Sugarcoat's dynamic web UI elements for Agents so each recipe can create its own interaction client.
  • Added dynamic UI callbacks for:
    • RGBD images
    • Detections and DetectionsMultiSource messages
    • Points of interest messages
  • Added UI element definitions for custom message types (extensible by sugar-derived packages).
  • Added logging output from StreamingString directly in the web UI.
  • Added utility for drawing images with bounding boxes for visualization of detections.

Components & Message Handling

  • Added Detection2D as an allowed input in the map encoding component.
  • Added callbacks for Detections2D and integrated their handling in LLM/VLM components.
  • Streamlined naming conventions for detection and tracking message types.
  • Added handling of additional custom message types from sugar-derived packages in agent components.
  • Added support for publishing single detection or tracking messages from the vision component.
  • Added UDP streaming to IP:PORT as an option in the TextToSpeech component when play_on_device is enabled.

LLM / MLLM Enhancements

  • Added alias VLM for the MLLM component.
  • Added warning messages when streaming is enabled without using the proper StreamingString message type.
  • Added StreamingString message type for managing text/audio streams in external clients.

🧰 Fixes & Improvements

  • Fixed image publishing in detection messages and empty image inputs in Detection2D publications.
  • Fixed raw message data access during execution to avoid redundant output calls.
  • Fixed topic passing from sugar-derived packages to agent components.
  • Fixed websocket receiving issues in Text-to-Speech.
  • Fixed keyword argument mismatches in detection and tracking message publishing.
  • Improved refactoring of callbacks for video and RGBD message types.
  • Added detections as handled output in the VLM component.
  • Removed StreamingString as an input option for LLM/VLM components to streamline behavior.

⚙️ Chores & Maintenance

  • Added websockets as an explicit dependency.
  • Added warnings for template usage with StreamingString.
  • Removed tiny web client and Chainlit-based client. Replaced with dynamic UIs for every recipe.

📚 Documentation Updates

  • Added instructions for using the new dynamic web UI.
  • Updated examples and guides for the new web-based client.
  • Updated planning model documentation.
  • Added a recipe for vision-guided point navigation.

🧩 Summary

This release completes the transition to a dynamic, ROS-integrated web interface for EmbodiedAgents, offering a more flexible and interactive experience for managing Physical AI agents.
It also enhances support for visual grounding, audio streams, and extended message types, and various bug fixes and improvements.

Full Changelog: 0.4.1...0.4.3

📖 Dive Deeper

Ready to explore these new features? Check out our updated documentation for detailed information, examples, and installation guide.

Assets 8
Loading