XR AI v0.2.0 Beta

This beta introduces a revised agent architecture and two new end-to-end samples.

🚀 What's New & Enhancements

• Revised Agent Architecture: Adds typed AgentRuntime and VoiceAgent components with Relay-managed native tools.
  • Note: In-tree MCP servers and public Pipecat pipeline assembly APIs have been retired.
• New Sample - Tea-Making Guidance: Features evidence-gated workflow progression, background observation, document retrieval, and durable artifacts.
• New Sample - Lab Instrument Monitoring: Features marker-associated display readings, change alerts, durable output, and live event viewing.
• Decoupled Image Selection: Separates image selection from inference. This supports current and historical frames, recorded windows, multi-image requests, and timestamped video-frame queries.
• Simplified Model Setup: Agent samples now share the same prestarted speech, language, vision, and embedding services, with configurations provided for local models and NVIDIA NIMs.
• TTS Migration: Switched to Pocket-TTS and completely removed Piper-TTS dependencies.
• Core Improvements: Enhances participant-scoped voice output, interruption handling, service readiness, model service reuse, and cross-platform network telemetry.
• Documentation: Published versioned documentation featuring generated SDK, command-line, and configuration references.

⚠️ Breaking Changes

Important: This release contains breaking package, API, and configuration changes, including the DeviceIOHub rename. Please consult the release migration guide when updating from the previous beta.

🐛 Known Issues in Samples

• Web Client: Retains the previous agent response after disconnecting.
• iOS Client: May remain in a reconnecting state for about a minute after server shutdown before reporting a track-removal error.
• XR Render Context: Discards conversation context older than ten minutes, even during an uninterrupted session.
• XR Render Video History: Deployments using recorded video must explicitly enable video_history_enabled; omitting this flag completely disables historical queries.
  • Bug: With video history disabled, historical questions may incorrectly query the current frame instead of properly reporting that history is unavailable.
• XR Render Live-Perception Evaluator: May accept a response that combines an inability statement with a fabricated physical claim. In certain cases, it may query the scene for a user query instead of the camera.