Refactoring of the Home Agent integration (UPSTREAM - https://github.com/aradlein/hass-agent-llm) to enhance it for better context management, guardrails, extra tools, A2A/MCP server support, and more for use in the Pepa AI for Aging project (https://ai4aging.org). All documents will be updated as the integration evolves.
A highly customizable Home Assistant custom component that provides intelligent conversational AI capabilities with advanced tool calling, context injection, and conversation history management.
- Vector Reindex Optimization - Skip reindexing entities when state and attributes are unchanged, reducing unnecessary re-embeddings
- Datetime Serialization Fix - Properly handle datetime objects in tool results and vector DB context provider
- Area Lookup Fix - Use entity/device registry for area resolution instead of state.attributes for more reliable room detection
- Embedding Cache Memory Leak Fix - Fixed a memory leak where the embedding cache accumulated stale entries for frequently-changing entities
- CI Fixes - Fixed version validation workflow, flake8, and black formatting issues
Home Agent extends Home Assistant's native conversation platform to enable natural language control and monitoring of your smart home. It works with any OpenAI-compatible LLM provider, giving you flexibility to use cloud services or run models locally.
Key Capabilities:
- Natural language home control through any OpenAI-compatible LLM
- Automatic context injection - LLM knows your home's current state
- Persistent conversation memory across interactions
- Extensible tool system for custom integrations
- Streaming responses for voice assistants
- Long-term memory system for personalized experiences
- LLM Integration - Works with OpenAI, Azure OpenAI, Ollama, LocalAI, LM Studio, or any OpenAI-compatible endpoint
- Entity Context - Automatically provides relevant entity states to the LLM
- Conversation History - Maintains context across multiple interactions with persistent storage
- Native Tools - Built-in
ha_controlandha_querytools for home automation - Custom Tools - Define REST API and Home Assistant service tools in configuration
- Event System - Rich events for automation triggers and monitoring
- Streaming Responses - Low-latency streaming for voice assistant integration (~10x faster)
- Vector Database Integration - Semantic entity search using ChromaDB for efficient context management
- Multi-LLM Support - Use a fast local model for control + powerful cloud model for analysis
- Memory System - Automatic extraction and recall of facts, preferences, and context
- Context Optimization - Intelligent compression to stay within token limits
- Tool Progress Indicators - Real-time feedback during tool execution
- Home Assistant - Version 2026.3.1 or later (for Python 3.14 support)
- Python Dependencies -
aiohttp >= 3.9.0(included with Home Assistant)
- ChromaDB - For vector database context mode
chromadb-client == 1.5.3- Required for: Vector DB context injection and memory system
- OpenAI - For embeddings in vector DB mode
openai >= 1.3.8- Required for: Vector DB entity indexing (alternative: use Ollama)
- Wyoming Protocol TTS - For streaming responses
- Required for: Low-latency voice assistant integration
- In HACS, go to Integrations → ⋮ → Custom repositories
- Add repository:
https://github.com/aradlein/hass-agent-llm - Category: Integration
- Click Add
- Search for "Home Agent" in HACS
- Click Install
- Restart Home Assistant
- Go to Settings > Devices & Services > Add Integration
- Search for "Home Agent" and follow the setup wizard
- Download the latest release from GitHub
- Copy the
custom_components/home_agentdirectory to your Home Assistantconfig/custom_componentsfolder - Restart Home Assistant
- Go to Settings > Devices & Services > Add Integration
- Search for "Home Agent" and complete the configuration
For detailed installation instructions, see Installation Guide
Navigate to Settings > Devices & Services > Add Integration, search for "Home Agent", and configure:
- Name: Friendly name (e.g., "Home Agent")
- LLM Base URL: Your OpenAI-compatible endpoint
- OpenAI:
https://api.openai.com/v1 - Azure OpenAI:
https://<resource>.openai.azure.com/openai/deployments/<deployment> - Ollama (local):
http://localhost:11434/v1 - LocalAI: Your LocalAI URL
- OpenAI:
- API Key: Your API key (if required)
- Model: Model name (e.g.,
gpt-4o-mini,llama3.2, etc.) - Temperature: 0.7 (recommended for most use cases)
- Max Tokens: 500 (adjust based on your needs)
Call the conversation service:
service: home_agent.process
data:
text: "Turn on the living room lights"Access Settings > Devices & Services > Home Agent > Configure to:
- Configure context injection mode (direct or vector DB)
- Enable conversation history
- Set up custom tools
- Configure external LLM for complex queries
- Enable memory system
- Enable streaming for voice assistants
For detailed configuration options, see Configuration Reference
- Installation - Get up and running in minutes
- Configuration - Essential settings explained
- Example Configurations - Ready-to-use configs for common providers
- FAQ - Top 20 questions answered
- Memory System - Enable long-term memory
- Vector DB Setup - Semantic entity search
- Custom Tools - Extend with REST APIs and services
- External LLM - Multi-LLM workflows
- Architecture Overview - Component diagrams and system design
- API Reference - Services, events, and tools
- Troubleshooting - Quick fixes for common issues
- Examples - 10 ready-to-use examples
- Migration Guide - Moving from extended_openai_conversation
# Use with Home Assistant voice assistant
# Just speak naturally to your voice assistant
"Turn on the kitchen lights to 50%"
"What's the temperature in the living room?"
"Is the front door locked?"automation:
- alias: "Morning Briefing"
trigger:
- platform: time
at: "07:00:00"
action:
- service: home_agent.process
data:
text: "Good morning! Please prepare for the day."
conversation_id: "morning_routine"# configuration.yaml
home_agent:
tools_custom:
- name: check_weather
description: "Get weather forecast"
handler:
type: rest
url: "https://api.open-meteo.com/v1/forecast"
method: GET
query_params:
latitude: "47.6062"
longitude: "-122.3321"
current: "temperature_2m,precipitation"For more examples, see Examples Documentation
Home Agent automatically maintains conversation context across multiple voice interactions, enabling natural multi-turn conversations with your voice assistant.
- Each user/device combination maintains a persistent conversation session
- Sessions automatically expire after 1 hour of inactivity (configurable)
- Conversation history is preserved within each session
- Different devices maintain independent conversation contexts
Before (without persistence):
User: "What's the temperature in the living room?"
Agent: "The living room is 72°F"
[Later]
User: "What about the bedroom?"
Agent: "I don't have context about what you're asking about"
After (with persistence):
User: "What's the temperature in the living room?"
Agent: "The living room is 72°F"
[Later]
User: "What about the bedroom?"
Agent: "The bedroom temperature is 68°F"
Configure the session timeout in integration settings:
- Minimum: 60 seconds (1 minute)
- Maximum: 86400 seconds (24 hours)
- Default: 3600 seconds (1 hour)
Reset your conversation context using the clear_conversation service:
# Clear conversation for current user/device
service: home_agent.clear_conversation
# Clear conversation for specific device
service: home_agent.clear_conversation
data:
device_id: "kitchen_satellite"
# Clear all conversations
service: home_agent.clear_conversationEach device maintains its own conversation context:
- Kitchen satellite remembers kitchen-related conversations
- Bedroom satellite has independent context
- Same user on different devices = different conversations
Contributions are welcome! Please:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Write tests for new functionality
- Ensure all tests pass (
pytest tests/) - Follow the coding standards in Development Guide
- Submit a pull request
Home Agent maintains >80% code coverage with comprehensive unit and integration tests:
# Set up environment
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements_dev.txt
# Run tests
pytest tests/unit/ -v
# Run with coverage
pytest tests/ --cov=custom_components.home_agent --cov-report=htmlTest Status: 400+ passing tests across core functionality, vector DB, memory system, custom tools, and streaming.
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Documentation: See docs/ directory
MIT License
Built with inspiration from the extended_openai_conversation integration. Special thanks to the Home Assistant community.
- Fix: Skip vector reindex when entity state and attributes unchanged — reduces unnecessary re-embeddings
- Fix: Handle datetime serialization in tool results and vector DB context
- Fix: Use entity/device registry for area lookup instead of state.attributes
- Fix: Evict stale embedding cache entries on entity state change — prevents memory leak from frequently-changing entities (#111)
- Fix: Version validation CI workflow grep pattern anchored to avoid false mismatches
- Fix: Black formatting and flake8 compliance
Compatibility Notice: This release targets Home Assistant 2026.3.1 and Python 3.14. It may have backwards compatibility issues with older HA versions or Python 3.12/3.13. If you experience problems, please try a previous release.
- Fix: Upgraded
chromadb-clientto1.5.3to resolve Python 3.14 incompatibility (removal ofimghdrstandard library module) - Fix: Updated dev requirements and test mocks to align with chromadb-client 1.5.3 API
- Changed: Minimum supported Home Assistant version updated to 2026.3.1
- Fix: Move initial entity indexing to background task to prevent setup timeout on large installations
- Fix: Resolve multiple memory leaks causing ~6GB growth over time — LRU eviction for embedding caches, debounced batch reindexing, HTTP session reuse, conversation history enforcement, and proper resource cleanup on shutdown
- Feature: Azure OpenAI support - native integration with Azure OpenAI deployments including API versioning and endpoint handling (#9)
- Feature: Universal language support - works with any Home Assistant language setting via MATCH_ALL (#15)
- Feature: Jinja template support for API key fields - use Home Assistant templates for dynamic secrets (#14)
- Fix: Improved compatibility with proxy gateways like Cloudflare AI Gateway (#17)
- Feature: Include entity/device labels in system prompt for better LLM context (contributed by @zopanix)
- Testing: Comprehensive test coverage for labels feature
- Feature: Custom service tools now support
return_response: truefor services that return data (likecalendar.get_events)
- Feature: TTFT (Time To First Token) and voice pipeline metrics for performance monitoring
- Feature: Configurable max context tokens option (fixes #65)
- Fix: Memory retrieval no longer creates transient memories or duplicates
- Enhancement: Comprehensive testing improvements and GitHub issue fixes
- Feature: OpenAI-compatible embedding endpoints - Vector DB now respects configured
embedding_base_urlfor OpenAI provider (fixes #6) - Enhancement: Switched to async OpenAI client using Home Assistant's native HTTP client
- Enhancement: Improved retry logic with proper exponential backoff for embedding requests
- Feature: Reasoning model support - Filter
<think>...</think>blocks from LLM output, enabling support for reasoning models like Qwen3, DeepSeek R1, and o1/o3 - Fix: Only send
keep_aliveparameter to Ollama backends; prevents 400 errors with OpenAI and other cloud APIs
- Enhancement: Updated minimum Home Assistant version to 2025.11.0 for improved compatibility
- Fix: CI test environment now uses pinned Home Assistant dependencies
- Docs: Successfully submitted to HACS for official community store listing
- Feature: Feature-based service filtering with intelligent parameter hints
- Enhancement: Brightness parameter standardization (brightness_pct 0-100 instead of brightness 0-255)
- Fix: Move play_media to base services for all media_player entities
- Enhancement: Comprehensive entity services reference documentation
- Enhancement: Improved system prompts with standardized parameter rules