AI Task for generating structured Data

[allenporter]

2025-04-01: Initial proposal using selectors as the schema
2025-05-17: Updated with JSON schema (common subset of Open API) as primary solution and simplified response format
2025-06-15: Updated to use AI Task API, reducing scope to not expose an action
2025-07-01: Updated again to go back to using selectors to use selectors

Background

Home Assistant LLMs can call tools by returning structured responses conforming to a specific set of commands. Additionally, this can be extended to support structured conversation outputs.

Proposal

Extend AI Tasks #1237 to accept an input schema and APIs to allow an AI task entity to generate a response with an object conforming to that structure.

Use cases

Users today can only prompt the model to output simple strings or number values, but instead they could:

• [Text] Produce a text summary of each area of my home and let me know if there are any anomalies (example)
• [Text] Home Assistant can produce an automation or blueprint to perform and provide the lower level schema to get back a valid object (datasets, example prompt) (structured outputs can improve quality)
• [MCP] Use MCP to scrape this website and return a structured response with the top 3 headlines for rendering on an E-Ink display (example)
• [Multimodal] Use Gemini to analyze my chicken coop and return the number of geese, chicken, or ducks.(example). These can also be separate entities using a template entity.
• [Multimodal] Use Gemini to process my journal hand written journal entry png file and extract todo items or calendar entries (example)

Solution Overview

The big picture shape of the solution is:

• The aI_task.async_generate_data will be a new type of AI task for generated structured outputs.
• The AI Task Entity is responsible for converting the schema to the appropriate format for the API (which is typically an JSON schema (see survey below) so no conversion needed for most) and for parsing the structured output response.
• The structured output is independent of the input type. e.g. you may want structured output for a text prompt, or image, etc

Solution Details

API

async def async_generate_data(
    hass: HomeAssistant,
    *,
    task_name: str,
    entity_id: str | None = None,  # Optional, will use preferred entity if set
    prompt: str,
    structure: dict[str, Any],
) -> GenDataTaskResult:

Both will end up calling the entity method

    async def _async_generate_data(
        self,
        task: GenDataTask,
        chat_log: ChatLog,
    ) -> GenTextTaskResult:

Input schema format

• Proposed format is selectors, using prior art from blueprint inputs or script fields and use selectors. All automations, blueprints, scripts, and action calls define their input schemas using selectors, and selectors are familiar to users. These have the same required fields as JSON OpenAPI schema and we already have code to convert selectors to JSON OpenAPI format.

• Alternative:

prompt: |-
    Please generate some mock user profiles with a name and age in structured response.
    Examples:
    Name: John Smith, Age: 30, Aliases: Johnny
    Name: Alice Parker, Age: 54
structure:
    profile:
    object:
      multiple: true
      fields:
      name:
        description: Full name of the user
        selector:
          text:
      age:
        description: Age of the user
        selector:
          number:
      aliases:
        description: Additional nicknames for the user
        selector:
          text:
            multiple: true

conversation_id: 8A8E3C8F288A41DBBB12073F6C
data:
  profile:
    - name: Emily Chen
       age: 42
    - name:  David Rodriguez
       age: 30
    - name:  Benjamin Lee
       age: 38
       aliases:
       - Ben
       - Benny

Selectors now support nested objects fields so we are not limited in what we can describe. However, if we are limited in the future we can extend selectors to support that type.

Selectors are primarily used for rendering select UI in the frontend, however, that functionality is not used here. They are only used for defining a voluptuous schema internally. We already have a decided on our approach for how integrations convert between vol schemas and openapi schemas for tool formats, and this will not be revisited for structured outputs and can all be reused. (See helpers/llm.py method _selector_serializer). e.g. (Selectors that describe extra things (e.g. a QR code) that could never be represented already have a way they are handled for tool calls).

We could also decide to explicitly limit to the basic types only.

The models of LLMTask and LLMTaskResult will be extended with structure input and output fields:

@dataclass(slots=True)
class GenDataTask:
    """Generate structured data task to be processed."""

    type: GenTextTaskType
    """Type of the task."""

    prompt: str
    """Prompt for the AI."""

    structure: dict[str, Any]
    """Response output structure in JSON schema format."""

@dataclass(slots=True)
class GenDataTaskResult:
    """Result of Gen Data task."""

    conversation_id: str
    """Unique identifier for the conversation."""

    data: dict[str, Any]
    """Generated data following the input structure."""

Future work

Future work that will make AI Gen Data Tasks much more useful:

• Expose actions for AI tasks. When used with action response values, we can power even more use cases e.g. template entities, automations, etc. The AI Tasksproposal does not yet propose adding an action which we can handle separately.
• AI Tasks should reconcile what functions are available e.g. MCP functions
• Accept image input for AI tasks

Alternatives

• Alternative: Use prior art, JSON Schema (subset of OpenAPI). These are popular in the LLM world, but not in Home Assistant. However, they are easy to understand and well documented and familiar for LLM users. Here is an example in practice of the task and task result. I've written them in yaml so you get the idea:

prompt: |-
    Please generate a mock user profile with a name and age in structured response.
    Examples:
    Name: John Smith, Age: 30, Aliases: Johnny
    Name: Alice Parker, Age: 54
structure:
    type: object
    properties:
      name:
        type: string
        description: Full name of the user
      age:
        type: number
        description: Age of the user
      aliases:
        type: array
        description: Additional nicknames for the user
        items:
          type: string

conversation_id: 8A8E3C8F288A41DBBB12073F6C
data:
  name: Emily Chen
  age: 42

• Alternative: Input format could be a new simple output type language that has equivalent support to OpenAPI schema, but is stripped down and simplified to easy use in yaml

Integrations supported

This feature can be supported by all the major LLM conversation agents

• Gemini
• Ollama
• OpenAI
• Anthropic
  see implementation notes

[balloob]

So thinking a bit more on this, and also discussing with Allen offline about upcoming LLM tasks*, I wonder if structured outputs should be a feature solely added to LLM tasks and not be part of conversation.process. Instead, conversation.process would focus solely on human conversations via voice/text, while a new conversation.run_task could be made to require structured output support.

(The idea of LLM tasks (architecture proposal TBD) provide a way to use conversation agents for general purpose tasks outside of an assistant pipeline (e.g. use for summarization in the frontend). This uses a conversation agent LLM for general use, without user prompts or customizations.)

[allenporter]

Updated to use the latest AI Tasks proposal.

[allenporter]

Changed back use selectors again based on offline discussion and new support for object fields.

[emontnemery]

The proposal was approved in the core meeting July 3rd