hass-closest-intent

Fuzzy intent matcher for HomeAssistant.
Garbled STT output in, actual intent out.

Speech-To-Text (STT) output, especially fast and local STT output, is often simply bad. HomeAssistant's own Hassil is incredibly picky: your STT output must match exactly to one of the configured intents.

There's two paths forward from this: Upgrade your hardware to support better STT, or try to figure out what the speaker probably meant to say from the garbled output.

This project does the latter.

With this custom integration, "Lights on in live in room" will actually turn on the lights in your living room. So will, for that matter, "lighrts on inn livainriomm".

Short demo, first with closest-intent, then with bare Hassil:

• Pattern expansion. Expanding <expansion_rules>, (alternatives|to), and [optional|alternatives] all work, including on HASS-defined lists like your home's areas and entities!
• Slot extraction. Both for wildcard slots (like for adding something to the shopping list, where the {item} is a wildcard), and against slots like {timer_hours:hours} with a fixed set of possibilities.
• Fuzzy slot resolution. For list-like slots and expansion rules (including your areas and entities!), fuzzy match the slot values to the available options. Allows "livikroom" to be corrected to "living room".
• Actual intent handling still done by Hassil. closest-intent simply corrects your STT output or typos to the closest matching intent, and then forwards a nice, canonical sentence to Hassil, who then deals with the intent just like if you had spoken/typed perfectly.
• 100% LLM-free. Just uses relatively simple fuzzy matching of the input against your intents, plus some clever-ish (well... working, at least) tricks to improve the results.
• Fallback agent support. OK, I said 100% LLM-free, but if you absolutely want to, you can use one as fallback. More on this below.
• Is fast :) (as in: basically instant for a couple hundred configured custom intents).

Note: closest-intent is completely language-agnostic. All the examples in this README are in English, but you can use it with any language you like; personally, I use it in German.

Here's some examples of things I said, what my STT (wyoming-faster-whisper-base) understood, what HomeAssistant was able to do/answer after passing the STT output through closest-intent, and what the same STT output would have resulted in with just bare Hassil.

Note: These are actual results I got when speaking the "what was said" sentences in my phone. I'm a native German speaker, and so I do have an accent, but this pretty closely matches my experience when using the German-language version of whisper. The "bare Hassil" responses are what I got after 1:1 pasting the STT output into the voice assist chat window with closest-intent disabled.

...you get the idea.

closest-intent is registered in HomeAssistant as a conversation agent. On startup, it pulls the already-loaded vocabulary (intents, expansion rules, slot lists) straight from HomeAssistant's default conversation agent. This includes the dynamically built name/area/floor lists derived from your exposed entities, areas, and floors.

By default, only user-defined intents enter the candidate pool. Built-in intents are kept aside and can either be included wholesale (include_builtins) or by name (builtin_allowlist).

When a user request comes in (via voice command or the chat box), closest-intent fuzzy-matches that request against those expanded rules. If the rule does not contain a slot, it is picked immediately. If it does contain a slot, closest-intent performs a sequence of fancy magic steps to find the best-fitting slot value among a range of possible positions within the top-scoring matched sentences. In practice, this often means "smallest slot-value on a word-boundary", but the extraction is not limited to that.

With the best match found, we then reconstruct the "canonical form", i.e. a sentence that Hassil will actually understand. If in your configured intents, "Play some music." exists, and closest-intent got "Place on music" and matched that to the intent, it will simply forward "Play some music." to Hassil. If the intent contained a slot, the extracted value will be substituted.

This guarantees that the sentence passed to Hassil will actually be understood, and allows us to not have to worry at all about performing actions, running scripts,...

If no matching intent could be found, we pass the exact input we got to the configured fallback agent. By default, that is simply Hassil (which again allows us to be lazy and not worry about proper error responses), or another agent, like a LLM.

The "happy path" compared between with and without closest-intent thus looks something like this:

flowchart TB
    subgraph Right[" "]
        direction TB
        L1["<i>“lights on in living room”</i>"]
        L2["<i>“likeson in live in room”</i>"]
        L3["<i>n/a<i>"]
        L4["<i>match: false</i>"]
        L5["<i>“Sorry, I couldn't understand that”</i>"]
        L1 -.-> L2
        L2 -.-> L3
        L3 -.-> L4
        L4 -.-> L5
    end

    subgraph Mid[" "]
        direction TB
        User[You]
        STT[STT]
        CI[Closest Intent]
        Hassil[Hassil]
        HA[HomeAssistant]
        User --> STT
        STT --> CI
        CI --> Hassil
        Hassil --> HA
    end

    subgraph Left[" "]
        direction TB
        R1["<i>“lights on in living room”</i>"]
        R2["<i>“likeson in live in room”</i>"]
        R3["<i>“Lights on in living room.”</i>"]
        R4["<i>match: true</i>"]
        R5["<i>“Turned on the lights in living room.”</i>"]
        R1 ==> R2
        R2 ==> R3
        R3 ==> R4
        R4 ==> R5
    end

    classDef out fill:none,stroke:none,color:#555
    class L1,L2,L3,L4,L5,R1,R2,R3,R4,R5 out
    classDef pane fill:none,stroke:none
    class Left,Mid,Right pane

There's two major limitations to consider.

1. False-positives are a real possibility. This is actually an error class that Hassil usually almost completely eliminates: since the input needs to match so perfectly to the configured intents, there's almost no way to trigger an intent unintentionally. This issue gets worse the more similar-looking intents you have configured. If you are experiencing issues with this, raise the configured threshold (see below).
2. Not entirely unrelated: by default, the builtin intents are disabled for matching in closest-intent. HomeAssistant configures a good amount of intents by default, and does so with a shitton of expansion rules to cover more possible commands. Expanding all those intents in full is simply not viable (see also: "combinatorial explosion"), so the same expansion_cap applies to them too. In general though, these expansions are not super useful either, since they often cover things like (How is|How's), which just do not matter much with our fuzzy matcher. If you want only specific builtins (e.g. HassTurnOn), prefer builtin_allowlist over setting include_builtins to true.

In consequence, I highly recommend configuring your own intents for your personal usecases, if you haven't already. The best and most flexible way is through custom_sentences.

However, please feel free to try enabling the built-in intents first - it may just work for your usecase!

⚠️ custom-intents is not yet in the official HACS repos. I will add this section once it's in.

1. Open HACS, click the three-dot menu (top right) -> Custom repositories.
2. Paste https://github.com/charludo/hass-closest-intent, set Type to Integration, click Add.
3. Find Closest Intent in the HACS list, click Download.
4. Restart HomeAssistant.
5. Search for Closest Intent in the HACS search bar, then install.
6. Follow the config flow.

The integration can be set up entirely in the UI (Settings -> Devices & services -> Add integration -> Closest Intent) or via configuration.yaml. Both paths accept the same options. UI options override YAML on a per-key basis, clearing them in the UI falls back to YAML.

closest_intent:
  threshold: 70
  slot_threshold: 70
  expansion_cap: 16
  denylist: []
  include_builtins: false
  builtin_allowlist: []
  slot_extraction: true
  fallback_agent: conversation.home_assistant
  startup_self_check: true

Here's some recommended setup ideas for different usecases. Mix and match :)

Just use the default config. The threshold should be fine, the builtins are not required. Only thing to check is if you can get rid of some of your expansion rules that fall squarely within the realm of the fuzzy-matchable.

Consider enabling Prefer handling commands locally in the HomeAssistant voice pipeline settings. The speed boost is minimal for mortal amounts of intents, but faster is faster 🤷🏼‍♀

Consider enabling Prefer handling commands locally in the HomeAssistant voice pipeline settings, but setting the Conversation agent to closest intent. In the closest-intent setting, choose your LLM as the fallback_agent.

The pipeline then becomes: Hassil -> success or closest-intent -> success or LLM -> success or failure.

The result is that usually, your commands will be handled by Hassil/closest-intent, and only rarely is a fallback to a much, much slower LLM required. In other words, you benefit from this integration even if you are using an LLM for intent recognition!

closest-intent provides two diagnostics tools, parse_sentence and dump_candidates. Both can be called from Settings -> Developer tools -> Actions, then search for Closest Intent. They also work from automations, scripts, and hass.services.async_call.

Runs one sentence through the matcher and returns a structured response, including the matched intent, the candidate pattern that won, its score, the captured slots, the canonical sentence that (would have been) forwarded to Hassil. Optionally, you can actually forward it to Hassil to see what action this would trigger. (The action is not actually triggered though.) On a miss, also surfaces a sample of the name/area/floor slot lists so you can immediately see whether entity exposure is wired up. Set debug_top_candidates: true to additionally get the top 10 raw-scored candidates regardless of threshold to see why your intended candidate did not win.

Useful when a sentence unexpectedly doesn't match, slot capture is wrong, and so on.

Debug-logs and returns the full per-language state. This includes every expanded candidate, every expansion rule and its surface forms, every slot list and its values.

Options:

• include_builtins: true: also build and include built-in intent candidates in the dump, even when the integration is configured without them. Best combined with intent_filter to keep the output manageable.
• intent_filter: <substring>: case-insensitive substring filter on intent names (e.g. HassTurnOn).
• include_exposure: true: adds a per-entity breakdown of which states are exposed to Assist. Useful when an entity isn't showing up in slot_values['name'], noisy otherwise.

Useful for "fuck why is this STILL not working".

Please use these tools before filing an issue. Often this already helps solve the problem. However, if you believe that you did indeed find a bug (for example, a wrong slot capture, or no match where one should have been found), please do file an issue with the output from parse_sentence!

I will try to reproduce the issue, add a regression test, and then hopefully fix this. In my humble opinion the project is already in a very useful state, but things can always be better, and examples of things going wrong are super useful for this!

If the issue is "my intent isn't in the pool / lists look wrong", also attach closest_intent.dump_candidates for the affected language. This is enough for almost every reproducer without needing your full HomeAssistant config.