LLM Language Engine

LLM Language Engine

Since 0.7.6, Ava can understand and act on a spoken sentence inside this app. No Home Assistant conversation integration, no YAML, no root. Wake, capture, STT, understanding, execution, and speech stay on one device.

This is not the same as Claw Assistant. Claw is a Home Assistant conversation hub. The language engine is Ava’s own “text to understanding” step. You can use either, or HA’s stock pipeline, or a conversation-engine mod that claims the wake instead of HA.

Path: Ava Settings → Voice configuration → Text to understanding → LLM language engine

---

Pick a brain

A small local model that runs with the network unplugged is enough. A cheap cloud model works. An existing OpenAI-compatible endpoint plugs in.

Easy path: a free key from build.nvidia.com, endpoint https://integrate.api.nvidia.com/v1, OpenAI-compatible.

Five endpoint slots. With the fallback chain on, two transport failures on the current slot move the same unfinished task to the next endpoint. Completed actions never run twice.

---

What a sentence can move

Piece	Covers	Examples
This Ava	Local screen, mic, overlays, volume	“dim the screen”, “screen off”, “lock this one”, “mute the mic”, “stop talking”, “set ten minutes”, “how long is left”, “open Dream Clock”, “open Quick Entities”
House devices	HA index	“kitchen lights on”, “dim the pendant to thirty”, “blinds halfway”, “run goodnight”, “is the garage closed”
Pages and settings	HA frontend and Ava settings	“open automations”, “switch to the dark theme”, “set TTS volume to sixty”, “go to touch pad settings”. “Settings” opens Ava’s settings.
The web	Isolated browser task	“will it rain in London tomorrow”, “open this URL”, “scroll down”, “tap sign in”. Website words cannot move house devices.
Music and nearby Avas	Players and intercom	“play some jazz”, “next”, “tell the living room dinner’s ready”, “which Avas are around”
Apps and the pad	Local apps, touch-pad slots	“open Calculator”, “open Spotify in a floating window”. Recorded pad automations replay as one move.

Rules in one breath:

• A sentence splits into ordered jobs. A failed first step does not fire the second blind.
• Same-named devices come back as candidates, not a guess.
• Locks and alarm panels act only on a named house device. An unnamed clock time stays on this Ava.
• Codes pass verbatim, never invented.
• A camera frame can go to the model’s eyes.
• Every page visit returns to where it started.
• A tap that did not land is never claimed as done.

Moves that need no pieces still play: “what’s the date”, “tell me a joke”. A recognised voiceprint gets its name. The reply uses the language that was spoken.

---

Every step has a receipt

The model saying “it’s off” is not enough. Each call has a status the model may speak to: observed, sent, queued, applied, unknown. “Sent” never becomes “done.”

Writes are remembered. The same command again returns the earlier result instead of re-running. A call that fails twice is blocked on the third try so the model says what is missing.

Unknown HA services are read one section at a time, not the whole manual at once.

A long job can keep going after a pause — no need to say “continue” again.

The draft in the model’s head stays inside the turn. What gets spoken is only the sentence meant to be heard (0.7.7).

---

Clock times, countdowns, alarms (0.7.7)

A length and a clock face are different:

You say	What it is	Where it lands
“Time ten seconds tomorrow”, “set ten minutes”	Countdown	The Dream Clock flip face. “How many minutes are left” reads that countdown.
“Wake me at seven tomorrow”, “remind me at two to take my medicine”, “an alarm for the day after tomorrow”	Clock time	This Ava. Any calendar day.

Say both in one sentence and both get done. No extra question about which is which.

“Set an alarm” sets an alarm. A house alarm panel still needs that device named.

When the time comes, an alarm sits on the screen. Tap it to change the time, pick a ringtone, or delete that one. While it rings the icon turns red and breathes; one tap stops it, and stopping clears it. Say stop while something is ringing and the one that is ringing stops — countdown, alarm, and reminder on the same path.

Five at most. Only the ones that are set show up in Home Assistant, named task event (alarm) and task event (reminder), with -1 from the second onward. Date and time go both ways. Empty slots take no place. They survive reboot.

---

Ask the house, and ask this device (0.7.7)

• “How many lights are on in the whole house” counts that type from the index, speaks the ones it can read, and skips a row it can’t. The sentence still finishes.
• “Is the live dashboard on” reads the switches on this page right now, and can flip them.
• The Mod Store needs no screenshot: catalog, installed, on or off, as text. Say install, turn off, or uninstall, and it does that by name.

---

Voice configuration around the engine

0.7.6 rebuilt Voice Config in the order of a turn: channel → wake → microphone → voiceprint → STT → this engine → TTS → ambient events.

STT and TTS engines can still come from the Mod Store (HA STT, HA Edge TTS).

Finished speaking detection (0.7.7) is the same item as on this satellite in Home Assistant:

Step	Silence
Aggressive	0.25 s
Default	0.7 s
Relaxed	1.25 s

A change on either side is the same change. Hidden until Home Assistant is connected.

Path: Ava Settings → Voice Config → Speech-to-text → STT listening engine → Finished speaking detection

Hold-to-talk on the wake button runs up to two minutes. Ava renews the ESPHome window at about 13 seconds so Home Assistant receives one complete sentence.

---

FAQ

Is this required?

No. Leave the language engine off and Home Assistant’s Assist pipeline runs as before.

Can a ZIP mod replace this and HA?

Yes, with "conversation_engine": true. See Conversation Engine API.

Thoughts leak into speech

0.7.7 keeps the model draft inside the turn. Update if an older build still reads the scratchpad aloud.

---

Back to Voice Control