[Exploration-POC] Voice can potentially be way easier as a a HA app

[laupalombi]

Problem statement

Voice has a couple of major issues in evolving moving forward, with the perceived complexity on jumping around for configuration (not enough easy paths).

Adding to this, most times other issues are reported are bad configurations of the implementation.

There are other pains to solve, but there is a real opportunity in making Voice perceived at market level in ease to configure, trust of the actions and answers and understanding of timing issues if we get it on a different structure.

This is huge, and I don't feel we should make this a decision based upon theoretical reasoning - we need a solid POC. @synesthesiam is already working on this, but I want to formalize this as a bet.

Here is a map of the information architecture changes proposed and the user values it could bring https://www.figma.com/board/0WFCJW3P77MWQJoOJyIIQD/Voice-Pipeline?node-id=0-1&p=f&t=UoEYS9TiTptQWkdt-0

Scope & Boundaries

In scope

• POC of Voice as an app that is functional enough to make an educated decision about moving forward or not, based on the user value added, the simplicity in setup and the not deterioration of current system. (aka it has to be at least better to grant a change).
• Valid way to use this for non-app users (docker)
• Testeable/installable by everyone (open), but not in an easy way or safe from breaking (labs maybe)

-- STRETCH GOAL - we see the experimental POC is in good enough shape to ship it under certain conditions, more publicly.

Out of scope

• Finessed version
• Edge cases or nice tweaks - the whole thing could be discarded, we need a finalized POC

Foreseen solution

• A POC that makes Voice run testable/installable by everyone (open), but not in an easy way or safe from breaking (labs maybe)

-- STRETCH GOAL - we see the experimental POC is in good enough shape to ship it under certain conditions, more publicly.

Community signals

We did a survey at the begining of the year (link)where you can see how different pain points are referring to complexity as the issue in some way or another.

Although this is heavily biased - we don't have a way to research Voice new users that are not technical, so the expectation is this representation of complexity would be much bigger.

Risks & open questions

• How much can we sacrifice in easy paths vs heavy configurations?
• Is it really going to be easier?
• Can we make this and not loose the "tinkering" angle? (custom wake words, LLMs)
• what is the loss and transition expected from migrating to one system to the other?

Appetite

Medium - should be done before the .6 release, if just the POC. I would advocate that unless we have other goals, we can use the remaining time to see this live tested and collect results, since we need that to make the decission.

Execution issues

No response

Decision log

Date	Decision	Outcome

[anakinsbrna]

Naive question: With "as an app" we mean within Home Assistant what used to be "add-on", right?
I read the title and had smartphone apps in mind, probably also because of the wake-word detection
"Voice can be way easier as an app" I assumed first that you mean "easier to use" - but you mean more the pipeline can be simplified, am I right?

What's the advantage of simplifying the voice pipeline for the user?

In the product goals on the Figma you wrote:

"Voice is more reliable towards my expectations as a non-contextual user, while being equally easy to set up."
I'm not fully sure I understand what's meant here - do I understand this correctly, that because the voice pipeline will be simplified and less things happen in the background, it's closer to how a person who doesn't know the details imagines how Voice works (aka it matches better the mental model of a lay person). Is that correct?

and I don't understand these two:

"Voice is more representative of market baselines in user perception."
"Voice uses HA Cloud as a shortcut for an even easier streamline path, without damaging the non-HA Cloud path as per current expectations."

What does this mean for users?

[anakinsbrna]

And you wrote above: there is a real opportunity in making Voice perceived as a lot more solid if we build it differently.

I'm not in the topic - how do we know that Voice is currently perceived as not solid? And in what way will the proposed changes affect how people perceive how solid Voice is?

[anakinsbrna]

The goal right now is to build the POC and decide based on that whether to implement this or not. What do you mean by the "POC is in good enough shape"? What are the criteria to judge whether its shape is good?

[laupalombi]

Yes, a Home Assistant App (former Add on)

because the voice pipeline will be simplified and less things happen in the background, it's closer to how a person who doesn't know the details imagines how Voice works (aka it matches better the mental model of a lay person). Is that correct?

Correct.

"Voice is more representative of market baselines in user perception."

With this POC we may be able to detect silences better, have a no-pause wake word like Alexa, and give people governance of their home intents, so they can tweak them. All this will mean less errors and frustration with voice responses not being accurate or taking a while and not knowing why, which is something the other players resolve better.
The perception in terms of speed or effectiveness should (we hope) be better overall and put us quite close to them, in terms of house management.

"Voice uses HA Cloud as a shortcut for an even easier streamline path, without damaging the non-HA Cloud path as per current expectations."

We will create a path for HA Cloud so its easier, but I wanted to make sure in the POC that it should also work and be presented as an equal alternative for non-HA Cloud or local users. Meaning, the POC must at least be equally good for them as it is now, ideally improve. It shouldn't downsize the local experience at all.

I'm not in the topic - how do we know that Voice is currently perceived as not solid? And in what way will the proposed changes affect how people perceive how solid Voice is?

Explained above. Solid is not an extra value, sorry if it was a poor choice of words, I mean added value. I am going to edit "solid" out so its not confusing.

[laupalombi]

The goal right now is to build the POC and decide based on that whether to implement this or not. What do you mean by the "POC is in good enough shape"? What are the criteria to judge whether its shape is good?

It improves the current implementation by adding all the potential user values noted in the figjam, and doesn't deteriorate current experience. The real comparison will only be available once the POC is finished, this is exploratory.

[jlpouffier]

A voice pipeline is today composed of

• A STT engine
• A TTS engine
• A Conversation agent
• (A device running a wake word engine)

How would porting our default conversation engine to an app solve "the perceived complexity on jumping around for configuration (not enough easy paths)" ?

Is this about merging all components of our voice pipeline into an app?
If yes, how do you deal with personalisation? Being able to use that voice or this voice?
If no, what are we solving with this POC?

[laupalombi]

I had a revelation overnight. cc @synesthesiam . I don't believe this is a bet (yet). This is something we are doing in shaping, full experimental, no idea yet of the benefit besides theory. So the issue is this is not something we can vote on (yet) because the POC is the tool that we should be using to test all these questions. There is a chance it doesn't solve anything, and its fine. This is fundamentally research.

I'm withdrawing it from the betting table 👍 Will land there once we finish the POC explorations and see if the final results are actually something we can bet on making it real.

Apologize for anyone spending time in advance on this 🙏