Strategy for config methods

[bachya]

With the 0.87.0 release, the top comment on the Reddit post says:

These kind of statements make me sad:

The SmartThings component is configured exclusively through the front-end. Manual setup through configuration.yaml is not available at this time.

Subsequent replies to that comment:

The configuration is getting more and more fragmented. Some via config, some via ui. Then you get to the front-end where you have to make a hard choice. Config or front-end, not both, not even a little.

I like being able to manually set/see everything. This is why I use yaml mode for Lovelace still. I'm sure people who use Unix on a regular basis are just more comfortable with configuration files rather than a black box UI

I'm having a small pity party about not being able to do everything in my IDE anymore, but I'll get over it.

To add onto what -Smokin- was saying, easier troubleshooting and deployment. What happens when HA is offline and I want to make a change? What if something caused by the SmartThings integration is causing an issue? Might not be the case now but I think that the ability to manage everything via terminal/config is the best approach. And, this has no impact who prefer the UI interface.

Removing manual control via configurations files from poweruser to make it easier for 'non-tech' people is never a good idea. Remember that most of us use HA because we don't support cloud control over our things, we like to control our things. Saying that, I really like to have my configuration on configuration files so I can change them to exactly I want or backup the files, share it, etc.

I don't mean to harp on the UI-only configs yet again, but the .storage directory has seriously complicated my configuration and usability within HA.

I know there's some great features coming out of them but recently I've had to restart several times and manually edit registries because I wanted to rename something and there were conflicts. I appreciate that these help UI users but it's made things harder for config-based, advanced users.

Edit: also, it's not clear to me as a user whether that hidden directory should be backed up or committed to source control.

Having configuration as a file means you can use a bunch of tools - rsync, git versioning, diff, grep, sed, bulk rename, etc - that you simply cannot if it is stored in a database that's only meant to be accessed through the UI. Those command line tools can make long-term maintenance easier: you can check your git log for things you've added, do find-and-replace operations on your configuration, archive old setups as you retire devices or take them off line, etc. And if you run into issues and need to troubleshoot or just decide to get rid of configuration clutter, you can just create a backup/git version, blank your config files, and then use diff/merge techniques to pick and choose what you want; with the database approach, you have to start from scratch without carrying anything over.

A few examples I can think of: HASS recently had a change that made it so entity names would no longer end up having double underscores - this affected a bunch of my entity names, and fortunately all it took was one sed command to fix it in the YAML, instead of having to edit 100+ entity names in the UI. Or when I switched to a new NAS and archived the configuration to integrate with the old one - I recently found a use for my old NAS, so all I had to do was dig the file up out of my archive and add one #include statement, instead of having to reconfigure the dozen or so entities and automations from scratch through the UI.

Perhaps a good intermediary would be to make it so the configuration file is the permanent storage, and the database are more ephemeral changes on top of that? So by default, the configuration is read from the YAML file; when you are ready to make changes through the UI, you turn on "Edit Mode", and all changes are applied to the database, but not to the file; and then when you leave "Edit Mode" you have the option to commit the changes back to the file, or discard the changes and reload the file? In a worst-case situation, you can nuke the database and only lose your recent changes.

That said, I do want to point out that this is nitpicking for the sake of providing you feedback - the UI approach might not be my preferred option, but it works just fine.

Having it in the config file allows me to track changes via git and also share configurations with others. Keeping it in the UI only makes that much more difficult.

I agree with this sentiment. Also, I just prefer yaml because I’m comfortable with it and I like having a central place that I can edit my config in a terminal or editor.

Sharing configurations will now be sets of screenshots and not code blocks.

Disclaimer: I'm a linux admin by day, not a typical user.

Because text configs are much much easier to manage than DB based configs. Text files are repeatable and easily manageable.

Text files also allow me to use config management so I can have repeatable builds so it's trivial for me to change a variable and have a full clone of my hass setup in a dev machine.

Note: When I say text files I mean files that I create and modify only, not the program writing config out to a text file.

When everything is configured purely with the UI this generally means the config is stored in a database of some kind (or shudders written to a text file) this means I now need to worry about making, verifying, and testing database backups.

Perhaps it has already been discussed elsewhere, but this indicates that (a) the config UI is not the preferred user experience for a subset of the user base and (b) there should be a well-defined config strategy going forward. Possible options below.

1. Promote a Hybrid Model for All Integrations

In this scenario, all integrations would support configuration in both a UI config entry and via configuration.yaml – they cannot cherry-pick the one they prefer. Config entries would focus on the "quickest possible route" (i.e., the lowest number of configuration options to get the integration up and running) and configuration.yaml would focus on exposing/tweaking all options.

2. Promote UI Config Entries as "The Future"

In this scenario, all integrations would be slowly migrated away from configuration.yaml and interacted with via UI config entry only. Over time, the config UI would be expanded to allow more configuration options for each integration.

3. To Each Their Own

In this scenario, each integration developer is left to their own devices. If they desire one method or the other (or both), implementation is up to their discretion.

[rohankapoorcom]

Throwing in another reason why the configuration.yaml approach is preferred by a set of users: clean backups (when not running in hass.io).

With all of my configuration managed in yaml files and living on github. I can bootstrap a replacement home assistant instance by just cloning the repo down.

With config entries managed by the UI, that approach no longer works. I need to keep the storage directory also backed up somewhere and need both of those to bootstrap a replacement instance.

I would vote for a hybrid approach for that reason. Allowing users to use either approach if they have a preference but still allowing for easy backups.

[emlove]

Configuration entries are still text files. They can still be edited by hand, and they can still be checked in to source control. The biggest difference is that breaking changes can be automatically migrated between versions without user intervention.

[carlos-sarmiento]

Something I think would be a good approach is using configuration.yaml as an override. That way, everything is configurable using the UI (which makes it user friendly) but power users can override that easily using more manual configuration methods.

In effect, this would mean that:

1. If there is nothing in the UI-first configuration configuration.yaml becomes de-facto the only configuration.
2. If there is some UI-first config, a power user can override all parameters using configuration.yaml That way config is manually customizable as little or as much as a user wants
3. A new user has zero need for configuration.yaml. Everything is configured UI-first and no overrides are needed.

Another advantage to this strategy is that as we transition to more UI based configuration tools, the need for a configuration.yaml shrinks without reducing the power of the file for those who want to use it.

[Jc2k]

(Sorry this got a bit long and rambly)

I am a new user who is starting to contribute more and wondered if my "journey" through the stages of HA was in any way helpful/interesting?

• I came to HA because I could use it like homebridge (expose my Hive heating to Home.app on my iPhone) but in python 3, which im much more comfortable supporting long term than a node app. I didn't care about the implementation too much - I just wanted to map a device through ecosystems to my phone. At this point I didn't really know how HA config worked I just thought it could solve my problem.

• There were certain YAML config errors that were annoying. I had a couple of typos that meant all the entities disappeared from HomeKit on my phone, so I had to configure the homekit side on my phone all over again multiple times. At this stage I thought YAML was OK, maybe a bit annoying. I hadn't really thought about any other way (maybe I thought 'gee, a UI for all this config would be hard'). I thought it was easy to backup and that I should think about doing that some day.

• I'd been doing a lot of GitLab CD so I built a GitLab/Docker deployment pipeline that lints and pushes about my config files. I really liked that although there was some state (the recorder) it wasn't mission critical - i could redeploy from nothing quickly if it all went wrong. At this stage YAML was the best thing ever.

• I kind of hated trying to config lovelace by hand, though. As a newbie it was complicated and required lots of editing and reloading cycles. There wasn't really a UI to edit it at this point. I started to feel a bit of a paradox. I wanted Lovelace to be as easy to edit as a Grafana dashboard. But I still wanted YAML in Git for everything else.

• Then I wanted to contribute to homekit_controller. Pairing with a device involves making multiple HTTP requests and generating secure crypto keys. I definitely don't want those keys in Git, and having a UI to handle the pairing process is a lot nicer than having to pair on the CLI and update YAML files and then trigger a deployment. This component never supported adding entities via configuration.yml, and it never occurred to me that it would need to.

• I'm now in the process of porting homekit_controller to ConfigEntries. Right now I don't know where ConfigEntries are stored (I didn't need to). I just want the pairing and management to be as frictionless as possible. I don't think manually configuring homekit pairings is a very sensible workflow right now. If I lost my VM right now I would end up re-pairing my devices by hand.

I kind of feel like i've had every viewpoint on this in my HA journey so far! And this ticket has made me realise i'm still quite conflicted about what I even want myself. I'm never going to want to check config entries into my Git CD, especially where they have secrets. And I don't like the idea of an approach where some of my yaml is in Git and some isn't.

On the whole based on my experience to date I think i'm currently a (2). If (1) was chosen, i'd want YAML configuration to enhance and enrich my existing configentries - i don't ever want to have to try and pair a homekit device by hand because i need to turn on a single advanced option for one device (luckily there are no such options right now).

[dgomes]

I'm all for (1) and guess that's what @carlos-sarmiento described, which I fully agree upon

[plett]

I came here from the link in the Reddit thread.

You're going to get a lot of selection bias on these comments - the people who already use HA, and who are likely to comment on a Github issue are much more likely to be power users who are comfortable editing a yaml file.

I have my HA config yaml in git, and use ansible to deploy HA and the config to a docker container. I don't currently do anything to back up HA's "running state" from its .storage directory. At the moment, the only state which I would lose if my docker host vanished would be the authentication tokens used to talk to my Tradfri lights.

Of the three presented options, my preferred outcome would be (1) as it means I can keep my current workflow. It wouldn't be the end of the world if (2) happened and I had to abandon my yaml, configure everything in the UI and take backups of it. I would really miss the ability to have version control, but I could probably find a backup strategy which works.

However, option (3) would lead to a mess. You'd have some modules which can only be configured in yaml and others which are only done through the UI, depending on the developer's preferences.

[ynazar1]

This thread was pointed out to me by a friend and I've been asked to chime in.

I'm also in favor of hybrid solution outlined in Option 1 by @bachya. It will provide good middle of the road option by implementing essentially a tiered config model where anything from config files overwrites similarly named entries. It shouldn't be too hard to implement it by reading whatever is in .storage (and configured by gui) then overlaying configuration parameters from .yaml on top.

Sometimes I want components to work the exact way that suits my case and other times I want components start working with a click of a button and both ways have their merit in respective situations.

[rpitera]

One thing I didn't see anyone mention which seems fundamental is that if you can't start the UI, you also won't have tools to access/edit/troubleshoot it. That's why I like @carlos-sarmiento approach. It extends the typical concept of HA supporting personal choices and providing options.

[point-4ward]

I think it is abundantly clear from that reddit thread, and many other moans across reddit and the forums, that the general consensus of the homeassistant user base is that they do not want to see things move to a gui setup unless that gui setup is complimentary to an easy to access, and easy to share, text based configuration.

When the automation ui came out, it output the things you put in to it in to a yaml file that you can access, edit and share. Everything now puts things in .storage where you can't (or you're certainly not supposed to, and you definitely can't easily share it).

As @rpitera says, if I cock something up in the ui and my system won't start, I won't have a clue what to look for in storage to fix it.

We're all here because we're tinkerers, let us tinker!

[blakeblackshear]

I am the author of the original comment on Reddit.

TLDR: I would love to see a pattern where the config directory is readonly YAML, and anything that is configured in the UI writes to .storage as JSON. I should be able to take the generated JSON for anything configured in the UI in .storage and convert it to YAML so I can manage secrets and add it to source control. On startup, homeassistant reads the YAML configuration first and then applies the configuration in .storage which can override the YAML. Seems like this could be done in a way that doesn't create two separate configuration systems to manage for the developers.

---

Over the past several years, cloud adoption has shifted the mindset to treating servers like cattle rather than pets. Basically, this means that you shouldn't be worried if a machine dies or you need to destroy it, as you should always be able to create one exactly like it very quickly. I see two (i'm sure there are more) general patterns that have become prevalent:

Configuration/infrastructure as code
In order to ensure a server can always be recreated with an identical configuration as before, you have to document every step of the configuration. The days of sysadmins making manual edits to server configurations ended as soon as you operate as though that machine may die at any minute. Creating step by step documentation isn't good enough because people often make mistakes, and they are slow. This accelerated the development of Infrastructure as Code projects (codes/scripts stored in source control) that are used almost universally for anyone managing infrastructure professionally. Years ago, I managed Windows servers and quickly realized that they were designed to be managed as pets, not cattle. The default assumption was that you were configuring things via the GUI, and you had to hunt and find ways to do scripted installs of software or configure things in the registry (i'm sure this has gotten better). In my opinion, linux has dominated the cloud space because it is dramatically easier to script and access remotely. Even Azure (Microsoft's cloud) runs primarily linux instances. I want to be able to treat my homeassistant container like cattle, not like a pet. In this regard, configuring everything from a GUI feels like a step backward.

Reducing the amount of state that needs to be managed
A service that can modify the state of its environment is the worst enemy of being able to restore a server that has died. Stateless servers and services are infinitely easier to manage. When they die, you can simply recreate them and nothing is lost. If it were up to me, I would be able to mount my source control managed config directory as read only in my docker container, and any state that I needed to maintain (even better if there is none) would be written to a database. I have always loathed the components that write files to the config directory (ecobee, html5_notifications, etc) as they make it a challenge to recreate my environment in the event of failure. I want to be able to checkout my github repo, run docker-compose up and have everything running on a new server exactly the way it was before in minutes.

In reality, homeassistant is used by a much broader set of users than those who want to manage stateless systems through configuration as code, and I benefit from the growing userbase as it accelerates development. However, I think you can have your cake and eat it too. Everything I have seen written to .storage is just JSON, which is also just YAML. I would love to see a pattern where the config directory is readonly YAML, and anything that is configured in the UI writes to .storage as JSON. I should be able to take the generated JSON for any component, etc in .storage and convert it to YAML so I can manage secrets and add it to source control. On startup, homeassistant reads the YAML configuration first and then applies the configuration in .storage which can override the YAML. Seems like this could be done in a way that doesn't create two separate configuration systems to manage for the developers. I would love to be able to manage my homeassistant users/auth as YAML too.

This way, users who want to can manage their configuration in source control as YAML files, but it is their responsibility to convert the things that homeassistant writes to .storage. It also preserves the ability to share configuration with others.

edit: Also, let me worry about the compatibility of my YAML configuration with new versions.

[OttoWinter]

A couple of notes from my side:

• Let's not forget that maintaining both config flows and YAML is a lot of work for developers.

• I'm still a bit skeptical about this whole "breaking changes" thing with config entries. Sure they can auto-migrate simple configuration changes, but those are not the breaking changes that I find annoying in YAML (as long as they're documented well enough). For example if an integration changes entity_ids (without unique_id) or some internal option changes, config entries won't be able to help either. Time will tell I guess.

• @armills While .storage is user-editable, there's no way to check it into version control. At least not if you want all your secrets exposed :)

• Also let's not forget config entries are still at an early stage (or better: are full of potential). For example editing config entry settings will surely become possible in the future. And I think there are still plenty of ways to work around all issues shown here within the GUI system (for example exporting/importing, backuping, the possibilities are endless now that we have a GUI to start actions without using the command line)

• Personally, I really liked the shareability of YAML - something that I think config entries complicate a bit (but can be solved!)

• The part where I really like config entries is the rich flows it enables. For example for the recent SmartThings integration I only had to press the CONFIGURE button and step through the guide to set up smart things. Previously, this would have involved reading lots of docs and settings things up correctly so this is much simpler. In the future I envision that the config flows will become even more responsive (i.e. guide the user through the entire setup, without telling user to read docs on home-assistant.io)

[pixeye33]

I might someday use HASS as a solution for a business in home automation.

The idea i have is being able to generate automatically the configuration via some tools and only having to start home-assistant for it to work.

I know some device entity_id would be known only once they are discovered, but i could then ask the API for the list and generate the final configuration files.

Also, for now( as said before) you can share YAML files way more easily than having to write a full tutorial saying where to click to have the same result...

[balloob]

1. Our goal for every integration is to have a config entry store the auth and options (coming soon), configurable via the UI, overridable via YAML. See the MQTT component for an example of this in practice.
2. Integrations are all build and managed by volunteers. It's their time and they decide what component configuration they implement when they introduce their integration, as long as it fits within our goal. Nothing is complete on its initial version. It takes a lot of time from both the developer and the reviewers to get things added to Home Assistant. For example, SmartThings doesn't even support sensors yet in 0.87 as it was not ready in time.
3. It's frustrating for contributors to see all these comments about things that people didn't get, instead of being happy about all the things we do deliver, in our spare time.
4. It is 100% unacceptable for anyone to tell other people how to spend their own spare time to build stuff. Imagine you're doing something somewhere, and a random stranger comes up to you and tell you that you are not doing it right.
5. It is acceptable to open a pull request and contribute a change to how an integration is configured, as long as it fits within our goal.
6. Components have always stored data. It used to be random files and we organized this in the .storage folder. People complained they didn't like random files, now people complain it's all in a folder 🤷‍♂️ Its data about what steps of onboarding are done, what the user ID of the Hass.io user is, and one file is to store the config entries. People talk about .storage like it's a replacement for YAML but it's not.
7. A lot of integrations have their own authentication schemes. YAML being read-only, it is just not enough to configure most things without having to store extra data outside of YAML. You can enter the hostname for Hue in your YAML, but you will still need to be guided to press the button and have Home Assistant generate a token. There are also integrations that will take a user/pass via YAML. With config entries we are able to use (but will not store) the user/pass to get access/refresh tokens. It's a lot more secure.
8. I don't buy the "config entries are bad because I like to share my YAML with other people". All config entries is being used for today is to store auth credentials. People don't share auth, they are sharing automations, Lovelace configs, scripts, templates etc. Those can all be configured via YAML.
9. Home Assistant has many different users. Each user has different things they like about Home Assistant. Some people love to play sys admin, other people love to create cool automations, others love building UIs, others like to optimize their house to save energy. A UI is the one thing that all these people can use. So when I choose where to spent my time, it's on UI based solutions.

[Kane610]

I don't understand how advanced settings in configuration.yaml would work with config entries that support more than one entry. Yes you have your configuration.yaml, but it is still dependent on config entries that live under a different paradigm. Better to expand on config entries.

Wouldn't it be possible to separate sensitive keys in a config entry into a config entry secrets file? If backup to GitHub is the main goal for config entries.

[bachya]

Returning to this after today's Reddit post re: the 108 release sparked another comment-storm on the topic.

I've been doing some thinking on how to keep option flows "in sync" with configuration.yaml keys – not a true synchronization (where changes to one automatically affect the other), but rather, an architectural experiment on giving integrations an easy way to access – and marry – the two data sources.

I quickly run into the problem that (I think) @Kane610 is describing above: how can you retroactively marry data from config entries back to their original counterparts in configuration.yaml? And when you do marry it up, which side – options flow or configuration.yaml – wins?

An example of mine is SimpliSafe, which accepts a configuration.yaml schema like this:

simplisafe:
  accounts:
    - username: ...
      password: ...
      code: 1234
    - username: ...
      password: ...
      code: 9876

Today, if someone were to launch HASS with that configuration, those two accounts would get imported into two config entries. From there on out, no real configuration can be done via configuration.yaml. this is most appropriately viewed in light of the code config key: changing it in configuration.yaml won't do anything; instead, I instruct users to update the code via the Integrations UI.

I could potentially solve the problem of marrying a config entry to its corresponding "spot" in configuration.yaml by using hass.config_entries, which has a (currently-private) _hass_config attribute that contains the contents of configuration.yaml. I could then search for the account whose username (which is defined as the unique ID in the config entry) matches the username defined into the config entry. Feels a little messy, but it could be done; the question is whether this principle will always be true? Put another way, can we always match a config entry's unique ID to something in configuration.yaml?

The second problem – determining who "wins" – is harder. Whose code – the config entry's option or configuration.yaml – wins? Would the user have to go back into configuration.yaml and delete the code if they wanted to configure it via the UI? There's a barrier without a clear answer.

Will think on it more.

[nikolak]

@balloob While I agree that everyone should be grateful for any open source contributions to home assistant and other projects, regardless of whether it's UI for config, or YAML support, but this is still a project that has certain guidelines. Just like you'd reject a contribution that relies on scraping web data, no matter how much effort went into developing it. And there's absolutely nothing wrong with having certain rules on what's acceptable in home assistant ecosystem.

Why can there not be a simple policy of "Every integration is to have a config [...] configurable via the UI, overridable via YAML"?

Because it seems like with the last release for the UniFi component there was actually work involved in removing the YAML support which was working, because the UI now supports all the features possible via YAML. I understand not adding new features to YAML considering that the developer might not have time or resources to support yaml and UI config, but removing existing features from YAML config does feel like the wrong way to go.

It would really be helpful for a lot of people to know what the official stance and goal on this topic is, because I do not think that your comment with "UI support, overridable via YAML" still applies, or is at the very best a personal wish from you but the project is going in a different direction.

[jjlawren]

My concern with maintaining both config methods is that it potentially adds a lot of complexity, both from a development perspective and also towards the end user.

For example, it's now possible to change options during runtime for some Integrations from the UI (gear in top-right). If these same options are exposed via YAML, which should take precedence? Perhaps the user creates a config via YAML but then decides to change it via the UI. Should it revert back to the YAML setting after restart? If they've changed it via the UI but then want to override it via YAML, which should win? Having two possible sources of truth is usually a Bad Thing.

[rytilahti]

So, the config handling has been my pet peeve for long, and I think I have written my proposal in some other issue I cannot locate right now. This following comment is how I would like to see it done on the conceptual level, so I'll try to keep implementation details at minimum.

But the gist is something following:

• Integrations expose their settings using yaml (or json), much like they expose their custom services via services.yaml. This configuration metadata will define the structure of the configuration, available configuration elements, their default values, descriptions, etc. much like what is done now with CONFIG_SCHEMA. (For extreme cases, it may be worth considering allowing developers to hook the validation to some piece of code they produce..)
• The metadata is read by homeassistant to create configuration objects, which act as an interface between the configuration and the integration.
• The configuration objects are responsible for serialization, deserialization and validation of the configuration data. This is all done in core, so it is something the integration developers do not necessarily be aware of besides defining the structure and accessing the object.
• Integrations will get pre-validated configuration objects they can act as they see fit, so all of this should be transparent to developers (outside of defining the configuration structure, elements).
• Now, on the UI side, "configurator" can modify these configuration objects directly to make changes to the configuration. The changes are again validated by the configuration objects, which are also responsible for saving the state to permanent storage.
• Integrations are (partially) responsible for proper life-cycle management. What I mean with this is that the integrations should contain logic to allow tearing them down (or simply to request them to re-apply the settings?) when asked to do so by the core. The core can then reinitialize it with changed configuration by calling the initialization routines of the integration.

The pros of this:

• Single source of truth. The running system always knows how the configuration should look like. This state can be easily backed up, as well as manually modified by those who want to do that.
• No need to synchronize between different types of ways to configure the system, which is error-prone and just complicates things. In the end everything is just in the "storage" (be it yaml/json/database, it should not really matter), which presents the full state of the configuration.
• We can apply new configurations on the fly without requiring restarting homeassistant for changes.
• We can remove lot of common boilerplate code for most common validation tasks. Your configuration metadata file defines how your configuration is validated.
• Users can freely modify the configuration file when homeassistant is not running (or depending on the implementation, homeassistant can simply reload the configuration file when changed). These settings can then again be modified in the UI as usual, everyone is happy.
• The configuration object allows us to map the configuration data to UI elements. This allows automatic creation of basic configuration for all integrations without any boilerplate required from the integration developer's side. Developers can, if they choose to do so, customize their configuration views.
• Another side effect is that we can use the metadata for documentation generation, so no more outdated docs. We can always map specific commit/version/release to a specific metadata file.

Some tricky parts that I can think of:

• This will obviously require quite a bit of design & implementation effort, but I think it's still worth it at the end.
• How to handle migrations? Should the configuration be versioned, and allow integrations to provide code to migrate? In the end, it would be the code who decides the configuration file contents.
• The lifecycle management can be tricky, but doing this properly would allow easy configuration for custom integrations (i.e., those installed from HACS).

What do you think? This way, in my opinion, it would be possible to keep both types of users (those who want UIs and those who want manual configs) happy while removing lot of cumbersome code duplication.

[kquinsland]

I'm a bit surprised that more of the commenters from the reddit 0.108 release thread are not present here...

Like many other people in this thread, i'm in the systems design/admin trade and put an extremely high value on the ability to modify a file and reload the process to see the configuration change take effect. Git repos are the new pets, servers are the new cattle as it were. I won't go as far as to say that i refuse to adopt a new release because the configuration of some critical component can't be managed by my existing git/file based flow, but I am concerned that each release will bring more things that can no longer be managed via some automation techniques borrowed from my professional duties.

I do understand that my needs are not going to align with the bulk of people that HA is (increasingly) aimed at, but I'd like to echo the calls for a more formalized and, possibly core integrated, way to configure things. A HA version of Py-Cerberus perhaps? @rytilahti and others have done a very good job of expressing most of my thoughts far more concisely than i could have; API First, backed by user-accessible file(s) on disk. However, i'd like to make a few additional points that I feel are not being discussed or serve as further support:

• A formal way for everybody to interact with the configuration for a given object will make building out the HA API a lot easier, especially for any module that may need more than very basic read/write permissions. As part of a formalized configuration metadata schema (see post by @rytilahti above) adding permissions will make the multi-user/role based API permissions much easier and more powerful. A simple example: every user/role can read the value of a configuration, but only users/roles at or above some level can edit configuration values so long as the value is between some bounds set by a higher privileged set of users/roles. A thermostat or media volume comes quickly to mind. I recognize this may seem like an orthogonal feature, but you're a fool if you don't think that enhanced ACL/Policy controls for the central automation hub are going to become a requirement as more things/devices in and around the home gain 'smarts'.

• Curation of configuration: It's really difficult to curate and maintain screenshots of configurations. When i first started working with HA, the amazing collection of public HA config repos quickly showed me many different things that were possible that i'd not considered trying. Bonus: I was able to copy, modify, deploy with minimal effort. Similarly, it's impossible to ctrl (or cmd) + f on an image or screenshot of a UI driven config. Using proper formatting blocks on the forums still seems hard for some users, requiring that they take a screenshot and then annotate it or host it somewhere seems like the barrier to getting community support will be higher.

• Versioning become an implicit requirement which implies rollbacks. I'll assume that any change made via UI is sent via the same API that the yml/json config file parser would use to communicate the state of a component to HA core. I am also assuming that any change communicated via the API is written back to disk for persistence. Versioning would allow a user to roll back the breaking changes they made in the UI even if the HA web UI failed to load. This will also help those more experienced support the users that need help if the user w/ the issue can communicate something to the effect of: "i did $this, and now things wont work, but as soon as i replaced the config file(s) with the $this-1 version, HA started back up again... please help!" They get back on their feet quickly and the rest of us have a very good idea of where the problem lies so we can quickly narrow down what the appropriate next steps will be. Assuming the problem can be corrected, it's likely that a working text based config will be in the thread. Future googlers will be grateful that the issue and corrected config were indexed!

• UI based configuration can be 'automatically' built. Yes, this will be much harder than it sounds for more complex configurations; even developers that prefer to build their UIs strictly from code and their platform's built-in widgets often need to override and tweak things. However, most of the property:type tuples that a configuration metadata file would expose for a given component, will have basic mappings to existing UI elements. It may not be pretty or the most usable, but a developer may not need to do any work to get a functional UI for the user to configure. The developer only needs to specify that the component requires a username:string and a password:secret and the necessary text fields can be shown to the user if/when they load up the UI configuration for said component.

TL;DR: a formalization for configuration metadata/schema will be non-trivial, but has many benefits that will make HA a more robust platform that's easier to support and develop for in the future. A patchwork of components all with their own preferred configuration options and a collection of tightly-coupled json files does not do anybody any long-term favors.

[dkagedal]

Another approach that is popular in many tools that combine a user-friendly UI with a batch- or scriptable interface is to define configuration in terms of a configuration "scripting" language, combined with a transaction system.

Ubiquii's EdgeOS is a good example of this (as is Cisco IOS etc). Another example is if you dump a SQL database into a series of CREATE DATABASE and INSERT statements etc.

In these systems, configuration can be driven from a CLI, and saving the currently active configruation produces a CLI script that will recreate it. A configuration script would do something like

edit configuration
  remove entity group.some_lights
  add entity group.some_lights
    entities: light.one, light, two
  save

Nothing actually happens until the save (or commit) step. Instead of save, you can do a diffto see what would change if you commit. When saving, the diff will be applied, and doesn't need to involve actually removing and adding the entities if just an attribute changed in the diff.

Any change done through the UI can be expressed by such a configuration snippet, and allows for having just a single entry point for configuration changes, a replayable log, and more.

This would obviously be a very significant change and maybe not feasible, but I wanted to include it in the discussion.

[Jc2k]

The previous 2 comments have both made me think of systems like Kubernetes or Terraform or CloudFormation where configuration can be expressed as a manifest in e.g yaml and applied via the CLI. The are idempotent and could have a dry run mode.

This is probably more achievable than a scripting approach.

You can then check in your manifest in git, share it on reddit, whatever. But it means that the actual CRUD is against the HA api, not by manipulating the state on disk.

Changes made this way wouldn’t need a HA restart.

Such a tool could be written for anything that already is UI only, and HA has a command line tool this could live in.

[Jc2k]

If we riffed off Kubernetes a bit it might look something like this:

kind: Integration
version: v1
domain: esphome
selector:
  ip: 1.2.3.4
password: apple-and-pear
---
kind: Scene
version: v1
name: movies
entities:
      light.tv_back_light:
        state: on
        brightness: 125
      light.ceiling: off
      media_player.sony_bravia_tv:
        state: on
        source: HDMI 1
        state: on
---
kind: Automation
version: v1
name: turn-on-lines
  trigger:
    platform: state
    entity_id: device_tracker.sweetheart
    from: "not_home"
    to: "home"
  action:
    service: scene.turn_on
    entity_id: scene.romantic

Some thoughts:

• Like kubernetes this should be brutally simple. If you want to complicate your config with variables, use a yaml templating tool. We don't need to deal with bikesheds about that in this tool.
• This is idempotent. Applying the 'Integration' type multiple times doesn't create multiple config entries.
• The versions are schema versions, not config versions. Kubernetes has versions like v1alpha1, v1beta1 and v1 and there are windows where the old versions are still supported as well as the new ones.
• We don't need the complication of includes with this because you can structure your manifests as you like. One big file. Lots of files. Nested directories. Just invoke the CLI to apply them as you want. We don't need to deal with bikesheds about that in this tool.
• For simple config entries I can see we might be able to statically provide answers to the pairing flow in a manifest like this. For others, we should be able to show the config entry prompts on the CLI.
• We could use the existing validation that the API has to make sure that manifests are valid.
• I think we could target a large part of the existing API surface without having to bother the core time and be able to manage things with yaml files outside of our HA installs.

[dgomes]

I like these two last comments!

We could move the source of truth from the yaml files to .storage as long as we have an API against which we can push the yaml files through the HA CLI. Half of it is already done, what is missing is a "dump" API of the configuration.

[frenck]

The strategy has been set out in ADR-0010: #366

[TechnicallyJoe]

Seems incredibly unfortunate to close this issue, considering the community response to it. Perhaps it would be better to have a healthy discussion regarding the possible solutions proposed here?

I personally like what @Jc2k has proposed.

[MrAdam]

I'm with @Jc2k and @TechnicallyJoe on this.

It seems like a solution was picked without further discussion with the community, which has proposed some very good solutions to this issue that should work for both power users and newcomers.

I've seen talks on the forums and on reddit about forking Home Assistant, which would be unfortunate, but understandable given the unwillingness to listen to the users in here.

I would hate to see the community split up, as fragmentation helps no one.

[Jc2k]

As i've been tagged twice - full disclosure - i support ADR-0010 and I am not someone who manages my config via yaml (any more). As much as possible i use the new UI features exclusively.

And in addition my proposal isn't "the core team should build a CLI for applying manifests for us". My proposal is "we can build this tool using the API's that are already there". So I don't think what i proposed isn't blocked by ADR-0010 at all.

It might be that API's might need improving but again, ADR-0010 does not forbid that either if the change is worthwhile in its own right.

We only rub up against ADR-0010 when we have some yaml config using the new approach and some using the old approach. If my suggestions are that successful then maybe there will be a conversation to have here, but it will take quite a lot of work to build something that is mature enough AND get the adoption so it is premature to issue an edict about it, I think.

I have started toying with a prototype. When its a bit further along i'll throw it up to get PR's.

[geiseri]

<rant>
As someone who has been active in the OSS world for over 25 years, I came to Home Assistant because it looked very pretty, and is very polished UX. Now one week in, I am starting to get a bit frustrated. Once you start trying to hook up anything outside of the core parts you require YAML. This is actually VERY nice vs JSON or TOML. The thing is now I have this mishmash of configs in two places. Databases are horribly fragile compared to YAML so that also concerns me even more. I really like the Lovelace approach of edit in YAML, or UI. The other issue is that since the general documentation of how to hook stuff up is so very light outside the code I end up finding nice YAML examples everywhere. Search, copy, paste, and tweak rocks in the world of Stack* and are lightyears better than any install wizard. These are handy but only amplify the issue.

I think while the low barrier to entry with configuration via the UI is a noble goal, but it's flawed. If someone wants a simple one size fits all you just use Google, Amazon, or something that does it all for you. I don't care how passionate the developers here are, they will NEVER keep up with a paid army of developers, graphic designers, tech writers, project managers, etc working 40 hours a week on only a product. Nor should they. (Unless the end goal for developers here is to make a platform as a service and have paid developers to do this. While totally valid and very likely profitable, they need to be honest about that.) If this is not the goal I think it should stick with a geek-friendly way of doing things, that is pretty and easy to tweak. I use OSS because I want to have the opportunity to control every aspect of the system. When it works great, but when things go south (read not my opinion of how it should work), I want to be able to dig into it and fix it (read make it my way of doing things). If I didn't care I would just pay for Google or Amazon since my time amounts to the same thing.

I do understand and respect where @frenck and @balloob are coming from, and in OSS, those who write the code make the rules. I more agree with @rytilahti's opinion and think the decision not to articulate the configuration flows in YAML, or even just dump/restore YAML is a mistake.

Just my 2 cents as someone who codes for fun, reads source code recreationally, and really likes the platform he has found.
</rant>