YAML-level diffing & rollback

[richardgaywood]

When coding with Claude, I like being able to keep an eye on it by looking at PRs and/or the commit history. And I can easily recover from any mistakes it makes with a few git or jj commands. This, in turn, gives me the confidence to lean into it more.

I'm slightly hesitant about letting it run amok over my sprawling HA config, however, because I don't have that fine-grained recovery to fall back on if everything goes kablooey. The agent does seem pretty good about showing me diffs as it works, but that's still not an easy recovery unless I keep careful track of them. I might not notice an issue with an automation right away. I could restore a full HA backup, but that's slow, and would lose entity data between the last good backup and whenever I notice an issue. And I'm wary of the way I've seen these agents silently scope-creep and start rewriting things they shouldn't be touching.

Is there room for a rollback process built into the MCP server itself? One that automatically captures the YAML being edited before and after each change it sends to HA. I don't think it needs a very sophisticated UX, I'd be happy just to have timestamped files somewhere I could look at by hand. I don't suppose the agent goes wrong that often or nobody would use it. But nevertheless, this would buy me some peace of mind, and that would make me more confident to use it more extensively. I suspect I'm not alone in this.

(I would be happy to try and put a PR together if this is a good idea!)

[kingpanther13]

Ha-mcp shouldn't touch any yaml directly unless you are using the opt-in beta yaml editing tool. I think that when it edits yaml with the beta yaml editing tool it is supposed to make backups.

However, I highly recommend NOT using the yaml editing tool. We placed it into the dev server only for a reason. There is very little that it can do that the current tools cannot do. The tool really only exists for people that still have legacy things set up in configuration.yaml . Almost everything else can be handled by helpers, template sensors, that kind of thing.

AIs are very happy to jump to saying "place this in configuration.yaml!" But it is very rare that that is actually needed. Just using all of the other tools in ha-mcp is very safe and relatively easy to roll back especially if you ask it to make backups ahead of time.

[richardgaywood]

Ah sorry, I worded the above confusingly.

I didn't mean "when the MCP server is editing the HA config that only lives in YAML" - i.e. your configuration.yaml and so forth. Agreed that this is an anti-pattern these days! Made worse, I suspect, by the models having a lot of stale training data from the 2001-2003ish era when you had to use those files to do a lot of stuff.

What I meant was when the MCP server is editing other bits of normal HA config, like dashboards and automations. When I ask the model to show me previews before it makes any changes, it dumps out YAML diffs (I know HA uses JSON behind the scenes, but I assume the model just uses YAML for the RPCs.) It is this layer that I wondering about, if some simple version management here might help.

So if we consider a simplifed flow through the stack:

user -(1)-> LLM -(2)-> MCP server -(3)-> HA integration -(4)-> HA server -(5)-> written to disk

I am assuming/guessing that step (2) is done in YAML. The model asks "what is the config of $FOO card on $BAR dashboard?" and a lump of YAML is returned. The model chooses how to mutate the YAML, asking the user when it feels it necessary, then sends a new lump down. The MCP server passes that through (3) and it is written to HA itself. (If I understand correctly, it's converted to JSON in (5), but I don't think that matters.)

So what I am wondering about is some kind of versioning that stores diffs to the YAML in step (3) or (4). So the user can go Settings | Apps | MCP Server and open a page that says "ten minutes ago, your agent made a change to four cards in the $BAR dashboard", and click a button to undo that change. And this happens at a layer under the LLM, so the LLM can't mess it up.

Think of it as the same thing as "hey Claude write code to do X" (time passes) "Oh my god what is this nonsense it has crapped all over my repo, git reset --hard HEAD".

[kingpanther13]

This is a very good idea. Our beta yaml editing tool does something similar I believe, but I don't think our other write tools do this. I would have to confirm. I personally have manually been asking my AI to make backups before making big changes, but setting up an automatic flow would be great for sensitive things like automations dashboards scripts and etc.

I think the easiest way would just to be to make a backup of the entire thing that is being edited each time and include a timestamp. So, we make a backup at the begining before doing any writing or editing, then we can easily revert or easily do a side by side comparison of changes. It can be stored directly on HA in a folder specific for our mcp server. That can be managed completely by the AI, and on the server side other than maybe a toggle to enable/disable the feature.

Having an interface within the addon (or more realistically within our new web interface introduced by #960) is a neat idea too but would add a lot of extra complexity, so would probably better to have that as a separate PR.... Just having the backup system in general would be a nice big addition. If you're willing to make up a PR for that then please feel free. I'll go ahead and make an issue to bring up both things, the backup system and the interface for it.

[Patch76]

For thread continuity: #1288 was filed as the implementation-tracking issue for this discussion. v1 scope is the backup capture + toggle; the browse/restore UI is a follow-up.