Controlling a switch with Insteon link is not reflected in the HASS UI.

[majortom64]

@teharris1 You wrote (in #44):

The other issue you are seeing, if I understand you correctly, is that state changes do not cascade along linked devices. In other words, if switchA controls switchB and switchB controls switchC, changes to switchA do not effect switchC. The PLM is like any other device. So the PLM can change the KPL the changes to the KPL don't cascade to the devices controlled by the KPL. This is by design in the Insteon model.

That is not the issue I am describing.

The most basic form of this is that if I turn on an Insteon device using an Insteon link, the HASS UI does not update.

For example:

Two SwitchLincs that have Insteon links between them. If I change the state on the SwitchLinc from the SwitchLinc, the HASS UI updates the state correctly. If I change the state by pushing the button on the other switch, the HASS UI does not update the state correctly.

Without this working, it means that the system will never know the state of a FanLinc's fan if it is controlled via a switch and/or a button/set of buttons on a KPL.

I think the code would need to understand the device links, so that when the PLM received a message from device X the system would know that device X was linked to devices Y, Z and A and set their states accordingly.

[licquia]

If I understand the PLM developer guide correctly, then you normally can't retrieve information about links that do not involve the PLM itself. And even then, the PLM is only linked to all devices being controlled if it is the controller. If the switch is the controller, then the PLM would only be a respondent, which means it would only get the message from the one switch.

There is one exception: monitor mode. In this mode, the PLM will capture and report any messages it sees that involve a device it knows about, not just messages targeted directly at the PLM. This way, it could pick up on the traffic between the two switches and pass that to Home Assistant.

I say "could" because Insteon bridges don't forward traffic farther than they need to. So, for example, if your PLM was on the "other phase" of your home's power network from the two switches, the dual-band devices on your network wouldn't necessarily forward those messages across the phases. That would mean the traffic between the two switches wouldn't even make it to your PLM, and Home Assistant wouldn't see it.

This isn't just a theoretical problem; here's a similar report on this problem on the Smarthome forums that appears to predate Home Assistant:

http://forum.smarthome.com/topic.asp?TOPIC_ID=13316

(FWIW, I believe the responder is slightly incorrect, in that the PLM might see the command from A to B, as well as the ACK from B to A, and software attached to it could react appropriately to that traffic.)

This is one reason why I only link my Insteon devices to my PLM, and handle scenes and complex automations in Home Assistant rather than with direct links. If we want this to be both reliable and easy, we probably want something like the Z-Wave control panel in Home Assistant for setting up Insteon links. As worthy of a goal this might be, the scope of that project would cover a lot more than just the python-insteonplm module. And even in this scenario, you'd still likely have problems seeing links that weren't set up in HA, because that's just how Insteon works.

All the same, implementing monitor mode could improve the user experience for some people. It might require modifying the HA drivers as well as python-insteonplm. But it could cause just as many problems, as some links would work properly and some wouldn't because of mysterious factors involving the exact layout of your home's wiring. That would be a nightmare to support for anyone, never mind an open source project run by volunteers.

[teharris1]

@majortom64 I think this is a good stretch goal for this project. However, per @licquia comment, this has two big challenges:

1. It is a ton of work to develop
2. Will likely be unreliable
   In the mean time, there is a reasonable workaround in HA. Set up a scene for the devices that control one another. For example, I have two sets of switches that are used as two way switches. I have set up two scenes for each of these switches. One for on, and one for off. Each switch can trigger the scene and when the scene is triggered it sets the other switch. This way HA reasonably knows the state of each switch. It is still not perfect but it does cover a good bit of the issue you are refering to.

[majortom64]

Sorry this took so long, been traveling internationally.

If I understand the PLM developer guide correctly, then you normally can't retrieve information about links that do not involve the PLM itself. And even then, the PLM is only linked to all devices being controlled if it is the controller. If the switch is the controller, then the PLM would only be a respondent, which means it would only get the message from the one switch.

Every Inseton device needs to be linked to the PLM for any of this to work. From what I have seen, the PLM can read the link database for every device linked to it. That would be all that was needed to correctly maintain state. What would need to be done is for HASS to maintain a link graph of the Insteon Network, so that when switch A sends an ON and the PLM receives it, the system would know that Switch A is linked to Switch B, Switch D, Button 5 on KeyPadLinc G, LampLinc L and LampLinc O. It would then need to update all those devices in its UI to match what was sent. In addition, it would, by default, send the commands to all the linked devices when someone changed its state from the UI.

This is one reason why I only link my Insteon devices to my PLM, and handle scenes and complex automations in Home Assistant rather than with direct links.

This approach eliminates Insteon’s nicest feature, not having a single point of failure and requires all control be done by HASS.

If we want this to be both reliable and easy, we probably want something like the Z-Wave control panel in Home Assistant for setting up Insteon links. As worthy of a goal this might be, the scope of that project would cover a lot more than just the python-insteonplm module.

That would make this a much more valuable system. It could be done either in python-insetonplm using the command line tools, or a stand-alone GUI - not via HASS. (I have a feature request for some level of link management).

And even in this scenario, you'd still likely have problems seeing links that weren't set up in HA, because that's just how Insteon works.

Again, that is not my understanding or experience. One can read the link DB for any device that is connected to a PLM.

[majortom64]

@teharris1 I accept that developing a link graph and then updating the UI based on understanding the link graph would be a bit of work. I am not sure why you think it would be unreliable. Every Insteon device is linked to the PLM. When any button is pushed, it sends a message to the PLM (as it also does to every other linked device) and the PLM can update the UI based on the link-graph.

Your work around is Ok, just generates extra Insteon network traffic (the original switch turns on the other switches and then HASS does it again). Instead of having the system build the link-graph, the user does it.

A first pass improvement would be a way to update the state of things in the UI without sending Insteon commands.
Once that was possible another interim step would be to read the link DB, using it to generate the scenes and add the appropriate new links to each switch. :-)

I want to be clear that I am not minimizing the work involved in building a link graph, but just want to be clear that doing so would make Home Assistant's Insteon support much better (and would make it much easier for existing Insteon user to migrate as they would not have to wipe all their links and reconfigure them to just be controlled by HASS).

[teharris1]

@majortom64 Welcome back.

Your understanding is correct. And yes, I believe your idea will work 90+ percent of the time. I have been doing a lot of work with the device all-link databases and here is what I have learned that make it a lot of work:

1. Reading the device database is VERY flaky. 75% of the time devices don't even respond to the request. I have been tracing HouseLinc messages and even it requires 5 or 6 attempts before it gets a record. This is the biggest problem. I don't know why devices have such trouble with this request but it is very flaky.
2. There are at least 3 different database schemes, v1, v2 and v2cs. There may be more. The same device manufactured a different times appear to have different database schemas. There is a query to find the engine version but it is also very poorly responded to.
3. Database changes are not communicated to the network. They are only communicated between the two devices that link. This means that any link changes need to be queried rather than listened for. In other words, a link graph that changes today will not be known about by the PLM until the PLM asks if any changes have occurred. Then you are back to the issue in number 1 above.
4. Optimistically, if you can create a link graph and you see a device change you can assume the linked device changes. This assumes the linked device received the message and responded correctly. If the three issues above are resolved (which I believe we can since other software like MisterHouse have done it), then I would prefer to take a pessimistic view and ask the linked device for it's new state. This way you are not assuming but in fact know the state.
   I agree this is a good feature to have and I would hope to have it ready in the next 6 months. But it is a lot of work and I am concerned about how reliable it will be.

[mstovenour]

Reading the device database is VERY flaky. 75% of the time devices don't even respond to the request. I have been tracing HouseLinc messages and even it requires 5 or 6 attempts before it gets a record. This is the biggest problem. I don't know why devices have such trouble with this request but it is very flaky.

On the Misterhouse project we spent a lot of time making this more stable. I'd say that ALDB read/write failures are relatively rare on the release I'm running. The problem is that both the PLM and the devices are sensitive to doing things too quickly. Pacing was added based on failures; more failures slower queries. Also we implemented a sophisticated system for setting the hop count. It turns out that always setting the hop count to 3 can cause problems. I suspect Insteon is susceptible to broadcast storms. So initial communication to a device starts out with hop count 1 and then it gets increased on timeout / retransmission. The hop count moving average is then stored for every device ID and future messages start out with the average.

There are at least 3 different database schemes, v1, v2 and v2cs. There may be more. The same device manufactured a different times appear to have different database schemas. There is a query to find the engine version but it is also very poorly responded to.

When implementing the I2CS devices on Misterhouse, I didn't see any issues getting the initial status messages (0x19). There is one complication in I2CS but it is by design. The I2CS devices must be manually linked to the PLM before they will respond to the message and before the PLM will forward the response. The exception is the 0x19 status request. That will always be NAKed or responded to even if the device is not linked. In Misterhouse, I wrote the startup to query 0x19 on all devices. The response then told me if it was I1, I2, I2CS unlinked, or I2CS linked. BTW, I don't recall the ALDB being different between those; but the I2CS does need the checksum added.

[teharris1]

@mstovenour great to have you partisipating here. THanks.

I feel like the current ALDB loading in insteonplm is pretty good since it does get the ALDB almost all of the time. The issue is the time it takes. The approach taken is to attempt to read all records 5 times. If that fails (fully or partially) then try to read the single records up to 20 times each. The time delay between attempts is static but your suggesting of an increasing delay makes a lot of sense. That could be a good incremental improvement over the current approach.

The only other issue is the lack of support for querying battery operated devices but that is a general problem rather than an ALDB load problem. There is a dev request for battery operated devices but it has not made it to the top of the queue yet.

Regarding 0x19 messages, that is exactly how insteonplm works. On startup all devices are queried for the current status. It works well but as you point out, it is only reliable if the PLM/modem and the device are linked.

[mstovenour]

I had to go look at the code; it's been a while since I had to mess with it. I'm using the 0x0D command which returns the Insteon Engine Version. That command will return NAK with cmd2=0xFF if it is I2CS and not linked. Otherwise it will return the engine version.

[bit-bat]

I am new to whole Insteon protocol so my idea might not make any sense. Could the software keep a list of all links and whenever a controller is pressed the software can check the status of responder after a few seconds (like we do during boot up)?

[mstovenour]

I've been working heavily on this for the past few months. It is a lot of work but I am getting closer. As you've stated my approach relies on the PLM and device ALDB entries to know which device endpoints respond to each controller group. I have it working from the PLM to the device where each PLM controller group shows up as an insteon device in Home Assistant; you can toggle the HA device and the group will go on or off while updating the local state of all responder devices. Now I am very close to getting the opposite to work as well. (i.e. some controller device issues an all-link broadcast/cleanup sequence and Home Assistant changes the local state of all responding devices). This is the OP's original request in this thread. Sadly I am very slow... I might need another few weeks; but my goals is to finish by end of year.

[jsenecal]

Hey @mstovenour, how is your progress so far? I have a spare PLM that I could dedicate to testing, giving feedback, and I am a seasoned Python guy... I'm new to HASS but all to say that I would gladly help.

[teharris1]

Closing this issue as the insteonplm module is no longer maintained. Please use the pyinsteon module instead.