Issue migrating Insteon from ISY to native HA integration

[bytenik]

The problem

I have a pretty extensive Insteon installation in my home -- perhaps 150 devices. Some of the devices are modules inside boxes and are not the easiest to get to. I do however have the addresses of every device, and, they are currently added into an ISY that is still up and running. I purchased and installed a new USB PLM hooked up directly to my Home Assistant computer. I have tried to manually add in each Insteon device by address. However, they all just stall and do not add. I however just discovered that new Insteon devices that are not connected to the ISY yet do pair successfully and work completely fine. But Insteon devices do support multiple PLMs. I'd be willing to reset my Insteon devices except that to do so, I would have to open up dozens of electrical boxes, some of which are underneath ceiling fans and etc.

What version of Home Assistant Core has the issue?

core-2023.5.3

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Insteon

Link to integration documentation on our website

No response

Diagnostics information

home-assistant_insteon_2023-07-10T12-16-22.644Z.log

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @teharris1, mind taking a look at this issue as it has been labeled with an integration (insteon) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of insteon can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign insteon Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

insteon documentation
insteon source
_{^{(message by IssueLinks)}}

[bytenik]

Is there any additional information I can provide to help with diagnosis/resolving the bug?

[teharris1]

Do you know if the ISY can place a device into linking mode? If so, you an put the device into linking mode using the ISY then use the "add device" button in Home Assistant. Not 100% sure it will work but it should.

[bytenik]

It does not appear that the ISY can do this. HomeAssistant should be able to do it, though -- other systems are able to link to devices that already have devices like the ISY in the link table. Can I help with finding that bug in some way?

[teharris1]

You can put the integration into debug mode which will identify what is happening under the hood. I can also ask the team at Insteon if they know of any reason this would not work. Linking is very straightforward forward but not all devices support remote linking.

Have you considered just using the same PLM that you use with the ISY? The devices in question are already linked there. Why switch PLMs at this point?

Another thing you could test is to attach the new PLM to the ISY and ask the ISY to link the devices to the new modem.

[bytenik]

How do I put it into debug mode? Fwiw I'm a software engineer, and while I've never worked on anything HomeAssistant related, I'm happy to poke around if you give me some pointers.

Its a serial PLM, and the new one is a USB PLM. The old PLM is also a bit flakey, possibly the capacitors inside are starting to go, so I'd prefer to migrate.

[teharris1]

Sorry for the delay here, work has been extremely hectic lately. You can put the integration into debug mode by going to Devices and Services, click on Insteon and look under the '...' menu.

Also, on the ISY, can you check if the Program Lock flag is checked? That might prevent the new PLM from being able to change the device.

[vult2000]

I am in the same boat. Coming from isy994. I have reset a couple of switches to see if they will link to the new USB PLM. Home Assistant says it discovers them and shows the Insteon address of the switches after clicking add devices, but they never show up in the devices list.

[teharris1]

@vult2000 I assume from your description that you do receive a message that the devices have been added successfully to Home Assistant. If that is the case, please try restarting HA and see if the devices show up. I believe there may still be a bug were devices add a certain way do not show up until after a restart.

[vult2000]

@teharris1 restarting does have them show up, but everything shows as a Generic General Controller.

[teharris1]

@vult2000 please delete the fille /config/insteon_devices.json. This file stores the device data. It is safe to delete. Deleting this file will force the system to requery the devices for their actual model information. I am concerned that the devices did not respond to a query correctly, and that they may not be linked correctly. If, after deleting the file and allowing the system time to require them (5 min or so), they still do not have the correct information then please try relinking them. Use the Set button if you can to see if that helps.

[vult2000]

@teharris1 , that actually worked after having to relink them. Now however, I cannot create an insteon scene. No matter what I do, it says that it can't write the scene to the devices.

[bytenik]

Sorry for the delay here, work has been extremely hectic lately. You can put the integration into debug mode by going to Devices and Services, click on Insteon and look under the '...' menu.

Also, on the ISY, can you check if the Program Lock flag is checked? That might prevent the new PLM from being able to change the device.

Didn't see this reply until just now. When I'm home later tonight I will try this out and see what I find.

[teharris1]

@bytenik OK, let me know how that goes.

@vult2000 Are you able to check if the "Program Lock" flag is set on the devices? Also, send me a debug log of adding a scene.

[vult2000]

@teharris1 , could you point me in the direction of that program lock flag?

[teharris1]

If you click on the "Insteon" link in the left panel of HA you will see a list of your devices. When you click on any of those devices you should see a list of properties. The first one is usually "Program Lock". If it is set to "True" then it is set, otherwise, it is set to "False" and is not set. (False is good in this case).

[bytenik]

Okay, I've tried a few more times. Once, I got the same behavior as @vult2000 has -- it added, I had to restart, then I got the generic device behavior, and deleting the json file you mentioned and restarting didn't resolve it. But almost every other time I got the same behavior. I've included a log of that.

home-assistant.log

I know not "every" device supports remote linking, but every device in my home was added to the ISY via remote linking. They're also all modern dual-band Insteon devices and they all support it. This is definitely a problem specifically with HASS Insteon and devices previously (still?) linked with another PLM.

[bytenik]

P.S. none of the devices have program lock enabled.

[bytenik]

Actually, having deleted insteon_devices.json about 15 minutes ago now, many of the devices that I did have working (devices that were not linked before and so they were willing to add) are now showing "An Insteon device was not found."

[vult2000]

@teharris1, confirmed nothing has program lock enabled. I thought maybe it would be easier to just make my serial PLM work instead since it already had all the devices linked to it. Home Assistant sees the new PLM (and still shows the USB one too), but it does not autodiscover any of the devices linked to the PLM and seems to be having trouble controlling the ones I've manually added via Home Assistant. Super frustrating.

[bytenik]

@teharris1 Is there anything I can do to help figure out this issue? Its been about 2 months now that I've had trouble with Insteon, and I'm happy to help in any way I can to solve the problem.

[bytenik]

Is this plugin under active development? From what I can tell, its totally broken. I'm happy to lend development effort as a former Insteon developer from 15 years ago, but I'd need some pointers into the codebase. But I've been waiting for 3 months now to even get some direction on where to look in the code. If I shouldn't be expecting real support for Insteon in Home Assistant, please just say that so I can move on to other potential avenues of controlling my home.

[Doughboy68]

@bytenik I am curious also. I am down to abut 40 Insteon devices but I currently use a Polisy and the HA ISY Integration which works real good... however I would prefer to get rid of Polisy. I did try months ago this Insteon integration using one of those USB key style PLM (I also have a couple full size USB PLM to test)... I was able to add a keypadlinc and messed around with it for a bit but did not figure out how to delete/remove it, and the entities seemed to linger around in HA.. I ended up removed the integration and maybe that json file also but I cannot remember for sure (which fortunately removed entities) and didn't look at again. I did find a program called LinkNet 2020 which was able to reset the PLM so maybe I will try again on a fresh test box.

Were you able to add a "new" insteon device and then delete it from the USB PLM in HA? I was not able to figure out how to do this.

As far as your original issue, maybe a workaround is to Delete the device from dropdown menu in ISY, and all the links should be removed in process, then try to add it in HA Insteon integration.

[elissonm]

@Doughboy68 Would you mind sharing the LinkNet 2020? Their website went down a couple of years ago and internet archive doesn't have a backup of the installer.

I'm also having the same problem with sensors being detected as generic devices. I currently use the discontinued Hollie's Misterhouse to control my Insteon devices with a Google's voice recognition work around. Its responsiveness is really fast and it works with my 2448a7 regardless of linked devices with it or other 2245-222 hubs. But I'm trying to find a plug-and-play alternative that works with Alexa. It looks like this Insteon module for HA is the best bet. I hope these sensor problems get fixed soon.

[teharris1]

@bytenik sorry for the long delay to respond. I finally looked at the log you sent back in August. The issue you are having is that many of the devices are rejecting the remote linking request with a direct NAK with reason code 0xFF which means it will not accept the request to link with the new modem because the new modem is not in its All-Link Database. I know this is a circular description but it is probably a security feature of some firmware versions. Here are the log records I am looking at:

2023-08-17 19:08:37.205 WARNING (MainThread) [pyinsteon.managers.device_id_manager] Device 23.12.55 refused the Device ID request with error: direct_nak_aldb (0xff)
2023-08-17 19:08:38.707 WARNING (MainThread) [pyinsteon.managers.device_id_manager] Device 29.1F.BB refused the Device ID request with error: direct_nak_aldb (0xff)
2023-08-17 19:08:39.091 WARNING (MainThread) [pyinsteon.managers.device_id_manager] Device 21.86.70 refused the Device ID request with error: direct_nak_aldb (0xff)
2023-08-17 19:09:03.604 WARNING (MainThread) [pyinsteon.managers.device_id_manager] Device 23.12.55 rejected an ID request with error: direct_nak_aldb (0xff)
2023-08-17 19:09:14.140 WARNING (MainThread) [pyinsteon.managers.device_id_manager] Device 21.86.70 rejected an ID request with error: direct_nak_aldb (0xff)
2023-08-17 19:09:16.667 WARNING (MainThread) [pyinsteon.managers.device_id_manager] Device 29.1F.BB rejected an ID request with error: direct_nak_aldb (0xff)
2023-08-17 19:10:11.433 WARNING (MainThread) [pyinsteon.managers.device_id_manager] Device 40.14.E2 rejected an ID request with error: direct_nak_aldb (0xff)

Unfortunately, I don't see the messages these devices are rejecting because I don't see the actual debug messages.

[bytenik]

I've been out of commission for a little while because the PLM hooked up to my HASS server failed.
But this even more highlights the importance of this issue. While I am planning on hooking up the ISY's PLM via a USB-Serial adapter today to get back up and running, which should clear most of this issue, what happens when that PLM dies?
The ISY is able to readopt all devices into a new PLM in case of a PLM failure. And we all know how the PLMs like to die. But it sounds like the mechanism HASS is using to pair with the various devices can't do that because the device won't allow for the linking.
Is there a difference in how linking is performed between ISY and the HA Insteon plugin?

[teharris1]

@bytenik I currently have a feature in development to replace a single device in the Insteon network with another device (ie. a switch fails so replace all references to it with a different switch). This is very different than replacing the modem (PLM or Hub). I would love to understand how the ISY does that. The issue with replacing the modem is the ability to automate linking the devices. While I know how to do this in theory (and the Insteon integration in HA does support it) there appear to be a lot of edge cases that cause odd behaviors (ie. devices don't like to be the responder they will only be a controller).

Any insight you have about how UD made this work would be welcome.

[teharris1]

@Doughboy68 the way to add a device to the native Insteon integration is to use the Insteon configuration panel that appears on the side bar. When you open that panel, there is (+) button in the lower right corner. Pressing that will open a dialog box that allows you to add a device.

Alternatively, if you manually link the device to the modem with the set buttons, the device will appear in HA. Let me know if you still have issues with that.

[teharris1]

@bytenik I took another look at your logs from 17-Aug and I see something interesting as it relates to device 25.71.D7. Based on the address of the device I can be almost 100% sure it is a i2c device, which means it should accept a remote linking command as an extended command. The system is sending a remote linking extended command but the device is not responding at all. The system then attempts (incorrectly) to send the device a standard command and it does respond to that with a message saying "sorry, I cannot do that because I need an extended command". I am going to bet all I need to do is retry the extended command a few times with some pauses in between in order to (hopefully) get the device to respond.

In the meantime, what is the status of that device in your system? Did you ever get it to link to the modem using the native integration?

FWIW, the standard command being sent is a fallback command and is by design. Most devices in the field today are i2c devices which require an extended command. But there are a good number if i2 devices which only accept a standard command. So the issue isn't the sending of the standard command, the issue is not giving the device enough retries until it feels like responding to the extended command. This is, unfortunately, a common situation when an extended command is required. Many of the extended commands need to be retried 3 or more times before the device seems to accept them.

[Doughboy68]

@Doughboy68 the way to add a device to the native Insteon integration is to use the Insteon configuration panel that appears on the side bar. When you open that panel, there is (+) button in the lower right corner. Pressing that will open a dialog box that allows you to add a device.

Alternatively, if you manually link the device to the modem with the set buttons, the device will appear in HA. Let me know if you still have issues with that.

Thanks, but is there a way to similarly remove/unlink a device from the panel, and removing the entity from HA?

[bytenik]

In the meantime, what is the status of that device in your system? Did you ever get it to link to the modem using the native integration?

Yep, I do have it up and running now using the original modem! I'm still terrified of what will happen though if that modem eventually fails.

[teharris1]

@Doughboy68 I have a pull request in that added the feature to remove a device from the Insteon integration in HA.

@bytenik I assume you have moved back to your modem that was connected to your ISY. One option for you is to add a link in each device that looks like this:

Mode: Responder
Group: 0
Target: <new modem address>
Data1: 0
Data2: 0
Data3: 0

This will add your new modem as a controller of each of the devices. This way the new modem will already be able to send a command to the device to start remote all-linking.

[vult2000]

@teharris1 I realized the task of resetting all of my devices and relinking them to a new modem was going to be too much work so I went ahead and bought a USB to serial adapter to I could use the existing modem. The problem is, even after deleting the /config/instead_devices.json file the few devices that were linked to the USB modem still appear and none of the devices that were linked to the serial modem ever appear even after choosing the "LOAD FROM DEVICE" option. Thoughts?

[rcb-hp3000]

Another user here with almost the exact setup of the original poster; existing UDI ISYi994 that I'd like to migrate away from. Over 150 Insteon devices in the home. Brand new install of HA (I'm new to HA, been running Insteon stuff for over 15 years) on a RPI with a USB 2413U modem attached to the Pi (I'm leaving the existing modem on the ISY until I'm sure HA will actually work - I have dozens of custom scripts, scenes, and schedules on the ISY). Insteon integration recognized the device but I get no options to add anything else; nor do I even get an "Insteon Panel" - though under devices it shows the single Insteon device(modem) with engine version: unknown.
Tried the UDI integration as well and that failed - apparently it won't work if your ISY is using https and a custom port?
I'd be willing to manually add all the Insteon devices to HA if I have to - but going to each device (opening panels and such) to "link" them is not an option.

[teharris1]

@rcb-hp3000 Thank you for the post. I really want to help you all migrate off the ISY. Also, I apologize, the documentation is not up to date with some recent changes. After the Insteon integration is installed you get to the Insteon panel, you navigate to Settings -> Devices & Services -> Insteon -> Configure. Alternatively, you can get there with http://<HA address>:8123/insteon.

One key concept to understand is that there is no true "Auto Discovery" in Insteon. The way HA discovery works it to read the modem's All-Link Database (ALDB) and look for all the devices the modem knows about. So in order for discovery to work, the modem has to be linked to the device (and the device needs to be linked to the modem). To migrate you have two options:

1. Use the modem that is currently attached to your ISY. All of the links are already there. HA will read that modem's ALDB and try to discover the devices that the modem knows about. Note that battery-operated devices need to wake up to be discovered so they may take up to 24 hours to be found unless you trigger them manually.
2. Relink all of your devices to a new modem. This can be done in any of the following ways:
   a. Press the Set button on both the modem and the device
   b. Use the (+) button in the Insteon panel to put the modem into linking mode (same as pushing the Set button of the modem)
   c. Use the (+) button in the Insteon panel and enter the device's Insteon address which should put the device into linking mode too. This will work on many devices, however, not all devices support remote linking.

Discovery in HA does the following things:

1. Asks the device what type of device it is (i.e. cat & subcat)
2. Reads the device's property settings (i.e. Ramp rate, program lock, etc.)
3. Reads the device's All-Link Database (critical in order to know if device 1 is triggered then device 2 changed)

[teharris1]

@Doughboy68 the delete device feature is in the 2024.5.0 release.

@vult2000 not sure I understand your issue so if I get this wrong, let me know. For devices that appear in HA that are not linked to the current (i.e. original modem), you can delete them from the Insteon panel. This will delete them from HA. To delete the device you should see the device in the list of Insteon devices in the Insteon panel. Click on the device which should bring you to the device's properties page. From that page click the ... menu in the upper right corner and select the Delete device option. If you don't see the device in the list you can go to the Utilities tab and select the Delete device option and enter the Insteon address for the device.

[teharris1]

I also made some changes to the remote linking function that should be in 2024.5.4 or 2024.6.0, whichever comes first.

[rcb-hp3000]

Don't know about opening up ~50 outlets to try and link them; plus devices in attics and other hard to reach places. UDI/ISY we never had to link to the modem (though you could) - usually I just manually discovered each device as I installed it by providing the Insteon address of the device and it polled for it and returned the device id.
I DO keep a spreadsheet of all my Insteon devices and their addresses, locations, and what they control - is it even possible to just manually add them one at a time through the UI? Or even a YAML file? The "Insteon Page" only has option to add scripts or add automations.

[Doughboy68]

I also made some changes to the remote linking function that should be in 2024.5.4 or 2024.6.0, whichever comes first.

Is there ability to unlink an Insteon device through the gui? My previous experience was that devices would seem to stay inside the plm. I should give it another try maybe but so far the isy integration is working for me. Ideally I would like everything in home assistant.

[teharris1]

@rcb-hp3000 the process you described for the ISY to discover the device is identical to option 2c above. In the background, the ISY is linking the modem and the device. An Insteon device can only be controlled by the modem if the two are linked.

@Doughboy68 Yes, you can go to the Insteon panel in the settings. If the device shows up in the list of Insteon devices click on the device and under the ... menu in the upper right corner and select the Delete device option. If the device is not in the list (which means the device was never in HA) then you can select the Utilities tab and use the Delete device option. Both options will remove all references to the old device.

[rcb-hp3000]

@teharris1 Thank you. Trying to add devices wasn't working (popped up a quick error then disappeared) until a restart of the PI. Now I can indeed add devices individually.

[teharris1]

@rcb-hp3000 I do not use labels, I rename devices. You can rename a device by going to Settings -> Devices & services -> Devices tab. In the Devices tab you will see a list of all devices in HA, including the Insteon devices. When you click on the device it will allow you to change the name by clicking the pencil in the upper right.

[rcb-hp3000]

Just an FYI on this issue. I am still running my UD ISY 994 in parallel with the new HA setup (keeping up with latest releases) and after deleting the last of the general controller 0x00 devices I have been able to successfully add almost all of the devices still in my ISY configuration. I don't have any automations running yet in HA so the ISY is still controlling all my automations, but the devices are all "talking" to both HA and ISY (each has its own PLM) and I can turn individual devices on/off in HA. I have about 120 Insteon devices currently in both systems - some 10+ years old and a variety of devices from switches (on-off and dimmers), relays, IOLincs, single and double outlets, door open sensors, water/leak sensors, motion sensors, smoke bridge, 8-button mini-remotes, 6 and 8 button Keypad links, indoor lamplincs and outdoor plugin outlets. Almost all of these I've been able to add from the UI by typing in the addresses - though some took a few tries (I have 2 separate 200A service panels plus a generator panel in a 75 year old house so there are connectivity challenges even with a fair number of dual-band devices!). I've had to go to a handful of the devices to put them in link mode manually.

So - from where I sit I think you've gotten the integration working very well - many thanks. I've also noticed in recent months the ability to find/update Insteon devices in the UI has become easier to find and use - so thanks there as well!

UD/ISY still has HA beat badly when it comes to the ease of creating the automations though (no reflection on your integration)... I haven't gotten a single non-trivial automation working in HA yet. I have dozens of "programs" on the ISY I need to figure out how to reproduce in HA.

One thing for the docs; please document how we're supposed to turn on/off a keypadlinc button since these are "stateful" and light up/go out when off?? :-)

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.