[insteon] Improvements for OH3

[jsetton]

This is a rework of the Insteon binding adding some much needed improvements for OH3. Apart from simplifying the user experience by retrieving all the configuration directly from the device when possible, and improving the way the Insteon things are configured in MainUI, here is an exhaustive list of the changes:

• introduced device configuration automated determination
• converted mode-based number items to string type with descriptive states
• added number items uom support
• added scenes and x10 device things
• added distinct bridge things for supported hub/plm devices
• added button event trigger channels
• added device products configuration layer
• added device link database support
• added device cache storage
• added device operating flags controls
• added modem configuration flags controls
• added related devices synchronization feature
• added ability to link a device to the modem
• improved console commands
• improved discovery service
• improved thing status
• rewrote the majority of the binding code
• removed ability to load new device config xml definitions

It is important to note that this is a breaking change. This means that users that are currently using the binding will be required to reconfigure their Insteon setup. However, that process should be much easier with the new improvements.

[norem]

Hi Jeremy lookin great so far ;) you have done an incredible amount of work!

Guess it makes my #12994 request uneeded, lost:comm is working great found and lost devices as expected.

There are multiple last heard from notices in the log for each device .

[robnielsen]

I have communicated privately with @jsetton about this PR. It is essentially a new binding and suggested that he publish it to the community marketplace. The current implementation is stable and has been that way for years.

[norem]

I'm safely testing on a second machine ;) it definitely is an alternative, just switching it on instead of migrating to it as a different entity could potentially unnerve some folks.

[jsetton]

suggested that he publish it to the community marketplace.

I have no problem with doing so while beta testing and making some adjustments based on user feedback.

However, on the long run, it doesn't make sense to have two competing bindings for the same integration. It will confuse further any new users that want to control their Insteon devices with openHAB.

The current implementation is stable and has been that way for years.

Stability shouldn't be a reason for not improving the user experience. The current implementation is certainly functional but its core hasn't evolved to what the latest OH framework has to offer since it was development for OH 1.

[jsetton]

@norem could you share more information about your Insteon environment? (number/type of devices)

It looks like Fast On/Off isn't supported, so saying light On or Off with Alexa no longer works...

What is your use case? Why are you using Fast On/Off opposed to a standard On/Off? Are you trying to control a single device or a scene?

The current fastOnOff channel has been moved between the DOUBLE_PRESSED_ON and DOUBLE_PRESSED_OFF button event triggers, and setting the rampRate channel parameter to instant (0.1 sec) on the dimmer channel.

I believe that the current channels such as fastOnOff or manualChange are mostly used as event triggers to determine when a button was pressed. This is why these are now taking advantage of the OH event trigger feature instead.

dim 0 and dim 100 is a bit unnatural to light on - light off

What do you mean by unnatural?

[norem]

@jsetton

What is your use case? Why are you using Fast On/Off opposed to a standard On/Off? Are you trying to control a single device or a scene?

Standard On/Off ?

could you share more information about your Insteon environment? (number/type of devices)

Windows 10 and 11

Currently using
Hub2

Amazon Devices:
1 Amazon Echo
8 Amazon Dots

Insteon Devices:
1 Smoke Bridge
2 Sirens
3 Fanlincs
7 on/off switches
12 dimmers
1 inline dimmer
2 outdoor switches
3 micro on/off switches
4 dual outlets
2 wireless moisture sensors
10 wireless open/close sensors
3 wireless hidden door sensors

1 Harmony Hub, we try not to use the remote unless we have to ;)

I think I missed a couple...

24 wireless IP cameras ( not connected to openHAB yet )

The dimmer channel alone doesn't allow you turn On or Off a light using Alexa at least not in any way I know of.

I use Fast on/off (Switch) or rather fastOnOff (Switch) channel to turn on or off dimmers using Alexa via The openHAB Alexa Skill, or in rules to just turn these devices on or off instead of saying "Alexa Dim Bedroom light 0" vs "Alexa, light off" (while speaking to bedroom Dot)

`actions:

• inputs: {}
  id: "14"
  configuration:
  command: OFF
  itemName: LivingRoomFloodsWEST_FastOnOff
  type: core.ItemCommandAction
• inputs: {}
  id: "13"
  configuration:
  command: OFF
  itemName: LivingRoomFanLight_FastOnOff
  type: core.ItemCommandAction`

What do you mean by unnatural?

What I meant by unnatural is... assume every room in our house including bathrooms have an Amazon Dot in them.
( because they do )

In the bedroom we might say "Alexa Light Off"

In the Livingroom I might say "Alexa, Floods 20% or "Alexa, Floods Off"
not "Alexa, Floods 0" having to use the dimmer (Dimmer) Channel only...

or outside "Alexa, turn on pond vacuum"

The wife and I rarely touch the Wall Switches we use Alexa 95% of the time unless standing next to the switch or if the Internet is down and if so, just sit quietly in the dark hoping the cat will press the switch on... ;)

Tim/norem

[norem]

@jsetton I do like that most if not all of my devices information was automatically made available and correct, I like having this to easily see what type or version of dimmer might be misbehaving without getting out a screw driver because I forgot.

Personally I used my existing Insteon things and just pointed them to Hub2 instead of Insteon Network of course there's no going back at that point and deleted the correctly discovered devices.

An existing user that may be like me and backs up and downloads every 2 days ( living on the edge baby! ) would be in for a scary surprise ;)

I am using a separate machine with back up copies of my set-up in multiple places, because as @robnielsen has mentioned the current set-up works and we are scared of the dark ;)

Loss of Comm is important to me in troubleshooting wireless devices and switches that may have inadvertently been left with its set switch pulled out or device gone bad or "Open/Closed Sensor" battery died.

Being able to create groups definitely is helpful to eliminate popcorn effect and cleaner rules.

Getting away from the Insteon app that hasn't been changed for 10 years is a must.

[jsetton]

@norem

Standard On/Off ?

Basically turn on/off a device using local on level and ramp rate settings. Fast On/Off is bypassing these local settings turning on at 100%/off a dimmer instantly (0.1 sec ramp rate).

From the physical device interaction standpoint, the standard on/off is tapping once on the dimmer paddle while fast on/off is accomplished by double tapping.

The dimmer channel alone doesn't allow you turn On or Off a light using Alexa at least not in any way I know of.

That sounds like an Alexa configuration issue. Would you care to share the metadata you configured? The dimmer channel certainly supports on/off commands as well as percent commands. You can model a single dimmer item to control power state and brightness on the Alexa side. It seems based on your explanation, you have separated these two abilities.

Insteon Devices:
1 Smoke Bridge
2 Sirens
3 Fanlincs
7 on/off switches
12 dimmers
1 inline dimmer
2 outdoor switches
3 micro on/off switches
4 dual outlets
2 wireless moisture sensors
10 wireless open/close sensors
3 wireless hidden door sensors

That's quite a good mix of devices. Let me know if you are having any issues with your other devices such as the smoke bridge and dual outlets. I might circle back to you about the siren integration. I considered adding support. I assume these are model 2868-222?

[jsetton]

I am using a separate machine with back up copies of my set-up in multiple places, because as @robnielsen has mentioned the current set-up works and we are scared of the dark ;)

I totally understand that part and I would certainly not encourage users to use these changes until a thorough beta testing is made. I expect to have to make some adjustment as well.

[norem]

@jsetton not sure of an easy way to share this, I use the openHAB GUI not paper UI I'm using windows for my set-up ;)

yep that's the siren

a poor example of my Model Pulldown... points that are enabled.

Hallway Light
Lightbulb
Dimmer Metadata Alexa LightBrightness
Point
Fast On/Off Metadata Alexa LightPowerState
Point
Last Heard From
Point

LightPowerState is greyed out in Dimmer Alexa Device Type/Attributes choices and not selectable as a second attribute.

just using dimmer didn't work with Alexa on/off commands

Sooooo I thought I had to use both and it worked. note: I did this in 2 days and the instructions not so clear to me on the Metadata settings.

I have a feeling now there is a better/correct way heh heh

[jsetton]

not sure of an easy way to share this

You can paste the content of the Alexa metadata code tab.

LightPowerState is greyed out in Dimmer Alexa Device Type/Attributes choices and not selectable as a second attribute.

It sounds like you have this setup in a group endpoint. You can only have one PowerState attribute per group. That's why it's greyed out. If you need to select multiple, there is a checkbox on the top right end side to do so.

Anyway, I would recommend switching to a single endpoint using the Dimmer item only and model it as Light metadata device type. To do so, make sure to remove your existing configuration from the relevant items.

[norem]

@jsetton crap that works, I think the greyed out thing thew me for a loop and I just carried suit heh heh

thanx man I have some/many changes to make ;) then go back to testing :)

Tim

[norem]

@jsetton thanx again "WE" could use some videos of how this is done "correctly using the pull down menus" seems most directions use text entry which differs greatly from paperUI

[jsetton]

that works

So in the end, you won't be using the fastOnOff channel 😄

could use some videos of how this is done "correctly using the pull down menus" seems most directions use text entry which differs greatly from paperUI

Good point. I have opened openhab/openhab-alexa#520. It might not be videos but we can add some screenshots as examples.

[norem]

@jsetton

That took a bit but not bad... now to copy it to the other machine.

So in the end, you won't be using the fastOnOff channel 😄

Correct I, I did some editing 😄

[norem]

@jsetton here's a present for ya apparently the SB type switches are read but Waiting for link database.
you of course know of the sirens ;) t

This data is all from openHAB 2475D, 2476D, 2477D, 2477S Are registering and able to be controlled with Model Slider.
( Controllable with Alexa with corrected Metadata and all fastOnOff removed )

The following Are registering Waiting for link database and NOT able to be controlled.

`Status: UNKNOWN
CONFIGURATION_PENDING
Waiting for link database.

deviceType SwitchedLightingControl_SwitchLinc
engineVersion I2
serialNumber 1B4570
productId 0x02 0x16
modelId 2876SB
vendor Insteon
firmwareVersion 0x39

deviceType SwitchedLightingControl_SwitchLinc
engineVersion I2
serialNumber 1BF762
productId 0x02 0x16
modelId 2876SB
vendor Insteon
firmwareVersion 0x39

DeviceType SwitchedLightingControl_SwitchLinc
engineVersion I2
serialNumber 1B467A
productId 0x02 0x16
modelId 2876SB
vendor Insteon
firmwareVersion 0x39

deviceType SwitchedLightingControl_SwitchLinc
engineVersion I2
serialNumber 1B4062
productId 0x02 0x16
modelId 2876SB
vendor Insteon
firmwareVersion 0x39

deviceType SwitchedLightingControl_SwitchLinc
engineVersion I2
serialNumber 1B3EB3
productId 0x02 0x16
modelId 2876SB
vendor Insteon
firmwareVersion 0x39

deviceType SwitchedLightingControl_ApplianceLinc
engineVersion I2
serialNumber 18ED52
productId 0x02 0x17
modelId 2856S3B
vendor Insteon
firmwareVersion 0x39

Status: OFFLINE
CONFIGURATION_ERROR
Unsupported device. These are sirens

serialNumber 3F0BC2
productId 0x07 0x1E
modelId 2868-222
vendor Insteon
hardwareVersion 0x45
firmwareVersion 0x45

serialNumber 3F0AF8
productId 0x07 0x1E
modelId 2868-222
vendor Insteon
hardwareVersion 0x45
firmwareVersion 0x45
`

[jsetton]

Nice you got some older ICON Relay devices. I would need you to start logging event messages for one of these devices for at least 5 minutes (default device polling interval).

To do so, you will need to run the below command in the openHAB console. Since you are running a Docker container, you will need to make sure you can retrieve the monitor file from the userdata/insteon directory.

openhab> insteon startMonitoring 1B.45.70
Started monitoring the device 1B.45.70.
Message events logged in /openhab/userdata/insteon/messageEvents-1B4570.log

The outgoing message for the get all link database request should look like:

OUT:Cmd:0x62|toAddress:1B.45.70|messageFlags:0x1F=DIRECT:3:3|command1:0x2F|command2:0x00|...

Once done, you should stop the monitoring, zip up the file and attach it to this issue if possible.

openhab> insteon stopMonitoring 1B.45.70
Stopped monitoring the device 1B.45.70.

Related to the offline status for your siren devices, this is expected behavior. Are you able to control these devices with the current binding version?

[norem]

@jsetton

2856S3B plug in on/off works as F00.00.02
2876SB switch on/off works as F00.00.02

[jsetton]

2856S3B plug in on/off works as F00.00.02
2876SB switch on/off works as F00.00.02

Thanks although I was only curious about your siren devices. I assume you can't control them currently?

The issue with your ICON Relay devices is that the binding doesn't seem to be able to download their all link databases. I have a suspicious of what might be the root cause but I want to see how these devices are responding to the current download request.

[norem]

@jsetton
messageEvents-1B4570.zip

[jsetton]

Thanks for sharing the logs. I found the issue and I am working to fix it. The database is being downloaded but the verification process on the binding side is failing for ICON Relay devices.

Interestingly, it seems that the top memory address for the all-link database of these devices is 0xFF while for most Insteon devices it is 0xFFF. It looks like these devices are only using one byte compared to two in most cases.

[norem]

No problem, I have plenty time for this stuff ;)

Smarthome didn't read the dev notes heh heh ?

[jsetton]

Smarthome didn't read the dev notes heh heh ?

Probably because these are older devices and seem to be the cheapest Insteon product line at the time from the information I found.

[norem]

@jsetton please let me know if you need anything in particular to help...

Interesting... if you follow the instructions in the comment window below to drag n drop or paste, it doesn't work but if you select the attach files box, you can select files to attach. grrrr

messageEvents-1B4570.zip

[norem]

Wasn't sure if just pressing the button would keep it awake long enuff...

openhab> insteon listDeviceDatabase 3F.C4.EC
The all-link database for device 3F.C4.EC is empty

[norem]

nope not a new device, it's one of 2 I tried to refresh the other devices ALDB is still intact.

[norem]

could it get stuck if the thing properties is filled but the ALDB cache is empty ?

[jsetton]

Yes, the device product data is retrieved before the initial poll.

On the first poll, the Insteon Engine is retrieved if not already defined, and then the link database is downloaded if not complete.

[norem]

Not sure how I managed to break it other than an attempt to refresh after cleaning it's ALDB up. The other Leak detector didn't get messed up after doing several refresh attempts... I'm guessing it cant be corrected until unless it has its properties dumped.

[jsetton]

What do you mean by "an attempt to refresh after cleaning it's ALDB up"? Can you run a monitored logs on that device? You should see 0x2F request being sent to the device.

[norem]

What do you mean by "an attempt to refresh after cleaning it's ALDB up"?

It's ALDB in OH didn't agree with what was viewed in Insteon-Terminal so I did an insteon refreshDevice 3F.C4.EC correct it, then it got stuck in config pending.

Can you run a monitored logs on that device? You should see 0x2F request being sent to the device.

being monitored now ...

[norem]

No 0x2F requests being sent to the device.

messageEvents-3FC4EC.zip

[norem]

fromAddress:3F.C4.EC|toAddress:11.1D.02 < thats not even a device I own ... ;(

[jsetton]

No 0x2F requests being sent to the device.

There are some but it looks like the device is asleep by the time it is sent based on 0x5C message replies. It should still be awake.

fromAddress:3F.C4.EC|toAddress:11.1D.02

That's the normal all link broadcast cleanup report.

[norem]

Gotcha on the cleanup report and see the 2 0x0F didn't look close enuff

[norem]

@jsetton this is the monitoring after setting and resetting the contacts randomly... still in config pending
messageEvents-3FC4EC.zip

[norem]

@jsetton I now have another device, for no reason it's a micro switch.

Status: UNKNOWN
CONFIGURATION_PENDING
Waiting for link database.in config pending

updated: and now the micro switch is back online

[norem]

@jsetton ok I cleaned up 3F.C4.EC with Insteon-Terminal, pointing Group 1 and 4 to my HUB again ... it was a mess, there were at least 12 entries! and couldn't get it to download its full ALDB so had to reset the device and set it up again. With its NOW 2 entries again, it had no problem reading the ALDB

Pressing and holding the set button until flashing is recommended by Insteon-Terminal, but pressing it all is probably a bad idea, so this time... I simply shorted the contacts on the Moisture Sensor several times. after doing insteon refreshDevice 3F.C4.EC it went back ONLINE and now insteon listDeviceDatabase 3F.C4.EC now shows the correct information.

btw I had changed to just shorting out the contacts before all the clean up and was still in CONFIG PENDING I have to assume that the binding wasn't able to complete and go ONLINE because it couldn't get the complete ALDB like Insteon-Terminal was experiencing.

Sorry about the inconvenience, Everything is back ONLINE, that was amusing ;) I don't think this was a binding problem as much as it is a documentation issue. I'm thinking if I knew the sensors ALDB had become a mess again I would have cleaned it up in the beginning.

dunno if this is possible ?

Status: UNKNOWN
CONFIGURATION_PENDING
Could not get complete list for link database.in config pending

[norem]

@jsetton I think having beat the 24 Hour Timer thing to a pulpy mass, 24 Hours is correct and think the over 24 timings were an anomaly, failing the heartbeat might just be giving a hint to an issue with the device, be that range or filled with junk.

[jsetton]

do either of you guys know what Group 0xFF is used for?

Some sensors allows linking group 0xFF to receive all group events instead of linking each group.

[norem]

Thanx, oddly some of my sensors had it set and others not ;)

[jsetton]

I think this is due to the lack of consistency when setting up a sensor device with Insteon App. My open/close sensors are linked through the group 0xFF while my hidden door sensor through the group 0x01 only. There is an operation flag to enable the all group 0xFF feature as well on these devices.

[norem]

@jsetton

I think this is due to the lack of consistency when setting up a sensor device with Insteon App.

yeah I think even with its quirks I'll try to stick with using I-T and the button on the Hub :)

Got anything ready to test unplugging and plugging in my pond vacuum etc ? 😄

[norem]

@jsetton Just a thought for documentation "When removing or changing a battery on a wireless device, be sure to Disable then Enable the device, using the pause button in it's Things Menu, this will ensure it's heartbeat won't be improperly logged" or something like.... I obviously get kinda wordy... I'm assuming the Pause Button does reset the heartbeat timer for a device.

[norem]

@jsetton Jeremy are you seeing an issue with the heartbeat updating to open when the device is clearly closed ?

I was so busy making sure heartbeats were happening to note the open/close status and have seen this happening on several sensors.

A google search pulled up this past issue... and looks exactly like I'm seeing. Nothin occurs between heartbeats with the sensor starting as closed.

https://forums.homeseer.com/forum/lighting-primary-technology-plug-ins/lighting-primary-technology-discussion/insteon-mnsandler/94389-open-close-sensor-triggering-events-on-heartbeat

UPDATED: https://forum.universal-devices.com/topic/16175-openclose-sensor-bug/

I've checked the contacts electrically they are closed but updated with heartbeat as open...

[norem]

@jsetton From what I'm starting to see, the heartbeat open/close status may have changed between versions, and may not be reliable... The heartbeat itself is not in question...

UPDATE: unfortunately All of my open/close sensors say they are version 0x43 even tho their stickers vary from v 1.0, 2.1 and 3.0

[norem]

@jsetton Commenting out all case 4 heartbeat contact updates in messageHandler.java seems to have alleviated the problem not that's the correct way to handle it ;)

UPDATED: This corrected my sensors false opening when heartbeat triggered,
( according to ALL docs I've seen should have worked great ! ) Insteons dev notes sometimes amaze me... btw if you look at open/close and leak detector docs side by side almost a carbon copy ;o

I'm not certain that doing a software rev check via ALDB can be used to enable/disable the heartbeat contact status update, my O/C sensors all say 0x43 but range rev 1.0 - 3.0 on the label... so sad.

---

I'm also trying a 48 hour time out, with the thought that 1 miss in 24 hours can happen and obviously not fatal but 2 missed in 48 hours is not likely unless the device is no longer transmitting.

UPDATED: This feels right and works, already made me aware of a battery that lost contact. I was getting what felt like random failures, but 2 days with no heartbeat, definitely made me verify the state of the sensor. I think this being adjustable has a benefit.

[lsiepel]

@jsetton are you able to fix the conflicts and proceed with this PR?

[robnielsen]

@jsetton are you able to fix the conflicts and proceed with this PR?

This PR is essentially a new Insteon binding that is not backward compatible with the existing one.

The current binding is stable, with many users.

With that said, this version of the binding should remain as an alternate binding from the Marketplace, and this PR should be closed.

[norem]

Rob,

I'm just curious, have you tried the new/other version, I'm not disagreeing with you it is a breaking change but having used both and for a new user, and with Insteon trying very hard to come back it is truly a benefit.

Please don't get me wrong I'm just curious about your opinion. :)

[robnielsen]

No, I didn't try this binding and don't plan on it.

IMHO, Insteon is a dead/dying platform. There will be very few new users of the Insteon binding. Why would we want to push breaking changes on existing users?

BTW, I would welcome improvements to the existing binding that are not breaking changes.

[tcgerhard]

@robnielsen -- new user of the binding here; moving my install from ISY to OH. I'd like to suggest if there's reluctance to push the breaking changes that it be considered a new binding (maybe something like "Insteon GUI Version" or "Insteon Gen 2"?). This binding and its documentation should be more easily discoverable for potential new users.

[tcgerhard]

@jsetton - I'm running wrap_file__var_lib_openhab_tmp_kar_org.openhab.binding.insteon-3.4.0-SNAPSHOT_org_lastnpe_eea_eea-all_2.2.1_eea-all-2.2.1.jar on OH 3.3 (upgrade to 4 is pending). A few items to note:

• I see the same issue noted earlier of multiple "last seen" updates. I'm seeing this on a battery device (deviceType
  SecurityHealthSafety_MotionSensor2
  engineVersion: I1; serialNumber: 4A705B; productId: 0x10 0x16; modelId: 2844-222; vendor: Insteon
  firmwareVersion: 0x46
• For the same device, it's perennially in "Configuation Pending" state; I did note the following in events.log:
  2023-11-04 10:22:20.479 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'insteon:device:67723519da:4A705B' changed from UNKNOWN (CONFIGURATION_PENDING): Waiting for link database. to ONLINE 2023-11-04 10:26:39.588 [INFO ] [ab.event.ThingStatusInfoChangedEvent] - Thing 'insteon:device:67723519da:4A705B' changed from ONLINE to UNKNOWN (CONFIGURATION_PENDING): Waiting for link database.
  It should be noted that the device works as expected otherwise (i.e., I get events when motion is triggered)
• Occasionally, I see randomly long wait times between a command being issued and the device responding (anywhere from a few seconds to many minutes).

How can I help sort these out?

[jsetton]

IMHO, Insteon is a dead/dying platform. There will be very few new users of the Insteon binding. Why would we want to push breaking changes on existing users?

@robnielsen Insteon is still around as far as I know and selling products. Many users are looking for an alternative to the subscription model offered by Insteon. You seem to imply that because you personally don't want to deal with reconfiguring your setup, every users of the binding would be content with that as well. These breaking changes if they end up being released, would only affect users that decide to upgrade to a new OH version which usually involves handling other changes.

Talking about dying platform, I will agree with you on that one but we should be talking about OH first and not Insteon. Because the whole point of adding new enhancements and features is to attract new users.

BTW, I would welcome improvements to the existing binding that are not breaking changes.

The binding in its current state is basically an OH 1.x binding that you ported to OH 2.x back in 2020, because it was no longer going to work with OH 3.0, without making much changes to the existing code. As I mentioned back then, there is no way to make improvements to this binding without rewriting a good portion of the code. As a matter of fact, the last enhancements, besides your port, was merged back in 2017.

I'd like to suggest if there's reluctance to push the breaking changes that it be considered a new binding (maybe something like "Insteon GUI Version" or "Insteon Gen 2"?).

@tcgerhard Sorry for my late reply. This exactly what I was trying to avoid and confuse any new users. Also, having to explain the situation to them on the forum when suggesting to use the new version over the official one is certainly not going to help the cause.

Anyway, I finally got around and released a new version for OH 4.1 and above with many new improvements including link database and scene management via the OH console. It is available on the Marketplace. Feel free to test it out and let me know if you experienced the same issue that you reported above.