How to beta test prior to the Home Assistant Pull Request

[creeble]

Hi David. Imagine my luck: I just got my Insteon PLM a few days ago, began researching Python control for it, and BOOM -- up comes your repo!

My inspiration is twofold:

1. Get rid of my stupid ISY99 controller, which demands that I upgrade Java on my laptop every time I want to do something with it, and
2. Learn Python more, especially async i/o.

I'm currently waiting for the delivery of another RemoteLinc (2342-222) with which to experiment -- I can't remove any of my current controllers from service (i.e., linked to existing PLM) or I endure the wrath of the other user of our Insteon control system, iykwim. So at the moment I'm unable to generate any events to be received by the PLM, but that will change in a couple of days.

It looks like the next steps for this library might be something like making a (public) send_message() in protocol.py that understands the various Insteon message types? Curious about your thoughts on this.

Ultimately, my needs aren't very fancy: I have a system that uses Insteon almost exclusively for lighting, and all the lights are controlled via (sunset-offset, mostly) timers, with a few wired and wireless controllers around for all-units-off commands (bedside) and to control some small groups (which doesn't require any PLM interaction, really).

I got all of your code running on a CHIP (https://getchip.com/pages/chip), although I may end up with a Raspi because I think I have wired Ethernet where I need it.

Looking forward to seeing where this goes, and how I might be able to help!

[nugget]

Thanks for the comment. I'm eager to get this module up to a point where I can integrate it with my Home Assistant platform (running on a Raspi).

One thing to consider is that there are actually three protocols at play here which will need to be embodied in the device. There's the INSTEON Standard with 9 byte packages, and the INSTEON Extended with 23 byte packets. Those are both encapsulated inside the PLM's protocol, which is what I'm currently focused on supporting with the module.

The PLM protocol is sort of wonky. Commands are prefixed with a 0x02 character, but since messages can legitimately contain a 0x02 you can't simply use it as a straight delimiter. Instead, the protocol handler will need to exhaustively understand the protocol in order to successfully parse out the receive stream into unique messages. I'm unsettled on the best way to store the constants and command attributes in the code to ensure that we can easily work with them and they contain all the relevant information. My immediate plan is to bang around with the commands "by hand" until I have more confidence that I understand it enough to map it out sanely in the code.

From there we get to tackle the actual INSTEON protocols that receive status and send requests to the actual devices.

[creeble]

Yes, I think there is an inherent problem in the PLM's protocol as result of reusing 0x02 in messages; I read this (documentation for a Perl module):

http://misterhouse.sourceforge.net/lib/Insteon/MessageDecoder.html

and interpreted it as that problem. Maybe if you update the state of all messages, verifying ACK/NAK for everything?

Because of my limited needs, I don't really care about things like Extended messages or even monitoring/inquiring about state. Being able to link devices is probably necessary just so that the all-devices-off command works, but that's only necessary to avoid running around the house once or twice.

Anyway, thanks for starting this; I hope I can contribute something useful in time.

[nugget]

Today's changes embody the PLM protocol in an abstracted class that I'm fairly content with. It should support the few weird variable-length message exceptions I'm seeing in the developer docs for the PLM device.

I did a quick and dirty wrapper for message x50 (standard INSTEON) just to sanity check the parsing and it looks good. This will need to be restructured, but it's serviceable for now. I haven't even started thinking aobut the device pairing process, although that shouldn't be too tough.

INFO:insteonplm.protocol:INSTEON message from 40.95.E6 to 39.55.37: cmd1:0x11 cmd2:0x1 flags:0x40
INFO:insteonplm.protocol:INSTEON message from 40.95.E6 to 11.01.01: cmd1:0x6 cmd2:0x0 flags:0xcf
INFO:insteonplm.protocol:INSTEON message from 39.5E.CB to 00.00.01: cmd1:0x11 cmd2:0x0 flags:0xcf
INFO:insteonplm.protocol:INSTEON message from 39.5E.CB to 39.55.37: cmd1:0x11 cmd2:0x1 flags:0x45
INFO:insteonplm.protocol:INSTEON message from 39.5E.CB to 11.01.01: cmd1:0x6 cmd2:0x0 flags:0xcf
INFO:insteonplm.protocol:INSTEON message from 39.5E.CB to 00.00.01: cmd1:0x13 cmd2:0x0 flags:0xcf
INFO:insteonplm.protocol:INSTEON message from 39.5E.CB to 39.55.37: cmd1:0x13 cmd2:0x1 flags:0x45
INFO:insteonplm.protocol:INSTEON message from 39.5E.CB to 13.01.01: cmd1:0x6 cmd2:0x0 flags:0xcf```

[creeble]

Terrific, I just got the new controller today and hope to be able to work on it tomorrow / this weekend. I'll let you know what I find, if anything.

[rstanley75]

I was working with my Insteon PLM and the developer docs a few weeks ago and was having the same issue with the Insteon message types. It's frustrating that you need to read the message to know how to read the message! The message flags can mean so many different things as well. I'm a beginner so this is probably ugly, but this was my solution (borrowing a bit from some other projects, of course) to parsing the first half of the message flags:

`

    flagBroadcast = messageFlags & (1 << 7) == (1 << 7)
    flagALLLink = messageFlags & (1 << 6) == (1 << 6)
    flagAcknowledgement = messageFlags & (1 << 5) == (1 << 5)
    flagExtended = messageFlags & (1 << 4) == (1 << 4)

    isAck = flagAcknowledgement and (not flagBroadcast)
    isNak = flagBroadcast and flagAcknowledgement

    isBroadcast = flagBroadcast and (not flagAcknowledgement)
    isDirect = (not flagBroadcast) and (not flagALLLink)
    isCleanup = ((not flagBroadcast) and flagALLLink and (not flagBroadcast)) or (flagBroadcast and flagALLLink and flagAcknowledgement)

`

At the very least it would have been nice of them to pass the checksum along so you could calculate it on the fly and (hopefully) know that you had a complete message instead of some garbage.

I like your project and I absolutely want to see this happen. I get that Insteon is old tech at this point but I really like their plug-in dimmer modules (A button for bright/on, a button for dim/off, local sensing - the things we take for granted.). I also have some Insteon hidden door sensors installed for my deadbolts to activate when latched. (I have to confess: I may have a few in-wall X10 modules lingering in my home. Something, something, no neutral wires.)

I'm really glad you're doing this and I'll be happy to help contribute in whatever small way I can.

[nugget]

If anyone is interested in testing, this process should get you up and running:

https://github.com/nugget/python-insteonplm/blob/master/HASS.md

If you do get it set up I'd love to know how it goes.

[creeble]

This looks great; I'm going to try and get it up and running sometime between now and this weekend. Eric.

On 2017-02-16 14:15, David McNett wrote: If anyone is interested in testing, this process should get you up and running: https://github.com/nugget/python-insteonplm/blob/master/HASS.md If you do get it set up I'd love to know how it goes. -- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links: ------ [1] #1 (comment) [2] https://github.com/notifications/unsubscribe-auth/AFxCbwmXLZspFP8CLEqshSoE4MMQrJwQks5rdMplgaJpZM4LaKW7

[rstanley75]

Alright, after installing the package in custom components and adjusting my config files this is what I've experienced so far:

• All of my 2457D2 dimmers are detected and can be switched on and off from HASS. When I operate the dimmers manually - either through the buttons on the module or the switch on the lamp itself - the status changes in the HASS interface. (No dimming yet, obviously. But I can see the messages scroll through in the console when I dim them manually at the module).

• I have an older Insteon in-wall switch (I forget what model, but I still have the documentation buried somewhere) that is detected and can be turned on, but the switch icon in HASS flips back to off immediately after so I can't switch it off once it's on.

• My 2845-222 hidden door sensor isn't detected at all.

• There is a mystery device: light.accf23c5814a. It doesn't appear straight away but a bit later. I don't see any relation to this and the modules I have paired. I have a hunch that this may disappear if I do a factory reset on the PLM (I returned some modules a while back, but they're probably still in the PLM's internal DB). What I find interesting is that it appears to be two Insteon addresses put together.

I haven't delved into the logging yet so I don't have any actual logs to provide for you yet.

That said I think you have gone above and beyond in your implementation. I was not expecting it to get the database of Insteon devices from the PLM, something that feels like magic to me right now. In fact, after looking at your code everything seems like magic to me and I'm stoked to see what you come up with next.

[nugget]

Excellent, I'm glad to hear you're up and running (to a point) with the code.

I got dynamic platform loading working, so it's no longer necessary to explicitly load the insteon_plm for each platform (light, switch, etc...). You just need to load the main insteon_plm component at the root level of your configuration.yaml now. I revised the beta docs in HASS.md to reflect that.

As luck would have it, I happen to have a few of the door sensor devices (2845-222) laying around the house that I never got around to installing. I was able to expand the IPDB in the codebase and add support for device category 0x10 generically. I've got one working with my hass install now.

I don't have any theory at all about the weird double-addressed light device. That's truly bizarre.

I'm curious about dimming for your 2457D2 devices. I have no way to test directly, but that should be working. The devices are advertising dimmability to hass and you should be able to both adjust the dim level from the hass frontend UI as well as see dimming reflected when you manually control the device. It's definitely working fine for my 2477D wall switches. Perhaps the 2457D2s behave differently somehow. I'll dig through the spec and see if I can find anything suspicious.

New insteonplm is pushed out to pypi and I updated the custom_component tarball just now.

[rstanley75]

I removed everything in custom_components and then re-downloaded and untarred the contents of insteon_plm_beta.tar.gz into it. I can see the devices being discovered when HASS starts up but I think I'm missing a piece of the puzzle because this shows up in my log now:

`17-02-17

16:23:21 ERROR (MainThread) [homeassistant.loader] Error loading custom_components.binary_sensor.insteon_plm. Make sure all dependencies are installed
Traceback (most recent call last):
File "/opt/hass/lib/python3.4/site-packages/homeassistant/loader.py", line 139, in get_component
module = importlib.import_module(path)
File "/opt/hass/lib/python3.4/importlib/init.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 2254, in _gcd_import
File "", line 2237, in _find_and_load
File "", line 2226, in _find_and_load_unlocked
File "", line 1200, in _load_unlocked
File "", line 1129, in _exec
File "", line 1471, in exec_module
File "", line 321, in _call_with_frames_removed
File "/home/hass/.homeassistant/custom_components/binary_sensor/insteon_plm.py", line 11, in
from homeassistant.components import insteon_plm
ImportError: cannot import name 'insteon_plm'
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.loader] Unable to find component binary_sensor.insteon_plm
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.bootstrap] Unable to find platform binary_sensor.insteon_plm
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.loader] Error loading custom_components.light.insteon_plm. Make sure all dependencies are installed
Traceback (most recent call last):
File "/opt/hass/lib/python3.4/site-packages/homeassistant/loader.py", line 139, in get_component
module = importlib.import_module(path)
File "/opt/hass/lib/python3.4/importlib/init.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 2254, in _gcd_import
File "", line 2237, in _find_and_load
File "", line 2226, in _find_and_load_unlocked
File "", line 1200, in _load_unlocked
File "", line 1129, in _exec
File "", line 1471, in exec_module
File "", line 321, in _call_with_frames_removed
File "/home/hass/.homeassistant/custom_components/light/insteon_plm.py", line 11, in
from homeassistant.components import insteon_plm
ImportError: cannot import name 'insteon_plm'
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.loader] Unable to find component light.insteon_plm
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.bootstrap] Unable to find platform light.insteon_plm
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.loader] Error loading custom_components.switch.insteon_plm. Make sure all dependencies are installed
Traceback (most recent call last):
File "/opt/hass/lib/python3.4/site-packages/homeassistant/loader.py", line 139, in get_component
module = importlib.import_module(path)
File "/opt/hass/lib/python3.4/importlib/init.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 2254, in _gcd_import
File "", line 2237, in _find_and_load
File "", line 2226, in _find_and_load_unlocked
File "", line 1200, in _load_unlocked
File "", line 1129, in _exec
File "", line 1471, in exec_module
File "", line 321, in _call_with_frames_removed
File "/home/hass/.homeassistant/custom_components/switch/insteon_plm.py", line 14, in
from homeassistant.components import insteon_plm
ImportError: cannot import name 'insteon_plm'
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.loader] Unable to find component switch.insteon_plm
17-02-17 16:23:21 ERROR (MainThread) [homeassistant.bootstrap] Unable to find platform switch.insteon_plm`

I've removed /deps and home-assistant_v2.db then did pip3 install --upgrade homeassistant but to no avail, I get the same error when starting hass. I also double checked my file permissions but those appear fine.

[nugget]

Ahh, I see the issue... from homeassistant.components import insteon_plm fails if the code is in custom_components instead of the real components directory. I'll try to figure out a workaround.

[nugget]

OK, I figured out a workaround. Need to change all the lines in the platform files that say from homeassistant.components import insteon_plm to instead say from .. import insteon_plm

I rolled a new tarball but you may find it easier to just edit the three files locally.

[rstanley75]

Aha! That makes sense now, since you mentioned earlier that you were intending this for inclusion as a regular HASS component. I made the changes and that solved almost everything.

1. I can dim/bright the dimmer modules now too! (It's possible it may have been an error on my part from some duplicate names I had forgotten to comment out in a different yaml.)

2. My binary sensor is detected now and functions just how I would expect it to.

3. The only thing I'm still having trouble with is that dang light switch. I can turn it on from HASS but the toggle in HASS flips right back into the off position leaving me unable to switch the light off again. However, I did manage to locate the documentation! It is a really old ToggleLinc V2 Relay (#2466SW).
   

This is what the log had to say about it when I switched it from off to on from HASS:
INFO:homeassistant.core:Bus:Handling <Event call_service[L]: service_call_id=140583722986800-2, service=turn_on, domain=homeassistant, service_data=entity_id=switch.146355> INFO:homeassistant.core:Bus:Handling <Event call_service[L]: service_call_id=140583722986800-3, service=turn_on, domain=switch, service_data=entity_id=['switch.146355']> INFO:homeassistant.core:Bus:Handling <Event service_executed[L]: service_call_id=140583722986800-3> INFO:homeassistant.core:Bus:Handling <Event service_executed[L]: service_call_id=140583722986800-2> INFO:insteonplm.protocol:Processing message: b'025014635534e62a201101' INFO:insteonplm.protocol:INSTEON standard 14.63.55->34.E6.2A: cmd1:11 cmd2:01 flags:20 INFO:insteonplm.protocol:INSTEON on event: cmd2: 1, log: <logging.Logger object at 0x7fdc184b1780>, address: 14.63.55, target: 34.E6.2A, flagsval: 32, cmd1: 17, userdata: bytearray(b''), rawmessage: bytearray(b'\x02P\x14cU4\xe6* \x11\x01'), code: 80, flags: {'extended': False, 'group': False, 'hops': 0, 'broadcast': False, 'maxhops': 0, 'ack': True}, {'capabilities': ['switch'], 'address_hex': '146355', 'address': '14.63.55', 'product_key': None, 'description': 'Unknown Device', 'model': None, 'firmware': 56, 'cat': 2, 'subcat': 13, 'onlevel': 0} INFO:insteonplm.protocol:Device 146355.onlevel changed: 0->1" INFO:custom_components.switch.insteon_plm:Received update calback from PLM for 146355

This is the log when I switched it on from HASS when the switch was manually turned on before starting the HASS process:
INFO:insteonplm.protocol:Processing message: b'025014635534e62a201101' INFO:insteonplm.protocol:INSTEON standard 14.63.55->34.E6.2A: cmd1:11 cmd2:01 flags:20 INFO:insteonplm.protocol:INSTEON on event: flagsval: 32, cmd1: 17, code: 80, address: 14.63.55, userdata: bytearray(b''), target: 34.E6.2A, log: <logging.Logger object at 0x7f4e3a9758d0>, cmd2: 1, rawmessage: bytearray(b'\x02P\x14cU4\xe6* \x11\x01'), flags: {'maxhops': 0, 'extended': False, 'broadcast': False, 'group': False, 'ack': True, 'hops': 0}, {'onlevel': 255, 'product_key': None, 'capabilities': ['switch'], 'cat': 2, 'subcat': 13, 'address': '14.63.55', 'model': None, 'description': 'Unknown Device', 'address_hex': '146355', 'firmware': 56} INFO:insteonplm.protocol:Device 146355.onlevel changed: 255->1" INFO:custom_components.switch.insteon_plm:Received update calback from PLM for 146355

It goes from 255 -> 1 when the switch is already on. I don't know if that's relevant but it's something I noticed.

4. I still have my mystery light.accf23c5814a but it doesn't seem to be affecting anything and it's not really in my way. I can hide it for now until I can be bothered to do some more troubleshooting. I think the next step is to do a factory reset on the PLM but the thought of going through that sounds awful right now. Maybe I'll try to tackle that next week.

Edit: Looks like 2466 is the product number for the ToggleLinc series. S means relay while D would mean dimmer. The W and I are just the color of the switch (White/Ivory)

[rstanley75]

That was easier than I expected. I inserted this line after line 48 in ipdb.py and now the ToggleLinc works correctly.

Product(0x02, 0x0d, None, 'ToggleLinc Relay', '2466S', ['light']),

[nugget]

Well that's perfect, excellent.

[rstanley75]

This is embarrassing. You know that 2845-222 Hidden Door Sensor I said was working as expected? Yeah, I was looking at a sensor template. I'm not getting anything from it at all. When I pair it I can see the ALL-Link command for it in the log, but after that when I press or release the button I get nothing in the log.

I found a document with developer notes for the Hidden Door Sensor, maybe it will help?
2845-222dev-102013-en.pdf

[nugget]

I found that with the 2845-222 I had to do the ALL-Link pairing in both directions before the device started sending updates to my PLM. I paired it the first time by press-holding the set button on the PLM and then press-holding the set button on the sensor. That added it to the ALDB on the PLM.

But then I was seeing what you're seeing -- no messages at all on the PLM. I did the pairing the other direction (press-hold on the device, then on the PLM) and I started seeing the traffic.

Before I did the second pairing, there was no insteon message traffic at all as seen by the PLM.

[rstanley75]

That did it! I should know better by now to always pair Insteon devices both ways.

I think I'll wait until tomorrow before I do that to the one installed in the front door. I think my wife is getting a bit tired of me constantly messing with the front door lock. (It's installed in the door frame so the deadbolt actuates the switch.)

How much effort do you think it would take to get the battery level data from the device as well?

[nugget]

Oh, the battery warning will be pretty simple, actually. I ran across that when I was looking at the dev docs for the device but I didn't want to get distracted so I shelved that work for now. It's definitely on the list and shouldn't be hard at all. The device doesn't support a literal battery level, but it will emit a low battery warning that will be no problem to detect and emit in hass.

[nugget]

Also, from the user manual, you can do the pairing for the 2845 after it's been installed by double-clicking the actuator button. So tomorrow you won't have to rip it out of the doorframe to pair it. The process is described in the user manual http://cache.insteon.com/documentation/2845-222-en.pdf

[rstanley75]

Oh sweet, thanks! That saves me a bunch of work.

Battery low info would be super handy. I'm pretty sure there's a way to get the battery level info as well but I only know this because the OpenHAB binding supports it (unless the OpenHAB binding has been lying this whole time, but I seems to accurate to be a fictional thing). I don't see anything about it in the dev docs so there must be some arcane magic involved, perhaps by reading memory locations?

[rstanley75]

Did a little investigation into the OpenHAB binding and I discovered something.

From src/main/resources/device_features.xml:

<feature name="HiddenDoorSensorData">
	<message-dispatcher>SimpleDispatcher</message-dispatcher>
	<message-handler cmd="0x03" group="1">NoOpMsgHandler</message-handler>
	<message-handler cmd="0x11" group="1">NoOpMsgHandler</message-handler>
	<message-handler cmd="0x13" group="1">NoOpMsgHandler</message-handler>
	<message-handler cmd="0x2e">HiddenDoorSensorDataReplyHandler</message-handler>
	<command-handler command="OnOffType">NoOpCommandHandler</command-handler>
	<poll-handler>NoPollHandler</poll-handler>
</feature>

Then, from src/main/java/org/openhab/binding/insteonplm/internal/device/MessageHandler.java:

    public static class HiddenDoorSensorDataReplyHandler extends MessageHandler {
        HiddenDoorSensorDataReplyHandler(DeviceFeature p) {
            super(p);
        }

        @Override
        public void handleMessage(int group, byte cmd1, Msg msg, DeviceFeature f, String fromPort) {
            InsteonDevice dev = f.getDevice();
            if (!msg.isExtended()) {
                logger.trace("{} device {} ignoring non-extended msg {}", nm(), dev.getAddress(), msg);
                return;
            }
            try {
                int cmd2 = msg.getByte("command2") & 0xff;
                switch (cmd2) {
                    case 0x00: // this is a product data response message
                        int batteryLevel = msg.getByte("userData4") & 0xff;
                        int batteryWatermark = msg.getByte("userData7") & 0xff;
                        logger.debug("{}: {} got light level: {}, battery level: {}", nm(), dev.getAddress(),
                                batteryWatermark, batteryLevel);
                        m_feature.publish(new DecimalType(batteryWatermark), StateChangeType.CHANGED, "field",
                                "battery_watermark_level");
                        m_feature.publish(new DecimalType(batteryLevel), StateChangeType.CHANGED, "field",
                                "battery_level");
                        break;
                    default:
                        logger.warn("unknown cmd2 = {} in info reply message {}", cmd2, msg);
                        break;
                }
            } catch (FieldException e) {
                logger.error("error parsing {}: ", msg, e);
            }
        }
    }

It looks like it's sending cmd 2e and then getting the battery level from the User Data 4 byte of the extended-length response.

On page 9 from the dev docs:

I had originally interpreted that little bit on page 9 as returning the level at which the low battery level alert is tripped, but since I get convincingly accurate battery level info from OpenHAB perhaps Data 4 is the actual battery level and Data 7 is actually the level where the alert gets tripped?

Edit: Finally read the docs on Markdown formatting.

[nugget]

I've submitted a pull request to have this component included in the main home assistant codebase.

home-assistant/core#6104

Lots of changes today and yesterday, reflected in an updated tarball as linked above.

[rstanley75]

I'm not sure if there's a better place to say this now that's in dev. I removed all of the old custom components and upgraded to the homeassistant dev branch. So far so good.

Also, that strange 'light.accf23c5814a' item no longer appears now that I'm on the dev branch. Not sure why but it could be worth mentioning.