Adds OpenZWave 1.6 compatability #182
Conversation
Read the docstring in the Singleton class for a detailed description on how it works. I added this class as a meta class to ZWaveObject and also ZWaveOptions. This should cover all objects created in python-openzwave that are public.
The deprecated function now handles classes, methods, functions, properties (get and set together or separate) and variables with any data type set to it. A custom message can also be added to the warning that is displayed. the logging decorators are used for tracking the data path.
the ZWaveOption class is now an instance singleton so there is no need for this class anymore.
this method is to alter the node reloading when downloading updated node config files from OZW servers
…n the Windows build)
|
no worries m8.. when you get the time. |
|
I would like to help if possible, I kind of need / want this on Home Assistant (since it's required for the Siren 6 I have). How can I contribute (work, hardware, money, etc)? |
|
I need someone to be able to test and to update the HA side of things. I am willing to work with someone to get the job done. I am not familiar with Home Assistant nor do I have the time to learn how to code for it. If there is someone that does know how I am more then willing to help. as far as testing goes I need someone that is familiar with how to use Python and to be able to compile and install python-openzwave and to run tests to make sure that it runs properly without it being integrated into some other software. I did write a small script to run that do perform some basictasks on the network. I have been writing a GUI type of manager that is cross platform so be able to view the network and see what is happening and to be able to make changes to devices. and donations are always appreciated for work that is being done. https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WDHNKUNC3XC9L here is the repo where the current code that I have been working on is. https://github.com/kdschlosser/python-openzwave/tree/test If you are willing to learn and want to help with testing I can walk you through what to do and how to do it. Open up an issue on my repo using the link above and I will tell you what needs to be done. |
|
@kdschlosser I've had my hands in the Home Assistant zwave component a handful of times and have a separate development system with an extra z-wave stick available. I'd be interested in throwing some code at the HA side to get things updated. Only thing I'm lacking is a decent collection of spare nodes to test (only have a couple) but I could move a few from my main system to the development system if needed. I'd need some help/guidance getting the new OZW compiled/installed and your python-openzwave branch. |
|
Are you familiar with slack? If you are not it is a chat app geared at programming. Let me set that up and we can go from there. it is going to be easier for me to pass code examples to you. it is also an instant messenger and not a forum style post/reply. Right now I am adding into python-openzwave the ability to connect to it remotely. this would be strictly for management purposes from a cross platform GUI based manager I am working on. The purpose to it is this.. Say you have python-openzwave installed into home assistant. without having to shut down home assistant you are able to use the GUI manager either from the same machine or from a remote machine and connect to the already running python-openzwave. This is all possible because i have made the program thread safe and removed anything that would not allow something like this to work. The system I designed runs over a websocket connection all of the data is encrypted using AES and the encryption/decryption key gets set on either end by the user as well as the ip address/port to use. and just to add another layer of security, optionally a CA Key file and CA Certificate file can be provided as well if the user wanted to use an SSL connection in combination with the AES. Because of the control of security systems and door locks we do have to make sure this thing is secure. Ideally a person would not want to forward the traffic over the internet but if it gets done the security is there. No passing of any usernames or passwords takes place, and only 1 connection is allowed at a time. everything has to be hard coded on either end and that information does not get sent over the wire at all. So even if someone decided to do a man in the middle they may be able to get past the SSL but not the 256bit AES. I am also going to add the option of providing a list of allowed IP addresses a user can connect from for yet another layer of security. Let me set up that slack chat and I will drop an invite to it here. |
|
here is that invite link. I need to know what OS you are running so we can get openzwave compiled for ya. |
|
python setup.py build build_docs --dev libopenzwave.pyx:3576:39: error: no member named 'GetNodeClassIds' in 'OpenZWave::Manager' I am not familiar with GetNodeClassIds on Manager... You run a modified version or OZW or did I have insufficient caffeine and missed something? :) |
|
ok first thing is that should not be taking place with this PR. I imagine you are testing against my "test" branch on my fork of pyozw. If that is the case then you didn't read the commit notes. and you probably have not seen the PR on OZW's repo either. 😄 I made a minor adjustment to the code of OZW that provides a way to get the supported command class ids on one function call instead of however many application command classes there are * the number of nodes. that number is up there.. 5000-7000 calls for a 100 node network. I added in the remote management code into my latest commit to my test branch. imagine how much additional traffic that is... |
Okay
Yes, a few posts back you've said "here is the repo where the current code that I have been working on is. https://github.com/kdschlosser/python-openzwave/tree/test" So I tried to test "the latest" which indeed is not kdschlosser:new-1.6 - sorry for the confusion
Only the readme, not the commit messages. I saw a reference to a non-existing PR ;)
Could not have seen it because I did my test and wrote my comment 12 h ago and you wrote your PR 9 hours ago. I get your point about GetNodeClassIds - but it is really for @Fishwaldo to comment on that PR |
|
OK cool. have you gotten around to testing at all? any issues? I want to know if the thing even compiles on anything other then windows. |
|
Your "test" branch got as far as that missing GetNodeClassIds and I did not try with your proposed PR yet, lack of time. I did not test your "new-1.6" branch, does it differ a lot from the test version you've sent on Discord a while ago? |
|
yes my test branch is very different then the 1.6. I pulled the PR for the command class ids. I came up with a work around for the problem. Tho I am not able to get the 12 ms speed like what that PR does. I am getting 19ms which is a hell of a lot better then 1.4 seconds! the code in my test branch has already been updated with the removal of GetNodeClassIds. It's kind of ugly what I had to do... But it works. |
|
I added a new feature to my test branch.. I have seen some complaints regarding the startup time of applications that are using python-openzwave. I came up with a mechanism to speed up the startup times of python-openzwave. (kind of) I added that remote ability. The key is this. Because of how I added the remote ability it allowed me to provide the option of installing python-openzwave as a service (Windows/Linux/Mac). Here is the trick.. Most times when an application needs to do a restart it is not because of the zwave stack. It needs to do a restart because of some kind of added module or plugin. or maybe to just clean out the buffers so to speak. Because of the remote system i designed and having 2 copies of the python version of the network. when the application reboots it resets the applications storage of the nodes and values but it does not restart openzwave. This remains running (as a service) the only time that will get reset is on reboot or if the application expressly instructs it to restart. This trims down perceived application startup times by 85-90%. You couple that with the other changes I have made and you are looking at close to a 96-98% is increase in startup performance. Now that is pretty cool. I do not think that if there is an issue with a device that restarting openzwave would fix the issue, I may be wrong in my assumption on that. If someone has seen where restarting openzwave has solved a problem please chime in and let me know. This way I might be able to code in some better mechanism to be able to handle this issue. if running the service on a NIX based system you have the traditional systemd start, stop, restart and status. they all work. I also implemented the watchdog that is built into systemd as well. so this will check to see if the service is still alive and kicking. and if not it will restart the service automatically. I have to set up an automatic reconnect routine on the client side but that is not to big a deal. on Windows the service will function like any other service and it will also write information to the event logs which can be viewed using event viewer. There is a single entry point to install/manage the service it is a script called pyozw_service and has a slew of command line options. basically an options for everything that gets set using ZWaveOption plus a few more for starting stopping installing removing the service. This script will need to be run using sudo or administrative privileges. There are dependencies that are needed for the service. to install the dependencies run the setup program by going into the folder with the setup.py file and typing in the following command.
|
|
OK so here goes. on my test branch.. Windows service is working 100% now. I have not tested the NIX based service as of yet. The client/server i have fixed a few bugs and I also added error checking/recovery with the resend of a message if by chance it got corrupted in transit. It also checks for message that may not have gotten delivered and it will do a resend if this is the case. I improved the performance by leaps and bounds. From the time the client connects until the network i considered fully loaded on the client side it takes 1.3 seconds for 13 that have a combined total of 301 values. on the client side if I iterate over all of the nodes print the node id and name then iterate over all of the values refreshing each value and printing out the value index, name and data. This whole process takes 1.1 seconds. or an average of 84ms per node or 3.66 ms per value/ I know my tests are local and have not been tested over a physical network. all you would really have to factor in is network latency into that. so you are still talking milliseconds. |
|
What's happening here? Is there any plans on fixing those tests and merge this? |
|
e are working on the update. It is a rather involved process because of the dependency to labels. There are also several performance related issues that are also being repaired at the same time. This is a very large undertaking almost a complete rewrite. This update is API breaking But the good news is it should be a lot easier to add to an application. It is being bug tested and added to HASS now. I have some life things that are happening over the next week that I have to attend to so I may not be able to work on it. |
|
I'm waiting for 1.6 compability in python-openzwave becase I'm hit by some bugs/shortcommings in openzwave 1.5. I've quickly had this code in a test setup locally with HA and it looked like it works. I'm always a bit afraid when doing multiple big changes at the same time, IE both the 1.6 compability and the new daemon-mode. That said, I'm not the one doing this, I'm just hoping that its ready soon. In that context, is there anything we can do to help getting this done? |
|
what exactly did you get working? This PR, or the test branch on my fork? there is a person I have been working with to get my test branch running in HASS. My test branch corrects a huge assortment of problems not just adding 1.6 compatibility. 1.6 cannot be added in it's entirety unless HASS undergoes some pretty large changes. The HASS author does not want to make any large changes so we are kind of at a road block with respect to that. I personally do not use HASS. I did look at the code and there is simply to much "voodoo magic code" in it for m to be able to write an interface for HASS. Someone that really knows the inner workings of HASS could write an integration in a very small time frame. I have been more then willing to help with the python-openzwave end of things. If you are proficient and writing code to work with HASS and you would be willing to write a new interface I am more then willing to help out in any way that I can. I have almost completely rewritten python-openzwave, huge speed improvements, proper handling of endpoints/instances and association groups, supports command classes in a correct manner. There is a mechanism that I wrote into python-openzwave that allows an application to never have to know what a label, name, id, index or instance is. This mechanism allows python-openzwave to become part of the application instead of operating outside of the main program. If you are interested then the best thing to do would be to fork HASS make a new branch and make modifications to that branch this way I can see what you are working on and make suggestions. Let me know what you would like to do. (fyi: I am also a diver (padi master diver)) |
|
FYI. The HASS team have decided since it would require them to rewrite their Zwave integration to support Kevin’s work, they will adopt a different path to integrate OZW that does not have a dependency upon pyozw (or any forks). I’ve provided a proof of concept on the OZW side to them, and they are starting work on that. For updates you should ask in the #zwave channel on their discord server. I am still looking for someone to provide incremental updates to this repository that would support OZW 1.6 while maintaining API compatibility with the original pyozw interface (to support existing projects that use pyozw). I don’t consider Kevin’s work as pyozw any more, but rather a new implementation. |
|
I tested the "new-1.6" branch (904b815) to the point that it did build, and I managed to start HA with zwave component loaded. I don't remember if I let it talk to my zwave stick or not. At that point I probably concluded that I don't have the stuff to do propper testing, without the posibility to break my existing network. Sad to see these massive rewrites breaking apis. |
|
The largest issue was that pyozw I think was written by mainly c developers and the knowledge was enough to get it working more as a low level kind of interface. The biggest thing about python is it is easy to use as are most of the libraries written for it. You have to keep in mind that python for many people is a first language because of it's ease of use. I had several years of python knowledge under my belt when i first tried using pyozw and was pretty confused at how a lot of it functioned because it wasn't "pythonic" when you look at the current release and compare the code against what developers do in c code it is very similiar, The problem with coding in python using a c developers style/mentality the code ends up being horribly inefficient. you end up with data objects that get created that really do not need to be, lots of iterations over objects, stuff like that. hence the reason why I was able to cut at least 80% of the overhead out of the library. also the way the library was written is an application would have to hold reference to node id's and value id's and do all kinds of checking to determine how and what to do. This makes it really complex to use. This can still be done if that is what they want to do. actually in the test version I wrote quite a bit of the API is still in tact. access to nodes is the same. access to values is the same. the methods that did searches have been removed but those are easy to duplicate on the application side if needed. The largest API breaking change made is the jumble of methods added to ZWaveNode. there were methods/properties to turn a light on and they would be available for a node that is a thermostat. I am willing to bet my branch could be used and not to many changes would need to be made to HASS in order to get it to run. I am curious now and I may go and tinker with it. |
|
I just modified HASS to run with my test branch. took me all of an hour to do.. I did not add anything all i did was make the changes needed to work with my test branch.. Took me a whole hour to do. I am sure there are some things that need to be hammered out with it like the automations using a an integer node id and the new node id's are strings. I didn't see where in the code that was located to make that adjustment. But the core of it is done... This is what I do not understand.. why is there such a grumble about working on this thing, I am beginning to think that the grumble is not about python-openzwave but the grumble is because of having to make any changes because of HASS, Looks are deceiving a lot of the old API is still in tact and is very much usable. The single thing that has been removed that HASS used a bunch was iteration over values returned by using the get_values method in ZWaveNode. That mechanism was inefficient so there for removed. this is an example of what I am talking about def set_wakeup(service):
"""Set wake-up interval of a node."""
node_id = service.data.get(const.ATTR_NODE_ID)
node = network.nodes[node_id]
value = service.data.get(const.ATTR_CONFIG_VALUE)
if node.can_wake_up():
for value_id in node.get_values(class_id=const.COMMAND_CLASS_WAKE_UP):
node.values[value_id].data = value
_LOGGER.info("Node %s wake-up set to %d", node_id, value)
else:
_LOGGER.info("Node %s is not wakeable", node_id)That just looks like a keyboard puked all over the screen. It's not easily readable. Plus it is also error prone because it is setting all variables to whatever the data is. In the case of the wakeup interval it does work but only because the rest of the values are read only. But what happens if the command class gets updated and another value gets added that is not read only? It also does not even get into the complete waste of overhead setting values that have no purpose being set in the first place. So the code above got replaced with. def set_wakeup(service):
"""Set wake-up interval of a node."""
node_id = service.data.get(const.ATTR_NODE_ID)
node = network.nodes[node_id]
value = service.data.get(const.ATTR_CONFIG_VALUE)
from openzwave.command_classes import COMMAND_CLASS_WAKE_UP
if node == COMMAND_CLASS_WAKE_UP:
node.wakeup_interval = value
_LOGGER.info("Node %s wake-up set to %d", node_id, value)
else:
_LOGGER.info("Node %s is not wakeable", node_id)no iteration. no trying to set something that should not be set. efficient equality testing for command class support which on the back end does not iterate either. neither does using the performance wise in this scenario you may only be talking 10 milliseconds or so difference. but add that time up for every call like this for every node on a network. milliseconds turns into seconds pretty quickly. total time to make the changes to this one method. 30 seconds (and that is being generous) |
|
Kevin: I suggest you read over the feedback given in the slack channel by the HASS devs, not just on the code but everything else they have mentioned around development style, lack of stability, no releases etc etc etc. |
|
I'd like to lurk in the slack channel if it is publicly available. Link in this conversation is no longer valid. |
|
Same. I'd love to be able to keep track of this in detail, as well as contribute where possible, either HASS side or PyOZW side, but I feel like I'm just barely seeing all of thoughts and effort that have gone into this so far. |
|
OK look it boils down to this.. The devs over at HASS do not want to update any of their code. I posted a drop in replacement that adds most of the 1.6 changes in the slack chat, that was a week ago.. and nothing has been done. This has nothing to do with making smaller releases I could make 1000 releases and they still would not do a thing with it. This is not the first time I have run across this same exact thing with HASS and them not wanting to update a thing in their application. That is their decision. I am not sure as to why there is the thought that i have some ability to create new releases of python-openzwave.. I do not. I am just a person that forked it and made changes to it to suit my needs. I have made a couple of PR's and some of them have been added but that is the extent of my "official" involvement thus far. and because of the nature of the changes I have made it would be very difficult to create separate PR's if the PR's are not being added because of the reliance one PR will have on another. If someone wants to use my fork of pyozw they are going to have to make changes to their code to use it. I do not know what else you want me to do?? I cannot force them to update their application. and I am not going to do it for them. When you released openzwave 1.6 what was stated in the release notes? that it is backwards compatible with 1.4... then why does it not work with the existing python-openzwave??? There are changes that are not backwards compatible i am guessing. The new features added to openzwave are what the users are interested in. So unless HASS is wiling to update their code and actually does the work their users will never get these new features even if 1.6 was backwards compatible. This update is not a bunch of small releases and changes. This is a major code change to openzwave and thus a major code change would need to be made to the python binding as well. and that trickles down to any application that uses the binding. HASS is not the gold standard and a developer should not have to write a library to work with HASS. HASS should be written to work with the library. I am the administrator of another HA program it has been around for 12 years, It has 400+ unique downloads a week. that's 249600 users. if only half of them are using it there are 124800 users. These are the people that I am responsible to and are the reason why I have made the changes to python-openzwave If some other application wishes to use the changes then they are permitted to do so. but in no way should they expect me to fix their application or make it work with their application. they need to fix and update their application to work with the changes I have made. I integrated python-openzwave into my application on Aug 06, 2018 and in that time I have had 39 support tickets. and of those 39 tickets less then 1/2 were to report problems most were questions. The last report of a problem was on Sep 13, 2018 since then not one complaint not a single bug report. I am curious to know how many issue in that time exist for HASS. I can say that every single one of mine are closed and fixed.. can they make the same claim?. I write code that runs and it runs without problem or errors. It works plain and simple. I have it tested as much as possible before i make any release. I work with the users/developers one on one. if there is a single problem reported that is one problem to many. In order to get python-openzwave to behave in a manner that was error free and was up to snuff in the performance department I had to write a wrapper around it it extended the functionality of it at the same time. I decided to add that directly to python-openzwave and to also fix/repair any other issues i came upon. I also found weak spots in it and corrected those as well. I wanted to be the "nice guy" and decided to share those changes and what do I get in return? A migraine. So now I stand here dumbfounded as to how I should be expected to make it work with some other application. HASS would not be able to offer ZWave to Windows users if it was not for me. have I even gotten a single thank you for it?? NO actually i got the "I don't give a shit" kind of response actually. It took me 6+ months to get the damned thing to compile on windows using python. 6 months of time and effort that they are using. Now has there been a single complaint about that portion not working since it was added?? NOT 1 issue. any problems that exist are because of HASS and HASS needs to be fixed, a library should not have to make special considerations and code around problems in HASS. I am honestly starting to get irritated like I did last time and that is not something I want to do. I am not being compensated in any manner other then gratitude and that I am not even getting for the time I am spending. I do this for enjoyment and when it is no longer enjoyable there is no reason to be doing it. This is now the 2nd time I am going to walk away from a project because of HASS, It is not my loss I will continue to do the work that I do for my users to benefit from, Its really a shame that the users of HASS are not going to be able to. As a side note. How is the above different from the present situation? an application has the ability to keep on using the old code base. or they can use something new and improved this is their choice their decision to make. If spending a single hour to make the changes that will make their users happy and fix bugs and performance problems is to much effort... Not using it is not going to break my heart at all. (this next bit is not directed at you @Fishwaldo or anyone else that has been involved and those people know who they are. I do appreciate the help and the education I want to make sure that you do know that) It is the end user that is the one that ends up suffering the most. All I can say is.. I tried... Life is to short to spend time trying to improve things for an application when the application author does not want to spend the time to help. @Fishwaldo is a great example of what an application author should do and how involved they should be, other should follow his example. 👍 There are 2 things that can never be replaced in life.. The first thing is a loved one.. The second is TIME --Kevin Schlosser I am not going to spend a single second more of my time doing something I am no longer enjoying, is not appreciated and keep on getting told I am doing it wrong with no offer of assistance. and on that note. |
|
@kdschlosser I went to grab your fork of this to see if I can hack something together for my own use, but I noticed that you took it down. Can you make it available again? |
|
I am not going to make it available as I am not going to support the use if it. I have decided to write a pure python ZWave binder. There are many parts of the ZWave specification that are not added to OZW and quite a few of these parts I have a need for in my personal network. I have had conversations about adding some of them but due to the large amount of work needed to add them it is not going to happen. I am more then 2/3rds done and I should be testing it in a month or so. I feel that removing a repository that I have no intention of ever supporting was the thing to do. If you want a copy of the library I can send it to you. let me know. |
|
@kdschlosser I'll have to keep an eye out for that, if you're planning to release it! A copy of your fork would also be great in the meantime. If you prefer email, you can hit me via the address on my GH profile. |
This PR adds in the necessary pieces to make python-openzwave compatible with OpqnZWave 1.6
@Fishwaldo has created the following issues outlining what has changed in the new version.
#180 Depreciated Features
#179 New Notification Types
#178 SendRawData Method
#177 Catching Exceptions from OZW
#176 Config File Revision Management
#175 Device MetaData
#174 BeginControllerCommand Depreciated
#173 Associations
#172 Instance Labels
#171 SwitchMultiLevel Command Class
#170 Central Scene Command Class
#169 UserCode CommandClass
#168 Alarm (Notification) Command Class Updates
#167 Handling Labels
#166 ValueID index param changed to uint16_t
#165 Support BitSet ValueID
There are also some other bug fixes as well. #1 being the requirements in the setup program. This is now fixed and the code removed for the redundant checking of requirements. In order to make the requirements work properly python-openzwave is installed as an egg. (not compressed). the wheel package did not make a working wheel when used. I wrote a custom bdist_wheel command class that creates a wheel that is in compliance with PEP, this includes the sha256 hash of the files.
During my work with fixing the requirements there was some code I cleaned up in the install command class and I noticed the blind deletion of the device configuration files. I did not really care for this to much. as a user I know I would be pretty mad if a file got removed that I had altered or created, even more so because there is no easy way to recover the file(s) (if at all). What it does now is it makes a backup of the files first. then does an install. it then check each device config file for the following things. and in this order
This is not going to cover every single possible scenario I am sure. But it is a whole lot better then a blind deletion of the whole tree.
Most documentation is in place (not all). There is no breakage of the API at all (none that I am aware of)
If using any of the new properties I coded in a type of caching into the program. So the constant iteration of a devices values will no longer take place. (unless it is a requirement of a node example is a device that supports COMMAND_CLASS_NOTIFICATION)
Unittests have also been updated to work with the new changes also.
fixes #180 fixes #179 fixes #178 fixes #177 fixes #176 fixes #175 fixes #174 fixes #173 fixes #172 fixes #171 fixes #170 fixes #169 fixes #168 fixes #167 fixes #166 fixes #165