Z-Wave JS official integration

[robertsLando]

Hi everyone,

Firstly sorry if this is not the right place to ask this, if not just redirect me to the correct place.

I'm the developer of zwave-js-ui, I started receiving many issues from your users that are currently using MQTT discovery in order to use their Z-Wave devices with Domoticz.

As I explained in docs:

MQTT Discovery is limited compared to a native integration and Home Assistant updates frequently break it. Based on this I would NOT RECOMMEND using MQTT Discovery, I don't plan to keep it maintained in the future.

So I'm here to ask you if you could consider developing a plugin that uses the websocket apis like Home Assistant Z-Wave JS integration does: https://zwave-js.github.io/zwave-js-ui/#/homeassistant/homeassistant-official?id=how-it-works

This is the most reliable way to work with z-wave js, informations about websocket apis can be found here: https://github.com/zwave-js/zwave-js-server

Ref: https://www.domoticz.com/forum/viewtopic.php?t=41959

[gizmocuz]

@robertsLando , the idea was to not write specific implementations for different middleware's but to use the MQTT Auto Discovery method.
This also works perfect with other middleware's like zigbee2mqtt.
Also, with a lot of other devices that support MQTT-AD (arduino projects, scripts, ... lots)

In the forum post I asked this user to follow a guide to report issues. In this post I only see ' I've seen that the discovery values send by zwavejsui for command topic were wrong'

That doesn't say me anything, and I have no clue if we could have worked around this.

Sure, there could be some possible differences that we can solve in the MQTT-AD implementation here in Domoticz.
In the end, I think you have all zwave command-classes covered by now.
So, there should be no or very limited changes to the MQTT-AD part in zwave-js-ui?

We tried to force all users to use ZWaveJS-UI instead of our deprecated OpenZWave implementation but have enabled OZW again...

You can imagine that implementing 'another' middleware will take a lot of time.
And the MQTT-AD implementation is 99% working? So maybe we could work around this to make it work.
But users have to report 'here' the issue first. With all information mentioned in our wiki/mqtt-ad debug guide.

I would tell them first to do this, and if they haven't done so close the issue.

[robertsLando]

We tried to force all users to use ZWaveJS-UI instead of our deprecated OpenZWave implementation but have enabled OZW again...

Could I ask you why you had to enable OZW again?

Anyway the point is that with MQTT AD it's a bit hard to handle composite devices like thermostats and fans that may need a custom json in order to be supported. Apart from this seems Domoticz isn't supporting all MQTT AD features very well as reported here

I dunno the effort needed on your side in order to build a "native" integration by connecting domoticz to the websocket exposed by zwave-js-ui but based on home assistant experience it's much more reliable then Discovery.

I asked the same to the OpenHAB team for the same reasons, this is just a suggestion BTW

[gizmocuz]

Could I ask you why you had to enable OZW again?

Sure, first let me say I mentioned a year ago that support for OZW will be dropped in Domoticz as there is no development for years now and that users should look at ZWaveJS-UI to MQTT

It was dropped with the first new stable release of Domoticz this year.
You would expect that users had already migrated since it was mentioned now for more than a year (by every stable release....)

But... of course... most users didn't care until it happened.

I advised to use 'docker compose' as this is a quite easy way to install services independent of the hardware platform (zwavejs-ui, MQTT, even domoticz)

Now most users expect an installation of Install->Next->Next->Running
Domoticz can be installed in this way via one command line, even docker compose should be easy, but it seems for most users to difficult. (for example, sharing a USB port in docker was already an issue)
The above is for maybe 70% of the users....

For the other 10-20% there where migration issues.
Some used secure zwave, had to copy/paste the network key, and it was not clear where to do this.
Some thought that they had to 'start all over again', no idea why because all configuration is inside the zwave stick...
And some users had problems getting everything to work. (half of it was because they had to wake up the battery nodes), but some hardware did not seem to work 'out of the box'

You can imagine that when we implemented MQTT-AD, a process that took homeassistant years, from time to time we had some challenges to get hardware supported.
Most Domoticz Zigbee users are using zigbee2mqtt, and these users are more 'tweakers' and helped a lot getting our MQTT-AD implementation working very nicely!
But then there are users that did not care installing zwavejs-ui, did not care about trying the beta versions, and when they upgraded to a new stable (a few times a year, compared to 100+ beta versions a year), then this was also the moment they noticed some malfunctions.

Very logical, and sure we can probably solve them by tweaking our MQTT-AD implementation, but this could have been avoided when more users where testing the beta versions.

So, for those users it came for them as quite a shock.

And so, for now OZW is back again... but I hope they all migrate to ZWaveJS-UI as it seems a great system!

I think we do a great job with MQTT-AD, zwavejs-ui is not the only platform working through it.
There are many projects (including zigbee2mqtt) that uses MQTT-AD and this is working correct.

It is also a matter of 'how it is implemented'.
For example, a light bulb, you can turn it on/off, some have brightness, and others have RGB/W/WW ... that's about it...
Very easy to implement in MQTT-AD, but why should the application (for instance Domoticz) care if the color is implemented as XY or color temperature... I think when the application sends rgb/w/ww, it is up to the middleware to translate that to the final device.
And if you need to use funky state templates (if a then c else if b then d), there is something seriously wrong.

In my opinion, one system to rule them all, would be ideal... (MQTT-AD)
It's a great system, and you can even control everything from multiple locations.

I hope for all zwave users that you will keep supporting it.

[sincze]

I can confirm @gizmocuz statements.
OZW deprecation was announced a long time ago, and indeed I was not happy with it.
But setting up ZWaveJS-UI in the end worked out.

It indeed requires basic docker-compose knowledge but nothing we can not document. (it still requires some reading skills from the user), but hey have you tried HA with all the custom YAML's ??

The issues that I had were part hardware (FW) related of the Zwavestick.
After that I am experiencing a stable environment. Happy to see how MQTT-AD actually worked. And yes I had to wake my doorsensors .... or just wait untill they woke up.

After that I started with ZIGBEE2MQTT then MILIGHTGATEWAY and even TASMOTA via teacherscripts -> MQTT AD.

All works as far as I can see.
This going to MQTT movement also ensures that I can use my hardware with Domoticz and HomeAssistant simultaneously (should I want that).

[robertsLando]

Firstly thanks for all the backgroud, I can say I definetely feel you, unfortunately telling users that something is deprecated is never taken so seriously until they find out it's removed and start bothering you.

And if you need to use funky state templates (if a then c else if b then d), there is something seriously wrong.

I definetely need to do that sometimes because MQTT Gateway functionality came before MQTT AD and so I had to adapt discovery to the topics/payloads of the gateway. This is easy for most entities but becomes so complex when it comes to thermostats and others like I said above (thermostats for example will send a new discover payload once the mode changes as the setpoint is different based on the mode).

This going to MQTT movement also ensures that I can use my hardware with Domoticz and HomeAssistant simultaneously (should I want that).

The same could be archieved by using Websocket BTW, just for reference

Anyway I have absolutely no problem to keep MQTT AD, I just don't have so much time to invest on it (I also don't have any Domoticz/OpenHAB or Home Assistant instance to test) as I give priority to other features

[gizmocuz]

Hi Robert, that's great news! Thanks for keeping it available!

Going to zwavejs-ui websockets indeed allows everyone to use zwave in both domoticz and homeassistant, but that's about it... there is much more hardware that (can) work with MQTT-AD
Still on my to-do list is to make available (selected) domoticz devices available as MQTT-AD

Regarding a thermostat, i see just a few discovery topics

• setpoint temperature (read/write)
• optional: actual temperature (readonly)
• optional: a few 'select' modes (home/away/holiday/... normal/comfort/fireplace)

I don't see what is so difficult here that needs to resend the discovery topics

When I implemented OZW (we had to implement all command-classes) this is exactly how I done it.

When you change a mode, the mode is just set to that mode. And yes, you probably get an update of the actual setpoint, and sometime later the temperature will change too.
In the end you just get some updates from the command classes

And (as far as I remembered... it has been a while...) those thermostat modes are also pre-documented?
But I have to admit, there are quite a few command classes that can be used for the thermostat.
But if you treat them all as separate entities, in the end they are just entities that have a value and can be set
If you can keep all complex logic inside zwavejs-ui, then MQTT wise they are (just) values.

Wish you all the best with your great project!!! I hope some more MQTT-AD developers will join to help as well!

I moved to zigbee, but have heaps and heaps of zwave stuff for testing available.

And Domoticz zwavejs-ui users should first report the issue on our forum, when it's a real problem, on github, and only, when it is really a possible zwavejs-ui issue, report to zwavejs-ui (after acknowledgement of our devs)

[robertsLando]

To understand what I mean maybe docs can help: https://zwave-js.github.io/zwave-js-ui/#/homeassistant/homeassistant-mqtt?id=custom-components

Anyway as long as users are happy with this I'm too :)

[robertsLando]

Just a reminder: user recently added the ability to discover Configuration CC values too. This will create MANY new entities for each device, all CC Configuration entities have a property that tells Domoticz to keep them disabled by default, this is handled by home assistant but not by Domoticz. Could you consider fixing that on MQTT AD? Where should I open the issue?

[gizmocuz]

Please feel free top open a new issue here, if possible post the config topic+payload

At the moment we use 'enabled_by_default' ... if this is set to default, it is added but not enabled in Domoticz.
Maybe we should ignore them?
But do you need to send these CC values to MQTT-AD?

[robertsLando]

At the moment we use 'enabled_by_default'

Yeah it's that, see PR

Some users reported it to be ignored but maybe they just saw the entities and were scared to see so many of them. Anyway the reasons are reported here: zwave-js/zwave-js-ui#3570 (comment)

[ghost]

Could I ask you why you had to enable OZW again?

From a user POV, understand that's much easier! On top of already explained issues, I see those:

-ZwaveJS not suitable for some existing systems due to OSes domoticz support or the heavy dependencies required (most related to JS choice IMO, HA core is in python as well as domoticz plugin system) for this single new zwave support: Those using windows hosts were left aside, as well as some using existing NAS (with low mem/cpu ressource) or routers that can handle user light VMs.
-Zwave is going down since a few years anyway, so why changing (when you can) if new hardware you add use zigbee for instance? There is still benefits to zwave (certification/better interoperability), but zigbee price tag is usually halved & choosing carefully devices can mitigate issues. If you add a now complex SW support setup for DIY stuff, at least for this market this combination IMO means the end of the road for zwave so keeping a legacy setup makes sense.

BR.

[daserra23]

Another vote for MQTT AD here for the same reasons as explained above by gizmocuz. I really hope that this remains an option, getting tired of breaking changes in the ecosystems.

[robertsLando]

IMO means the end of the road for zwave so keeping a legacy setup makes sense.

@yml78 LOL 🤣 Just give a look at this to understand what you are missing from OZW: openhab/openhab-addons#16374 (comment)

And that's only few points, consider also that OZW is simply unmaintained so most devices will not have a configuration provided and may not work at all (expecially locks and secure devices that require S2 security that is not supported by OZW).

Then ever heard about Z-Wave LR (long range) support? That's one of the new features that are coming that will have a range of ~1.5 miles.

Yeah, Z-Wave is going to die 😆

[ghost]

@robertsLando
Maybe I'm wrong, but not sure long range makes so much sense for home users. Not telling zwave will die soon, so much users, but if I was starting my home management now the devices I needed at the time & made me go zwave just does not exist anymore... but now do in zigbee.

[robertsLando]

@yml78 each protocol has it's pro/cons but I would not say that's the end of the road for z-wave, that's a very wrong statement expecially if you never tried zwave-js, I could agree with you if we were speaking about OZW as it made me want to throw all my zwave devices out of the window but that's because of the software not the devices/protocol itself.

[ghost]

@robertsLando
Both are low power mesh networks, zwave interoperability for sure better. But in the end, that's device choice driven by market: 5-10 years ago zwave was offering best device choice, now that's zigbee & for half to quarter of the price. That's not opinion just facts.
On my side & for legacy HW, OZW just does the job since years. Home management SW or plugins were built with the library without any heavy SW setup on user side.

[daserra23]

@yml78 each protocol has it's pro/cons but I would not say that's the end of the road for z-wave, that's a very wrong statement expecially if you never tried zwave-js, I could agree with you if we were speaking about OZW as it made me want to throw all my zwave devices out of the window but that's because of the software not the devices/protocol itself.

OZW was terrible and I stopped buying Zwave devices once I discovered Zigbee2mqtt. Zwave-JS-UI made Zwave so much better to manage but the transistion from OZW was traumatic and the thought of having to transition to yet another protocol would probably be the end of zwave for me. Life is too short.

[ghost]

OZW was terrible...

That's IMO not fair for the guy that kept this project alive (started with reverse engineering, don't forget it) for almost a decade.
On my side, out of device additions that sometimes caused issues (none I could not solve), I had uptimes sometimes over 4 months between 2 kernel updates that forced a reboot: So that's for sure stable. Can go holidays for weeks, my home management will be a no brainer.
What I really don't understand is why zwave chip provider did not built on a open source SW stack to help support their protocol after publishing specifications 7 years ago. They also later preferred to push for bulky zway: Don't understand the need to make SW driver so complex to handle a protocol mostly driven by serial-usb from host side... That's clearly not driving a 10G Eth controller over PCIe with many HW accelerators to handle it's pace, but the latest is more transparent to users. Unbelievable.
Now, as a consequence, Qubino is owned by Shelly, Aeotec goes zigbee... They just shoot themself in the foot with a machine gun. Silab & sigma design are anyway now under the same umbrella as well (i.e. they don't care who wins, will always be them!).
After zwave+/500 for so many years (indeed doing the job well) and zwave 700 finally out, 800 was soon there to help forbid a transition with many buggy devices (Transition from 8051 based HW, driven by bare metal SW, to ARM with RTOS was probably a bumpy road on device side).
That's quite many red signs IMO. But as I wrote, I may be wrong & time will tell...

[sincze]

IMO means the end of the road for zwave so keeping a legacy setup makes sense.

@yml78 LOL 🤣 Just give a look at this to understand what you are missing from OZW: openhab/openhab-addons#16374 (comment)

And that's only few points, consider also that OZW is simply unmaintained so most devices will not have a configuration provided and may not work at all (expecially locks and secure devices that require S2 security that is not supported by OZW).

Then ever heard about Z-Wave LR (long range) support? That's one of the new features that are coming that will have a range of ~1.5 miles.

Yeah, Z-Wave is going to die 😆

Link sums it up quite nicely. (migrating ozw to Zwave-js-ui took some steps) In the end gave me back all joy in life as now i can experiment with multiple domotica systems simultaneously.

Maybe zwave is less vulnerable to jamming security sensors compared to their zigbee Brothers? Both seem to offer me long battery life.

[robertsLando]

That's IMO not fair for the guy that kept this project alive (started with reverse engineering, don't forget it) for almost a decade. On my side, out of device additions that sometimes caused issues (none I could not solve), I had uptimes sometimes over 4 months between 2 kernel updates that forced a reboot: So that's for sure stable. Can go holidays for weeks, my home management will be a no brainer.

I know that but what I said is based on my personal experience. My first UI project (Zwave2MQTT) was using OZW but I never got so much support from the OZW maintainer, he only wrote me once to add the project under OpneZwave org but then I wrote him many many times both privately and on GH to ask for support with issues and so on but I was lost. This until I found zwave-js few years ago and his maintainer @AlCalzone, one of the best devs I ever worked with 🙌🏼

What I really don't understand is why zwave chip provider did not built on a open source SW stack to help support their protocol after publishing specifications 7 years ago. They also later preferred to push for bulky zway: Don't understand the need to make SW driver so complex to handle a protocol mostly driven by serial-usb from host side... That's clearly not driving a 10G Eth controller over PCIe with many HW accelerators to handle it's pace, but the latest is more transparent to users. Unbelievable. Now, as a consequence, Qubino is owned by Shelly, Aeotec goes zigbee... They just shoot themself in the foot with a machine gun. Silab & sigma design are anyway now under the same umbrella as well (i.e. they don't care who wins, will always be them!). After zwave+/500 for so many years (indeed doing the job well) and zwave 700 finally out, 800 was soon there to help forbid a transition with many buggy devices (Transition from 8051 based HW, driven by bare metal SW, to ARM with RTOS was probably a bumpy road on device side). That's quite many red signs IMO. But as I wrote, I may be wrong & time will tell...

This is a good point and I partially agree, anyway only time will tell us which protocol will win the "iOT protocols war", IMO z-wave and zigbee will both be still around for many years so keeping a "legacy OZW setup" would be a limit for Domoticz users that may prefer another OSS project with a better support of it like Home Assistant.