Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REQUEST] LiXee ZLinky TIC Zigbee OTA Provider for LiXee ZLinky TIC Router device firmware images from fairecasoimeme? #1060

Closed
Hedda opened this issue Oct 3, 2022 · 16 comments
Labels

Comments

@Hedda
Copy link
Contributor

Hedda commented Oct 3, 2022

@fairecasoimeme Wonder if you would consider contributing "download code" for ZLinky TIC Zigbee OTA provider to zigpy/ZHA?

https://www.home-assistant.io/integrations/zha#ota-firmware-updates

https://github.com/zigpy/zigpy/blob/dev/README.md#zigbee-device-ota-updates

Back-story: ZLinky_TIC, which is also known as "LiXee ZLinky TIC", is a Zigbee device made by Frédéric Dubois, a.k.a. fairecasoimeme (who also makes the ZiGate series of Zigbee Coordinator adapters) which should now work with zigpy dependent applications (with ZHA Device Handlers) but it first requires that the user upgrades to the latest firmware on the ZLinky_TIC device.

https://lixee.fr/produits/37-73-zigate-usb-ttl-3770014375148.html#/27-antenneexterne-non

Update: LiXee has since also released additional products such as ZiPulses (pulse counter) and ZigLight (WS2812 LED Controller):

https://lixee.fr/10-zigbee

https://lixee.fr/produits/40-zipulses-3770014375155.html

https://lixee.fr/produits/28-ziglight-controleur-leds-ws2812-3770014375063.html

Note! I think that ZLinky_TIC do support firmware updates Zigbee OTA if it already has been pre-flashed with firmware version v7.0 or later, but more importantly, there looks to be two variants of Zigbee OTA firmware for ZLinky_TIC; one "router" variant (i.e. Zigbee Router firmware) and one "non-route" (e.i. Zigbee End Device firmware), so not sure which should be the default for it?

https://github-com.translate.goog/fairecasoimeme/Zlinky_TIC/blob/master/README.md?_x_tr_sl=fr&_x_tr_tl=en&_x_tr_hl=sv&_x_tr_pto=wapp#route-or-not-route-from-v7 (Zlinky_TIC README.md translated from French to English with Google Translate).

https://github.com/fairecasoimeme/Zlinky_TIC

https://github.com/fairecasoimeme/Zlinky_TIC/releases

zigpy/zha-device-handlers#1165

https://lixee.fr/10-zigbee

https://lixee.fr/produits/37-zigate-usb-ttl-3770014375148.html

FYI, as you may or may not know, Home Assistant is currently the world's most popular open-source home automation software application and its native "Zigbee Home Automation (ZHA) integration" is a Zigbee gateway solution implementation that is one of the most popular platform integrations in Home Assistant as per statistics provided by the users who actively choose to opt-in to submit usage analytics (as analytics is optional and disabled by default in Home Assistant).

ZHA have support download code for third-party OTA providers but someone first needs to code a provider for each download source:

https://www.home-assistant.io/integrations/zha#ota-firmware-updates

https://analytics.home-assistant.io/#integrations

Anyway, having built-in support for more Zigbee OTA providers included with Home Assistant's ZHA integration is a common feature request from many users in the Home Assistant community, so before leaving this as an open request would like to ask if you yourself as the manufacturer could perhaps submit the small piece of code needed to allow the zigpy library (script code written in Python programing language) to download Zigbee OTA images (Over-The-Air firmware updates) for your own devices from your public Zigbee OTA provider as the source:

https://github.com/zigpy/zigpy/blob/dev/README.md#zigbee-device-ota-updates

See zigpy OTA provider code (provider.py) in Python which include the URL links as source of Zigbee OTA files for direct download:

https://github.com/zigpy/zigpy/tree/dev/zigpy/ota

https://github.com/zigpy/zigpy/blob/dev/zigpy/ota/provider.py

It will need a "fairecasoimeme ZLinky TIC OTA Firmware provider" to handle image download (see "Skeleton OTA Firmware provider" for reference, as well as note provider.py also already has code to handle the download of OTA files from IKEA, INOVELLI, and LEDVANCE as references).

Maybe also consider copying the concept of index.json with metadata on all files like Koenkk repository has to make scripting easier:

https://github.com/Koenkk/zigbee-OTA/blob/master/index.json

zigpy does not yet support this json index or other OTA Index files but Zigbee2MQTT allow users to use it locally as descibed here:

https://github.com/Koenkk/zigbee2mqtt.io/blob/develop/docs/guide/usage/ota_updates.md#local-ota-index-and-firmware-files

For example, code to download files from a GitHub repository using index.json metadata file see the zha-toolkit by mdeweerd here:

https://github.com/mdeweerd/zha-toolkit/blob/main/custom_components/zha_toolkit/ota.py

(zha-toolkit itself uses that to enable OTA notifications in, see -> https://github.com/mdeweerd/zha-toolkit#ota_notify).

PS: Please note that other than in the ZHA integration for Home Assistant the same zigpy library is also used as a dependency by the official Zigpy plugin for Jeedom as well as the Zigbee plugin for Domoticz, see:

https://doc.jeedom.com/en_US/plugins/automation%20protocol/zigbee/

and

https://www.domoticz.com/wiki/ZigbeeForDomoticz

PS: By the way, please also see this somewhat related discussion regarding here Koenkk zigbee-OTA repository -> #535

@Hedda
Copy link
Contributor Author

Hedda commented Oct 20, 2022

One device-specific hurdle to overcome that probably only applies to the LiXee ZLinky_TIC device is that @fairecasoimeme offer two differently configured Zigbee Router firmware image variants with the same "Zigbee Device Signature" for this device, with one of the OTA firmware images pre-configured with routing enabled (files tagged "route" in their name) and the other OTA firmware image pre-configured with routing diable (files tagged "no-route" in their name).

See example the files "ZLinky_router_v9.0.ota" and "ZLinky_router_v9.0_no_route.ota" in the ZLinky_TIC Router v9.0 release:

https://github.com/fairecasoimeme/Zlinky_TIC/releases

That problem was first mentioned here -> zigpy/zha-device-handlers#1146

Please also see the related and more detailed discussion in zigbee-herdsman-converters repo which have the same OTA firmware image selection dilemma with there being two firmware with the same "Zigbee Device Signature" -> Koenkk/zigbee-herdsman-converters#4785

This surprising lack of OTA firmware image variant compatibility is an illogical discrepancy due to historical reasons, as I understand, the reason for this is that the LiXee ZLinky_TIC device was initially released with just a Zigbee Router firmware that had routing enabled as this module was originally only designed and tested for use in Linky power-meters that has stable mains-power so the manufacturer did not see a problem with it having a Zigbee Router firmware that had routing enabled, but latter compatibility was extended for possible installation in other brands/models of power-meters and some of those does not offer stable mains-power and thus the manufacturer's workaround was to release an alternative OTA firmware image which is still practically the same Zigbee Router firmware but with routing disabled.

In hindsight, I think it would be better if the manufacturer had only ever released a Zigbee End Device (ZED) firmware for it.

https://shop.smarthome-europe.com/en/domotique/5111-lixee-tic-module-to-zigbee-30-for-linky-meter-3770014375148.html

image

image

image

@Hedda Hedda changed the title fairecasoimeme ZLinky TIC Zigbee OTA Provider for LiXee ZLinky TIC Router device firmware? [REQUEST] LiXee ZLinky TIC Zigbee OTA Provider for LiXee ZLinky TIC Router device firmware images from fairecasoimeme? Oct 20, 2022
@pdecat
Copy link

pdecat commented Oct 20, 2022

I think it would be better if the manufacturer had only ever released a Zigbee End Device (ZED) firmware for it.

Well, having it in router mode allows me to extend the battery life of other sensors located right next to it. I have no issues with this mode.

@pipiche38
Copy link
Contributor

pipiche38 commented Oct 20, 2022

You are 100% right. I have told the manufacturer from the beginning that it should a ZED and not a router.

The approach with 2 OTA is a nightmare and will be source of a lot of mistakes and problems.

Taking in consideration that I did some testing on upgrading and moving from one router to end device and back to router and got a dead Zlinky

@fairecasoimeme
Copy link

You are 100% right. I have told the manufacturer from the beginning that it should a ZED and not a router.

I have already said to you the reasons and why It's not a good solution. (Some users need childs routing and no sleep loop) I have already tested the ZED mode here : https://github.com/fairecasoimeme/Zlinky_TIC/tree/device_mode
This mode is not OK for this kind of object for many reasons.

The approach with 2 OTA is a nightmare and will be source of a lot of mistakes and problems.

Ok, and I leave some users in trouble because 2 OTA is complicated to manage ?

Taking in consideration that I did some testing on upgrading and moving from one router to end device and back to router and got a dead Zlinky

What is a dead ZLinky for you ? what is the relationship with the issue ?

@MattWestb
Copy link
Contributor

If devices is shipped to customer i think its to late making automatic OTA then its destroying the ZSED or router functionality then changing the firmware mode.

If only have the firmware for manual download and the user must selecting the right one then its working OK but not good in the long run.

If making one "universal" firmware that can changing mode by setting on the cluster level (its possible then some manufacture have making it but most likely its need being done in the device init phase and rebooting for getting it working OK) but its looks being not so essay implanting in the firmware for not braking devices that is use.

One other way is making 2 separate firmware with the same image number but one have 2.X (ZED) as app version on the basic cluster and the other have 3.X (Router) and the original is 1.X (Not known and shall being updated to app ver >2).

Then the host system must checking the app version then doing OTA updates and is working if app > 2 its updating ZED OK and router OK.
If having 1.X then need doing manual OTA with downloaded OTA with app 2 or 3 for getting the right function but i dont knowing if the flash layout id OK in the device for converting the device with OTA.

I think hardware version is not working then all device is shipped with the same setting. Is it more possibility getting firmware sub version from the device that can being used for automatic OTA selection ?

My feeling is that it shall have being made 2 different OTA imaged IDs for the different working mode but its now to late changing in the shipped devices.
Only the firmware cooker can telling what is possible,

Some more thinking:
IKEA is using Silabs EFR32MG1P chips and PCB for the On/Off dimmer and Open / Close. They is using the same OTA / App firmware / code but have different setting in the user data that is sett from the factory (also is containing the manufacture and model ID). By flashing (with SWD) user data from the other version the device is changing its functionality to the new type (have converting OnOff to OpenClose and its working OK.
With Silabs its not easy / not possible flashing the user data with OTA files but its very likely can being done with Silabs premium support that some of there customer is having.

Sorry for spamming the issue i only doing little brain storming that can helping the device devs getting better function in there device for the end users.

@pipiche38
Copy link
Contributor

pipiche38 commented Oct 20, 2022

You are 100% right. I have told the manufacturer from the beginning that it should a ZED and not a router.

I have already said to you the reasons and why It's not a good solution. (Some users need childs routing and no sleep loop) I have already tested the ZED mode here : https://github.com/fairecasoimeme/Zlinky_TIC/tree/device_mode

This mode is not OK for this kind of object for many reasons.

The approach with 2 OTA is a nightmare and will be source of a lot of mistakes and problems.

Ok, and I leave some users in trouble because 2 OTA is complicated to manage ?

Taking in consideration that I did some testing on upgrading and moving from one router to end device and back to router and got a dead Zlinky

What is a dead ZLinky for you ? what is the relationship with the issue ?

Totally disagree with the statement that ZED is not applicable. We have integrated the Wiser BMS which is going more less what Zlinky is doing and it is a ZED and not a router.
There is capabilities to make fast polling when needed and go to slow polling otherwise.

As a dead Zlinky the relationship is because moving from router to no router is jeopardising tables, and makes thing not working.

@fairecasoimeme
Copy link

The Wiser BMS is totally different that a Linky and the datas capture way is different. If you think the Wiser BMS do the same thing, you're totally wrong. I have already tested with fast polling and that's never fix the problem. (But I had already explain to you)

@pipiche38
Copy link
Contributor

I don't see the relation between data capture which I can understand is complex with Linky and the ZIgbee communication.
And to come back to the discussion having 2 OTA firmware with the same naming and especially Image Type is a source of problem and will create challenges on troubleshooting.

@fairecasoimeme
Copy link

fairecasoimeme commented Oct 20, 2022

I had already explained here : fairecasoimeme/Zlinky_TIC#100 (comment) . with sleeping device even if it's fast polling, there are packets loss
And for OTA firmware : Koenkk/zigbee-herdsman-converters#4785 (comment)

@Hedda
Copy link
Contributor Author

Hedda commented Oct 21, 2022

@fairecasoimeme Hopefully that will also mean that those separate image types get different "profile id" and/or "device type" as well so that the device types of the two images will be unique enough so that ZHA Device Handlers for zigpy will not be presenting devices with those different firmware images as the same "Zigbee Device Signatures" but instead seen as two different device models?

For reference, also see the latest updates about this in Koenkk/zigbee-herdsman-converters#4785 where @fairecasoimeme yesterday wrote a reply to @vk496 that he now has plans on changing OTA download releases so that there in the future will only be one official OTA download for the full version with routing enabled for OTA provider updates, but that he still plans on separately offering a light version without routing enabled via separate download which will require manual TTL/Serial flashing over UART for those who want to flash that in the special cases they are using a power-meter that does not allow the module to get stable mains-power.

https://github.com/zigpy/zha-device-handlers#signature

Originally posted by @vk496 in Koenkk/zigbee-herdsman-converters#4785 (comment)

I personally don't see any issue regarding the OTA and 2 firmwares. The current behavior in Z2M is to get the last release and take the first .ota file as update. If the headers of that ota match the device and are different in version, OTA popups for the user.

If @fairecasoimeme will continue putting the releases as it does until now (first ZRD then ZED), it will work for people with the current implementation as usual.

And after putting the hint in the images headers, a user will be able to change the default behavior from specific options and reflash as OTA update.

Originally posted by @fairecasoimeme in Koenkk/zigbee-herdsman-converters#4785 (comment)

Yes, I think I will make an official version with full routing functionality for children with a classic OTA header and later a version with a light routing functionality with another OTA header to avoid errors of update.

By default, ZLinky will be flashed with the official version. If a user wants to upgrade to the light version, the first flash will have to be done with the usb TTL module.

@Hedda
Copy link
Contributor Author

Hedda commented Feb 15, 2023

@fairecasoimeme FYI, puddly has just now added a new feature to the "zigpy-cli" tool that manufacturers can use to produce a skeleton JSON index from a given set of OTA files, and that way make sure manufacturers OTA files are correctly-structured so that they will be compatible with zigpy out-of-the-box. See the README.md for that "zigpy-cli" tool. See:

https://github.com/zigpy/zigpy-cli/blob/dev/README.md#ota

and

https://github.com/zigpy/zigpy-cli/blob/dev/README.md#generate-ota-index-files

PS: puddly mentioned when adding that feature for generating OTA index files via zigpy-cli was done in preparation for some OTA rewrites in the main zigpy library that he is planning to do.

@Hedda
Copy link
Contributor Author

Hedda commented Feb 27, 2023

@fairecasoimeme FYI, puddly is now working on supporting "Generic OTA providers" for zigpy via this new JSON format schema:

https://github.com/zigpy/zigpy/wiki/OTA-Information-for-Manufacturers

"This allows for new providers to be added to zigpy without extra code. This also allows for manufacturers to distribute test feed URLs to customers without deploying them globally."

This feature will be introduced via this new pull request that is still a work-in-progress. Please see -> #1165

@Hedda
Copy link
Contributor Author

Hedda commented Apr 3, 2023

FYI, “Manufacturer OTA Firmware Update Support” (also referred to as “Generic OTA providers”) should now be available for use in ZHA as that was added with zigpy 0.56.3 release and that was very recently bumped in Home Assistant core version 2023.8.1(?)

#1165

This new feature in zigpy (and ZHA) should allow for new OTA providers to be added to zigpy (and ZHA) without extra code, which in turn also allows both manufacturers (and end-users) to distribute test feed URLs to customers without deploying them globally.

https://github.com/zigpy/zigpy/wiki/OTA-Information-for-Manufacturers

PS: Again, puddly also added a new feature to the "zigpy-cli" tool that manufacturers can use to produce/generate a skeleton index from a given set of OTA files, and that way makes sure manufacturers OTA files are correctly-structured so that they will be compatible with this new JSON format for zigpy OTA providers out-of-the-box. Check out the README.md for the "zigpy-cli" (zigpy CLI tool) -> https://github.com/zigpy/zigpy-cli/blob/dev/README.md#generate-ota-index-files

Copy link

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Feb 12, 2024
@Hedda
Copy link
Contributor Author

Hedda commented Feb 18, 2024

More relavant now with #1340

@puddly
Copy link
Collaborator

puddly commented Feb 18, 2024

No more OTA providers will be implemented and existing ones will be deprecated in the future. New images will be submitted to a central repository instead.

@puddly puddly closed this as not planned Won't fix, can't repro, duplicate, stale Feb 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants