[REQUEST] Zigbee 3.0 and Zigbee PRO 2017 (R22) network forming support in bellows?

[Hedda]

This is a request/suggestion to make bellows support Zigbee 3.0 and/or Zigbee PRO network forming?

This directly relates to Zigbee 3.0 network supported requested for zigpy in zigpy/zigpy#360 and zigpy/zigpy-znp#29 and zigpy/zigpy-cc#64

That is, make it possible for bellows with or without zigpy to form a Zigbee 3.0 network allowing newer devices to use a Zigbee 3.0 profile as an option instead of ZHA 1.2 profile in a Zigbee Home Automation 1.2 network if and when newer devices support Zigbee 3.0 profiles?

If I understand it correctly, some of not all newer Zigbee Stacks support Zigbee 3.0 (Z3.0) network forming which allow a Zigbee network formed which support multiple profiles at the same time, allowing Zigbee 3.0 devices to connect using a Zigbee 3.0 profile, Zigbee Light Link 1.x devices can connect using a ZLL 1.0 profile, and Zigbee Home Automation 1.2 devices can connect using a ZHA 1.2 profile, or?

• Zigbee Cluster Library (ZCL) v7
• Zigbee PRO 2017 (R22)
• Child device management
• Base Device Behavior (BDB)
• Green Power

I know that bellows radio library just very recently got support for the later EZSP API versions used Silicon Labs EmberZNet Zigbee Stack firmware that supports Zigbee 3.0 and the zigpy-znp radio library supports the newer Texas Instruments chips use Z-Stack 3.0 Zigbee Stack firmware which all do support Zigbee 3.0 standard.

https://www.silabs.com/community/wireless/zigbee-and-thread/knowledge-base.entry.html/2018/04/24/how_to_form_zigbee3-zrzB

https://www.silabs.com/documents/public/application-notes/an1117-migrating-zigbee-profiles-to-z30.pdf

"At the core of Zigbee 3.0 is the Zigbee PRO 2017 (R22). Note that previously Zigbee Pro 2015 (R21) was required for Zigbee 3.0, which has now been replaced with the newer Zigbee Pro 2017 (R22) specification. Older implementation based on the R21 specification are still compatible with the new R22 specification. The Zigbee PRO Specification adds child device management, improved security features, and new network topology options to Zigbee networks. Commissioning devices into networks has also been improved and made more consistent through Base Device Behavior (BDB). Zigbee 3.0 furthermore requires Green Power Basic Proxy v1.1.1 functionality in all devices to further support Green Power capabilities and compiles all profile clusters into a single specification, Zigbee Cluster Library (ZCL) v7. Finally, it formalizes common Zigbee device application architecture nomenclature, expands on the Zigbee Lighting & Occupancy Device Specification, and comments towards Zigbee 3.0 certification. The following sections address each of these attributes."

https://www.ti.com/lit/swra615

Previous versions of Silicon Labs firmware was also Zigbee PRO compatible, and the core of the Zigbee 3.0 standard is based on the Zigbee PRO 2015 specification, while the core of the Zigbee 3.1 standard will be based on the based on the Zigbee PRO 2017 specification.

[MattWestb]

The key is the NCP firmware that doing more or less all under the hood. Its possible forcing the NCP to working in lower stack profile but i think no one is doing that. Take a look for certificated NCPs (https://zigbeealliance.org/zigbee_products/?product_type=compliant_platform) for latest certified stack version . The rest is how much of the ZB3 you like implanting but LL and HA profile in in place and most other is not worth inplanting in a general system like HA. Zigbee Pro is stack profile 2 (0 = private and 1 stack v1) is putting the network layer of all version 2 profiles stacks together and on that ZB3 its built.
In some months the R23 is released and more demands its coming and more back and forwards problem with it.

[Adminiuga]

Yeah, I don't think from ezsp protocol perspective forming a ZB3 network is very different. I would think mostly difference if you define the well known TC link key or use install codes

Would need to check network creator plugin sources

[Adminiuga]

@Hedda did you get Ebyte module ? And can you build a swd programmer?

[MattWestb]

@Adminiuga I have 2 working set for the EFR32MG1P132F256GM32 with both bootloaders and EZSP app (6.6.4.0 and 6.7.4.0). grobasoz EZSP firmware
If i'm right its suld working on the Ebyte module if flashing it over SWD (Gecko 0<->1<->2 its not working then different CPU and flash size).
Have you reading / dumping the internal flash (region 0) and user mem (region 1) of the flash it should being safe trying flashing it.
Its verified working with tasmota zigbee bridge firmware ad zigbee host.

[Hedda]

This won't be worked on for a while because I don't have any Zigbee 3.0 devices to test with (although I think I heard somewhere that IKEA might be pushing OTA updates to make their end-devices Zigbee 3.0 compliant?).

FYI, most added Zigbee 3.0 support via OTA updates so you probably have many more Zigbee 3.0 capable devices than you know ;)

All new devices from a while back must be Zigbee 3.0 compatible to be added on Zigbee Alliance list of their certified devices:

https://zigbeealliance.org/zigbee_products/

Just some examples of very popular and commonly available devices include those from manufactures/brands such as; GE, HEIMAN, IKEA, Innr, LEDVANCE, Leedarson, Legrand, Linkind, Müller-Licht, Namron, Paul Neuhaus, Paulmann, Philips (Hue), Schneider Electric, Sengled, Samsung (SmartThings), Sunricher, Sylvania, TaHoma, TuYa, Ubisys, and even Xiaomi (Aqara).

https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Aqara
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=GE
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=HEIMAN
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=hue
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=IKEA
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Innr
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=LEDVANCE
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Leedarson
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Legrand
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Linkind
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=M%C3%BCller
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Namron
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Neuhaus
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Paulmann
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Schneider
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Sengled
https://zigbeealliance.org/zigbee_products/page/2/?product_type=certified_product&se=SmartThings
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Sunricher
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Sylvania
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=TaHoma
https://zigbeealliance.org/zigbee_products/?product_type=certified_product&se=Ubisys

As you can see on that list many popular devices that have already been available to buy for a while are now Zigbee 3.0 certified.

Zigbee Alliance is pushing very hard for Zigbee 3.0 certification and devices need to be independently tested to be listed there.

PS: I also read that ITead is aiming to get its latest Sonoff branded Zigbee sensors (Silabs based) certified for Zigbee 3.0 now.

[Hedda]

Yeah, I don't think from ezsp protocol perspective forming a ZB3 network is very different. I would think mostly difference if you define the well known TC link key or use install codes

Would need to check network creator plugin sources

@puddly wrote this reply in zigpy/zigpy-znp#29

I believe that there are no changes to network formation, you just need to supply the device's install code to the coordinator before it joins the network.

For reference, here is how bellows implements it:

bellows/bellows/zigbee/application.py

Lines 541 to 564 in 5874e59

	async def permit_with_key(self, node, code, time_s=60):
	if type(node) is not t.EmberEUI64:
	node = t.EmberEUI64([t.uint8_t(p) for p in node])
	key = zigpy.util.convert_install_code(code)
	if key is None:
	raise Exception("Invalid install code")
	link_key = t.EmberKeyData(key)
	v = await self._ezsp.addTransientLinkKey(node, link_key)
	if v[0] != t.EmberStatus.SUCCESS:
	raise Exception("Failed to set link key")
	v = await self._ezsp.setPolicy(
	t.EzspPolicyId.TC_KEY_REQUEST_POLICY,
	t.EzspDecisionId.GENERATE_NEW_TC_LINK_KEY,
	)
	if v[0] != t.EmberStatus.SUCCESS:
	raise Exception(
	"Failed to change policy to allow generation of new trust center keys"
	)
	return await self.permit(time_s)

[Hedda]

@Hedda did you get Ebyte module ? And can you build a swd programmer?

I never got the Ebyte module, but should be easy to do like @MattWestb and rip out ICC-A-1 Module from IKEA TRÅDFRI device:

https://github.com/MattWestb/IKEA-TRADFRI-ICC-A-1-Module

You can use that either via USB-to-serial converter or create own WiFi-bridge based on ESP8266 and Tasmota or JeeLabs ESP-LINK.

@MattWestb Do you need an SWD programmer to flash an IKEA TRADFRI ICC-A-1 Module?

[MattWestb]

New ZB devices perhaps but updated devices can being "recertified" as Ikeas old RGBWW bulb LED1624G9 was getting one update for some month and still being LL but its still working OK in a ZBB3 mesh.
The problem is the chinese "ZB3" device like my "HOMA" serie that is "chinese ZB3" and don't have replay protection implanted (if some sniffing and getting the network key they can steering the device very easy). Tha same is very likely for all Xiami and COs not ZB3 certified devices :-(((
What i knowing from looking in the OTA files is Ikea only having 3 LL devices that not having being upgraded to ZB3: The old RGBWW, One old LED driver and the od motion sensor (and the GWs NCP is LL certified but very likely R21 or R22 now).

Stefan H have the tasmota running in ZB3 mode and my Ikea bulbs is reporting children after repower as from the ZB3 book.

09:16:39 RSL: RESULT = {"ZbParent":{"Device":"0x8BAE","Name":"IKEA_E27_WW","Children":3,"ChildInfo":["0x14B457FFFE70C4BB","0x14B457FFFED4D031","0x90FD9FFFFE59BB73"]}}

I think its not being so big problem for @Adminiuga getting it working in bellows then hi have getting the multiversion up and running and being stable (first priority and the rest comes later).

[Hedda]

The rest is how much of the ZB3 you like implanting but LL and HA profile in in place and most other is not worth inplanting in a general system like HA. Zigbee Pro is stack profile 2 (0 = private and 1 stack v1) is putting the network layer of all version 2 profiles stacks together and on that ZB3 its built.

Personally I am mostly curious about improved/enhanced power-saving/sleep features in Zigbee 3.0 profiles for wireless sensors so that batteries do not have to be replaced as often. As well as interested in new "Zigbee Green Power" (ZGP) devices and its profile:

https://zigbeealliance.org/zigbee_products/?product_type=certified_product&certified_type=green-power-device
https://zigbeealliance.org/solution/green-power/
https://www.silabs.com/community/wireless/zigbee-and-thread/knowledge-base.entry.html/2020/02/13/zigbee_green_power-fKk4
https://www.qorvo.com/resources/d/qorvo-zigbee-green-power-multi-sensor-comparison-to-battery-powered-zigbee-devices-white-paper
https://training.ti.com/zigbee-green-power-enable-battery-less-devices

To clarify, I am not only integrated in ZGP (Green Power) devices, but also power-saving improved/enhanced in other profiles too.

I think others are also be interested in the additional power-saving/deep-sleep functionality implemented in Zigbee 3.0 profiles.

I believe most of us probably hate having to replace batteries in wireless sensors, buttons and remotes as often as needed today?

[Hedda]

have the tasmota running in ZB3 mode and my Ikea bulbs is reporting children after repower as from the ZB3 book.

I would think that it would not be possible to change profile on an already paired device(?), so I assume that if you perform an firmware OTA update on a device from ZHA or ZLL to Z3.0 then you would have to remove and re-pair that device again to get it to run in Z3.0 profile/mode if you do not want it to run in a backwards-compatible mode using its old ZHA or ZLL profile?

[MattWestb]

Yes @Hedda its needed SWD programmer, then devices is shipped with (application)bootloader for OTA updates (signed ones) and must being flashed with one standalone bootloader (aka Xmodem).
If you have a real L-Link its easy but expensive. I have building on J-Link clone of STM23 and is working but its little dangerous and having hard bricking 2 ICC-A-1 with it. Stephan H have bricking one sonoff zigbee bridge EFR chip with his original J-Link last weekend (and his first one have the ESP bad flash).
If you is having one ESP its easy and reliable making black magic probe (BMP) of it and using GDB for flashing.
Also possible making one BMP of one STM23 but easier using ESP for both flashing and tasmoting ;-)
I recommending using one E1743 aka On/Off Dimmer Switch (ca 6€ in EU) as it easy to disassembling without destroying it or the case and easy soldering connections on it. Its also gets in "FAMELY PACK" with one E27 WW bulb for around 10€ in EU (Not for Staphan H then needs souldring and no code braking), I was buying 3 of the family packs for testing as soon Ikea was opening after corona shutdown.
For the moment we have 6.6.4.0 and 6.7.4.0 EZSP as s37 files (s37 = SWD flashing but no working GB files = Xmodem flashing) all with EZSP com pins and bootloader trigger for the E1743.

@Adminiuga Do you need more versions of the EZSP for testing the multiversion belloes ?
I can asking our firmware maker (grobasoz) for other versions if you need that for doing it easier testing (I think V4 its too old or the EFR23 but all newer may being possible to make).

Edit: @Hedda One danich have using Rpi and GDB for flashing the MG1 with SWD so not needed having one STM32 or ESP. The EZSP can also being connected thru GIPO to the Rpis commport (have not trying it then my RaspBee I is occupying the GIPOs and running deCONZ). Then it should not being any problem running ZHA on the Rpi.

[MattWestb]

@Hedda What I have see then devices getting upgraded to ZB3 thru OTA they can still working or getting problems (experience from deCONZ) and need being deleted and repaired for "talking" ZB3 and using the new / changed cluster.
LL to ZB3 its have many cluster being changed, HA1/2 to ZB3 its more likely that all clusters its the same but better repairing the device for sure.

As mented about is how the paring is working if ZB3 or backward compatibility mode is in use.
With wrong keys and / or configuration both the coordinator and the devices is falling back to lower security and not being ZB3 (can being that the devices like ZB3 but the coordinator trustcenter dont have the right keys = No ZB3 pairing) and so on, its very much that must being OK from both sides for real ZB3 to working and not downgrading the mesh.

PS: All Ikeas ZB3 router devices have GP cluster so also the new HUE switch suld being happy with them.

[MattWestb]

I was looking little in deCONZ on my Philips HUE devices and THAT is one large MESSSSS !!!
All have ZCL set to 1 (= no Zigbee PRO = LL 1 or 2 cant being HA 1 or 2 with ZLL commission cluster).
The bulbs have LL Light and LL GP as endpoints and one ZLL commission cluster.
The dimmer switch have LL light and HA GP as endpoints and no ZLL commission cluster.
Motion sensor have LL light and HA GP as endpoints and ZLL commission cluster.
True LL devices cant having HA endpoints (and i think no GP endpoints).
True HA devices cant having LL commission cluster if not being ZB3 (ZCL = 2)
For having mixed mode profiles the ZCL must being 2 and using Zigbee PRO for building multi profile zigbee on and having GP cluster implanted.
Its looks more or less as our "chinese ZB3" devices, but its its working well and its much $€$€.
It can being that deCONZ is doing crazy things here but the ZCL attribute is red from the device and not manipulated of deCONZ but the endpoint type is deCONZ made.
Philips also having problems with not braking the compatibility with the old 1 gen HUE gateway.
What i can see have the crazy swedes manage to making their devices more Zigbee standard then converting the Tradfri system from LL to ZB3 (the GW is still LL certified but new ZB3 devices with HA profile is in the pipeline). I think they have learning from the first LL compatibility problems they was having and getting much blame for and was being fixed with OTA updates.
I think its better not deep diving in the Xiaomi here then all (except the ca 10 last one) is not zigbee standard nor certified and cant being updated thru OTA.
So wat is left inverting the $€$€ in ?

[Adminiuga]

Reading zcl_version to tell the Zigbee version is unreliable IMHO. Too much variance between the vendors. But if there's a GP endpoint, then you could assume it is a ZB3 device

[MattWestb]

Yes most of the Philips HUE have being updated to ZB3 and certificated (have not checking all but the list is very long).
Philips have the same problem as Ikea with the GWs is still LL certificated and cant braking the backwards compatibility.
I think Ikeas GW is on EZSP 5.7.2.0 (from OTA file name) and don't have older GW HW as Philips having.
I think the only false is ZCL = 1, its the falling price then its definitely Zigbee PRO with LL, HA and GP profiles with or without ZB3 (I think the right name is Zigbee PRO 2015 or 2017).
The good thing its looks like it working for Philips :-))
And all community devs is getting more problems to solving and making work arounds for bad implementations.

How is it going with the multiversion EZSP ??
I'm curious and like testing it on my mini test setup with Ikea and Xiaomi devices then you is ready.

[Adminiuga]

multiversion is going fine, bulk of work was done, just need to update tests and do more testing with EZSP v8 joining and forming.
if you modify the homeassistant/components/zha/manifest.json to replace bellows==0.18.x line with git+https://github.com/zigpy/bellows@experimental#bellows==0.23.0 then you could test it

[MattWestb]

I trying to making the mod but normally HA don't like my doing things in it (Perhaps is deCONZ that is spamming HA to death) :-((
If it working I can use both 6.7.4.0 for v8 and the 6.6.4.0 for v7 testing.
Is looks like my Ikea ZB3 bulbs is doing fine with tasmota EZSP (v8) after Stephan H was fixing the reply protection in ZB3.
And the "bulk" its only a small understatement what i have se of you commits ;-))

[Hedda]

I was looking little in deCONZ

FYI, I am pretty sure that deCONZ and ConBee/RaspBee firmware only support "Zigbee PRO" and is not Zigbee 3.0 compliant.

If I remember correctly, deCONZ developers wrote that not even newer ConBee II / RaspBee II firmware support Zigbee 3.0

its definitely Zigbee PRO with LL, HA and GP profiles with or without ZB3 (I think the right name is Zigbee PRO 2015 or 2017).

https://www.silabs.com/documents/public/application-notes/an1117-migrating-zigbee-profiles-to-z30.pdf

"At the core of Zigbee 3.0 is the Zigbee PRO 2017 (R22). Note that previously Zigbee Pro 2015 (R21) was required for Zigbee 3.0, which has now been replaced with the newer Zigbee Pro 2017 (R22) specification. Older implementation based on the R21 specification are still compatible with the new R22 specification. The Zigbee PRO Specification adds child device management, improved security features, and new network topology options to Zigbee networks. Commissioning devices into networks has also been improved and made more consistent through Base Device Behavior (BDB). Zigbee 3.0 furthermore requires Green Power Basic Proxy v1.1.1 functionality in all devices to further support Green Power capabilities and compiles all profile clusters into a single specification, Zigbee Cluster Library (ZCL) v7. Finally, it formalizes common Zigbee device application architecture nomenclature, expands on the Zigbee Lighting & Occupancy Device Specification, and comments towards Zigbee 3.0 certification. The following sections address each of these attributes."

[MattWestb]

Derstens second gen NCP have better HW (CPU men flash and so on) but they have not implanting all the good things from the first gen and they is saying they is "software" comparable between G1 and 2.
The braking thing for My is that tuch link is not implanted in the G2 NCPs.
Its one of the best thing with it and using the old "Web app" for resetting / steeling and showing other zigbee meshs.
The Ikea module (6€) have the same or better HW and RF performance as the second gen from Dersten (40€) and the EFR 2 is one generation higher.
Then bellows have the EZSP working for gecko 0, 1 and 2 gen its killing deCONZ true ZHA and if Z2M is coming on the train its RIP for deCONZ.

[Adminiuga]

after Stephan H was fixing the reply protection in ZB3.

What was the fix? Any links?

[MattWestb]

The issue is here and the commit is linked little under (tasmota #9038)

The thred its very much brainstorming so not so easy following all things but you understand it.

[MattWestb]

@Adminiuga Was trying your test setting but HA in docker don't like that.

Then I was putting this in the "Custom deps deployment"

pypi:
  - 'git+https://github.com/zigpy/bellows@multiversion#bellows==0.19.0.dev0'
apk: []

Its looks like HA have installing the multiversion but my devices acting like crazy (leaving the network all the time the trying pairing them and getting time out).
Trying reflashing the NCP and erasing the flash for cleaning the NVR.

v6.6.4.0 working better only the E1743 leaving then paring and have not reading all the cluster of the old motion sensor.

[Hedda]

@MattWestb testing multiversion testing is really off-topic here, maybe start separate issue for that discussion to not spam this?

[MattWestb]

@Hedda Bellows multivision with EZSP v6.6.4.0 and v6.7.4.0 is indeed ZB3 on Zigbee PRO 2017 (R22) (with no "and / or" then R22 its one of the moderate things for ZB3 as the Green Power routing is).

This is a request/suggestion to make bellows support Zigbee 3.0 and/or Zigbee PRO network forming?

If all is going well then is bellows "ZB3 ready" and the devs can impleniting more ZB3 profiles then they like.
Or do you like that the devs to stopping the refactoring of bellows ?

[Adminiuga]

What is different in Zigbee 3.0 network forming?

[MattWestb]

For forming i don't knowing what policies that must being set or unset for the EZSP to being ZB3 compient.
The ground is that the network suld being R22 or higher and using ZCL. Devices shall reporting ZCL attribute in the basic cluster as 2 or higher (also trust center / coordinator). Thats is already implanted in the EZSP stack. The Base Device Behavior Specification should being implanted for the trustcenter = the network.
The largest security change is how to using the default link key as a transient link key (a link key with a timeout, after which it will no longer work) and not allowing it as permanent link key for ZB3 devices. ZB3 devices shall leaving the network if the security is not correct configured in the network (= trustcenter policy is wrong / security is to low).
Frame counter is moderate in all communications links and so on.
Silabs version is the paper "AN1233: Zigbee Security" that describing the behaviour and problematic but not how to configuring the trust center for being compliant to the ZB3 standard.

As Adminiuga is knowing is the multiversion bellows for the moment running ZB3 devices in ZB3 mode with updated Link Key after Joining (verified from device key exchange in the NCP log) = true Zigbee 3 !!!

I only hoping its going well with with the falling back security for old not ZB3 devices (most extreme incompatibility ones like Xiaomi) so the ZB3 security is not being broken then finilacing the implantation in the last stage of development. Much testing is needed and very likely fine tuning the code for getting it running running stable with good compatibility for old / bad behaving devices.
Then its stabile its still a large work to implanting more applications profiles that opening for more devices being used in the zigbee system and making it more capable and flexible for our systems in the future.
I hope the maturing the multiversion bellows is going well and soon going in the master branch before the next big step is coming and need more large work from the devs the R23.

Great work is done and more is coming if I knowing the devs right !!!