Z-Stack 3 on CC2531/CC2530 adventures

[Koenkk]

This issue is created in order to gather feedback of users using the CC2531/CC2530 AND Z-Stack 3.0 (= Zigbee 3.0)

You are about to enter a highly experimental area (no guarantees)

Notes

• Zigbee 3.0!
• Firmware can be found here in the bin folder: https://github.com/Koenkk/Z-Stack-firmware/tree/dev/coordinator/Z-Stack_3.0.x
• It's not possible to migrate your Zigbee 1.2 network (requires repairing)
• The network restore trick that we used in Zigbee 1.2 doesn't work anymore (due to better security policies). A backup of the CC2531/CC2530 is taken at every shutdown of zigbee2mqtt, file can be found under data/coordinator_backup.json. This backup is automatically restored when starting with a new flashed CC2531/CC2530.
• Requires latest zigbee2mqtt dev branch.

Known issues

• Pan ID and extended pan ID can't be set (probably not something to worry about)
• Pairing only seems possible during the first 180 seconds (even with permit_join: true, a workaround is to restart zigbee2mqtt.

[jarrah31]

What are the benefits of Zigbee 3.0 for zigbee2mqtt? Will it allow more than the default 15 devices on a CC2531 coordinator (and if so how many)? Do we need new firmware on the CC2531 routers as well?

btw, thanks @Koenkk for a fantastic product and all the effort you put into this project, it is very much appreciated!

[Koenkk]

@jarrah31 it also has a direct children limit of 15. No new router firmware is needed (zigbee 1.2 devices can join a zigbee 3.0 network). More about zigbee 3.0 changes can be found here: http://www.ti.com/lit/an/swra615/swra615.pdf

I've just released a new firmware 20190423 (https://github.com/Koenkk/Z-Stack-firmware/tree/dev/coordinator/Z-Stack_3.0.x/bin).)

• Increased stability
• Try to fix issue that kicked of Xiaomi devices from the network
• Increased number of direct children from 10 to 15

[Z0l]

Hey, I'm new to this whole Zigbee thing but I got this one up and running with the future intent of integrating with Openhab to finally make my Hue Tap do things Philips never dreamed of :) Is there documentation available how GPP devices can be paired? Perhaps something specific for Hue Tap? I'm running 20190423 with the latest dev Docker container.

[Koenkk]

@Z0l you can try to pair it, setup procedure seems to be here: https://huetips.com/tutorials/how-to-add-a-hue-tap/ (10 seconds press a button).

[runningman84]

I have flashed my cc2531 with CC2531ZNP-with-SBL.hex:

sudo ./cc-tool -e -w CC2531ZNP-with-SBL.hex                                                                                                                                            ±[●][master]
[sudo] Passwort für phil: 
  Programmer: CC Debugger
  Target: CC2531
  Erasing flash...
  Completed       
  Writing flash (241 KB)...
  Completed (17.97 s.)

but the stick does not seem to work with zigbee2mqtt anymore:

> zigbee2mqtt@1.3.1 start /opt/zigbee2mqtt
> node index.js

  zigbee2mqtt:info 4/28/2019, 9:59:45 PM Logging to directory: '/opt/zigbee2mqtt/data/log/2019-04-28.21-59-45'
  zigbee2mqtt:info 4/28/2019, 9:59:46 PM Starting zigbee2mqtt version 1.3.1 (commit #39d6a8e)
  zigbee2mqtt:info 4/28/2019, 9:59:46 PM Starting zigbee-shepherd
  zigbee2mqtt:info 4/28/2019, 9:59:51 PM Error while starting zigbee-shepherd, attempting to fix... (takes 60 seconds)
  zigbee2mqtt:info 4/28/2019, 10:00:51 PM Starting zigbee-shepherd
  zigbee2mqtt:error 4/28/2019, 10:00:56 PM Error while starting zigbee-shepherd!
  zigbee2mqtt:error 4/28/2019, 10:00:56 PM Press the reset button on the stick (the one closest to the USB) and start again
  zigbee2mqtt:error 4/28/2019, 10:00:56 PM Failed to start
        {"message":"rsp error: 2","stack":"Error: rsp error: 2\n    at rspHdlr (/opt/zigbee2mqtt/node_modules/zigbee-shepherd/lib/components/controller.js:309:29)\n    at CcZnp.<anonymous> (/opt/zigbee2mqtt/node_modules/cc-znp/lib/ccznp.js:264:13)\n    at Object.onceWrapper (events.js:277:13)\n    at CcZnp.emit (events.js:189:13)\n    at CcZnp._mtIncomingDataHdlr (/opt/zigbee2mqtt/node_modules/cc-znp/lib/ccznp.js:366:14)\n    at Immediate._onImmediate (/opt/zigbee2mqtt/node_modules/cc-znp/lib/ccznp.js:342:22)\n    at runCallback (timers.js:705:18)\n    at tryOnImmediate (timers.js:676:5)\n    at processImmediate (timers.js:658:5)"}
  zigbee2mqtt:error 4/28/2019, 10:00:56 PM Exiting...
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! zigbee2mqtt@1.3.1 start: `node index.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the zigbee2mqtt@1.3.1 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/pi/.npm/_logs/2019-04-28T21_00_56_252Z-debug.log

after pressing the reset button on the stick I get this error:

pi@raspberrypi:/opt/zigbee2mqtt $ npm start

> zigbee2mqtt@1.3.1 start /opt/zigbee2mqtt
> node index.js

  zigbee2mqtt:info 4/28/2019, 10:03:41 PM Logging to directory: '/opt/zigbee2mqtt/data/log/2019-04-28.22-03-41'
  zigbee2mqtt:info 4/28/2019, 10:03:42 PM Starting zigbee2mqtt version 1.3.1 (commit #39d6a8e)
  zigbee2mqtt:info 4/28/2019, 10:03:42 PM Starting zigbee-shepherd
  zigbee2mqtt:info 4/28/2019, 10:03:44 PM Error while starting zigbee-shepherd, attempting to fix... (takes 60 seconds)
   zigbee2mqtt:info 4/28/2019, 10:04:44 PM Starting zigbee-shepherd
  zigbee2mqtt:error 4/28/2019, 10:04:45 PM Error while starting zigbee-shepherd!
  zigbee2mqtt:error 4/28/2019, 10:04:45 PM Press the reset button on the stick (the one closest to the USB) and start again
  zigbee2mqtt:error 4/28/2019, 10:04:45 PM Failed to start
        {"message":"request timeout","stack":"Error: request timeout\n    at CcZnp.<anonymous> (/opt/zigbee2mqtt/node_modules/cc-znp/lib/ccznp.js:261:22)\n    at Object.onceWrapper (events.js:277:13)\n    at CcZnp.emit (events.js:189:13)\n    at Timeout.<anonymous> (/opt/zigbee2mqtt/node_modules/cc-znp/lib/ccznp.js:240:18)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)"}
  zigbee2mqtt:error 4/28/2019, 10:04:45 PM Exiting...
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! zigbee2mqtt@1.3.1 start: `node index.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the zigbee2mqtt@1.3.1 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/pi/.npm/_logs/2019-04-28T21_04_45_888Z-debug.log

the same stick worked fine with the old version:

pi@raspberrypi:/opt/zigbee2mqtt $ npm start                  

> zigbee2mqtt@1.3.1 start /opt/zigbee2mqtt
> node index.js

  zigbee2mqtt:info 4/28/2019, 8:29:24 PM Logging to directory: '/opt/zigbee2mqtt/data/log/2019-04-28.20-29-24'
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM Starting zigbee2mqtt version 1.3.1 (commit #39d6a8e)
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM Starting zigbee-shepherd
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM zigbee-shepherd started
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM Coordinator firmware version: '20190223'
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM Currently 1 devices are joined:
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM 0x7cb03eaa00a6f7a3 (0x7cb03eaa00a6f7a3): AB3257001NJ - OSRAM Smart+ plug (Router)
  zigbee2mqtt:warn 4/28/2019, 8:29:25 PM `permit_join` set to  `true` in configuration.yaml.
  zigbee2mqtt:warn 4/28/2019, 8:29:25 PM Allowing new devices to join.
  zigbee2mqtt:warn 4/28/2019, 8:29:25 PM Set `permit_join` to `false` once you joined all devices.
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM Zigbee: allowing new devices to join.
  zigbee2mqtt:info 4/28/2019, 8:29:25 PM Connecting to MQTT server at mqtt://192.168.178.71
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM zigbee-shepherd ready
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM Connected to MQTT server
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'online'
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM MQTT publish: topic 'zigbee2mqtt/0x7cb03eaa00a6f7a3', payload '{"state":"ON","linkquality":78}'
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM MQTT publish: topic 'homeassistant/switch/0x7cb03eaa00a6f7a3/switch/config', payload '{"payload_off":"OFF","payload_on":"ON","value_template":"{{ value_json.state }}","command_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3/set","state_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3_switch","unique_id":"0x7cb03eaa00a6f7a3_switch_zigbee2mqtt","device":{"identifiers":"zigbee2mqtt_0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3","sw_version":"Zigbee2mqtt 1.3.1","model":"Smart+ plug (AB3257001NJ)","manufacturer":"OSRAM"},"availability_topic":"zigbee2mqtt/bridge/state"}'
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM MQTT publish: topic 'homeassistant/sensor/0x7cb03eaa00a6f7a3/linkquality/config', payload '{"unit_of_measurement":"-","value_template":"{{ value_json.linkquality }}","state_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3","json_attributes_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3_linkquality","unique_id":"0x7cb03eaa00a6f7a3_linkquality_zigbee2mqtt","device":{"identifiers":"zigbee2mqtt_0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3","sw_version":"Zigbee2mqtt 1.3.1","model":"Smart+ plug (AB3257001NJ)","manufacturer":"OSRAM"},"availability_topic":"zigbee2mqtt/bridge/state"}'
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM MQTT publish: topic 'zigbee2mqtt/bridge/config', payload '{"version":"1.3.1","commit":"39d6a8e","coordinator_firmware":20190223,"log_level":"info","permit_join":true}'
  zigbee2mqtt:info 4/28/2019, 8:29:26 PM Successfully configured 0x7cb03eaa00a6f7a3 (0x7cb03eaa00a6f7a3)

[runningman84]

Btw. the same error happens in the dev branch (commit #4472c01)

[Koenkk]

Can you provide the output of DEBUG=* npm start (on https://pastebin.com/)

[Z0l]

For me it works with the dev version:

4/28/2019, 8:58:36 PM - info: Logging to directory: '/app/data/log/2019-04-28.20-58-36' 4/28/2019, 8:58:36 PM - info: Starting zigbee2mqtt version 1.3.1 (commit #unknown) 4/28/2019, 8:58:36 PM - info: Starting zigbee-shepherd 4/28/2019, 8:58:38 PM - info: zigbee-shepherd started 4/28/2019, 8:58:38 PM - info: Coordinator firmware version: '20190425' 4/28/2019, 8:58:38 PM - info: Currently 0 devices are joined: 4/28/2019, 8:58:38 PM - warn: 'permit_join' set to 'true' in configuration.yaml. 4/28/2019, 8:58:38 PM - warn: Allowing new devices to join. 4/28/2019, 8:58:38 PM - warn: Set 'permit_join' to 'false' once you joined all devices. 4/28/2019, 8:58:38 PM - info: Zigbee: allowing new devices to join. 4/28/2019, 8:58:38 PM - info: Connecting to MQTT server at mqtt://localhost 4/28/2019, 8:58:38 PM - info: zigbee-shepherd ready 4/28/2019, 8:58:38 PM - info: Connected to MQTT server 4/28/2019, 8:58:38 PM - info: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'online' 4/28/2019, 8:58:38 PM - info: MQTT publish: topic 'zigbee2mqtt/bridge/config', payload '{"version":"1.3.1","commit":"unknown","coordinator":20190425,"log_level":"info","permit_join":true}'

However nothing happens when I keep pushing the button for 10 sec (I tried pushing all 4). Is the limitation on pairing has to happen in the first 180 seconds still in place? Also, is that normal that the led turns off after some time (didn't measure how long, but when I plug the USB in and start the docker container, the LED turns off).

[lolorc]

@Z0l the pairing limitation is actually a zigbee 3.0 feature :-)
https://www.ti.com/lit/an/swra615/swra615.pdf "Permit join can no longer be enabled forever as networks will automatically close joining after a maximum of 254 seconds."
just resend the mqtt cmd permit_join true when the 254s have passed.

[runningman84]

@lolorc it would be cool if zigbee2mqtt would expose a switch like "permit_join" to homeassistant.

[lolorc]

@runningman84 a hack is probably already possible. using zigbee2mqtt/bridge/config/permit_join as a mqtt switch with retain true. I guess it would also be possible to tell home assistant to set it to false 254s after it has been set to true.
but I agree it would better to be able to read from zigbee2mqtt/bridge/config/permit_join and get the updated status, and also having z2m setting it to false after 254s

[Koenkk]

@runningman84 @lolorc don't spend time on this. Soon I will implement something that zigbee2mqtt does this automatically.

[runningman84]

@Koenkk

Can you provide the output of DEBUG=* npm start (on https://pastebin.com/)

https://pastebin.com/h0hFLEuA

[Koenkk]

@runningman84 after updating, did you do a rm -rf node_modules && npm install?

[runningman84]

I just run rm -rf node_modules && npm install and flashed the firmware once again. Now it seems to start:

pi@raspberrypi:/opt/zigbee2mqtt $ npm start

> zigbee2mqtt@1.3.1 start /opt/zigbee2mqtt
> node index.js

  zigbee2mqtt:info 4/29/2019, 8:49:59 PM Logging to directory: '/opt/zigbee2mqtt/data/log/2019-04-29.20-49-59'
  zigbee2mqtt:info 4/29/2019, 8:50:00 PM Starting zigbee2mqtt version 1.3.1 (commit #a54e256)
  zigbee2mqtt:info 4/29/2019, 8:50:00 PM Starting zigbee-shepherd
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM zigbee-shepherd started
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM Coordinator firmware version: '20190425'
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM Currently 1 devices are joined:
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM 0x7cb03eaa00a6f7a3 (0x7cb03eaa00a6f7a3): AB3257001NJ - OSRAM Smart+ plug (Router)
  zigbee2mqtt:warn 4/29/2019, 8:50:02 PM `permit_join` set to  `true` in configuration.yaml.
  zigbee2mqtt:warn 4/29/2019, 8:50:02 PM Allowing new devices to join.
  zigbee2mqtt:warn 4/29/2019, 8:50:02 PM Set `permit_join` to `false` once you joined all devices.
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM Zigbee: allowing new devices to join.
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM Connecting to MQTT server at mqtt://192.168.178.71
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM zigbee-shepherd ready
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM Connected to MQTT server
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'online'
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM MQTT publish: topic 'zigbee2mqtt/0x7cb03eaa00a6f7a3', payload '{"state":"ON","linkquality":0}'
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM MQTT publish: topic 'homeassistant/switch/0x7cb03eaa00a6f7a3/switch/config', payload '{"payload_off":"OFF","payload_on":"ON","value_template":"{{ value_json.state }}","command_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3/set","state_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3_switch","unique_id":"0x7cb03eaa00a6f7a3_switch_zigbee2mqtt","device":{"identifiers":"zigbee2mqtt_0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3","sw_version":"Zigbee2mqtt 1.3.1","model":"Smart+ plug (AB3257001NJ)","manufacturer":"OSRAM"},"availability_topic":"zigbee2mqtt/bridge/state"}'
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM MQTT publish: topic 'homeassistant/sensor/0x7cb03eaa00a6f7a3/linkquality/config', payload '{"unit_of_measurement":"-","value_template":"{{ value_json.linkquality }}","state_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3","json_attributes_topic":"zigbee2mqtt/0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3_linkquality","unique_id":"0x7cb03eaa00a6f7a3_linkquality_zigbee2mqtt","device":{"identifiers":"zigbee2mqtt_0x7cb03eaa00a6f7a3","name":"0x7cb03eaa00a6f7a3","sw_version":"Zigbee2mqtt 1.3.1","model":"Smart+ plug (AB3257001NJ)","manufacturer":"OSRAM"},"availability_topic":"zigbee2mqtt/bridge/state"}'
  zigbee2mqtt:info 4/29/2019, 8:50:02 PM MQTT publish: topic 'zigbee2mqtt/bridge/config', payload '{"version":"1.3.1","commit":"a54e256","coordinator":20190425,"log_level":"info","permit_join":true}'
  zigbee2mqtt:warn 4/29/2019, 8:50:14 PM No converter available for 'AB3257001NJ' with cid 'genBasic', type 'devChange' and data '{"cid":"genBasic","data":{"manufacturerName":"OSRAM","modelId":"Plug 01","powerSource":1}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:14 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:warn 4/29/2019, 8:50:14 PM No converter available for 'AB3257001NJ' with cid 'genBasic', type 'devChange' and data '{"cid":"genBasic","data":{"zclVersion":1,"appVersion":12,"stackVersion":2,"hwVersion":1,"dateCode":"20140331DEOS****","swBuildId":"V1.04.12"}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:14 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM No converter available for 'AB3257001NJ' with cid 'genBasic', type 'devChange' and data '{"cid":"genBasic","data":{"deviceEnabled":1}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM No converter available for 'AB3257001NJ' with cid 'genIdentify', type 'devChange' and data '{"cid":"genIdentify","data":{"identifyTime":0}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM No converter available for 'AB3257001NJ' with cid 'genGroups', type 'devChange' and data '{"cid":"genGroups","data":{"nameSupport":0}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM No converter available for 'AB3257001NJ' with cid 'genScenes', type 'devChange' and data '{"cid":"genScenes","data":{"count":0,"currentScene":0,"currentGroup":0,"sceneValid":0,"nameSupport":0}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:15 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:info 4/29/2019, 8:50:16 PM MQTT publish: topic 'zigbee2mqtt/0x7cb03eaa00a6f7a3', payload '{"state":"ON","linkquality":84}'
  zigbee2mqtt:warn 4/29/2019, 8:50:16 PM No converter available for 'AB3257001NJ' with cid 'haElectricalMeasurement', type 'devChange' and data '{"cid":"haElectricalMeasurement","data":{"measurementType":8,"activePower":400}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:16 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:warn 4/29/2019, 8:50:16 PM No converter available for 'AB3257001NJ' with cid 'manuSpecificOsram', type 'devChange' and data '{"cid":"manuSpecificOsram","data":{"0":0}}'
  zigbee2mqtt:warn 4/29/2019, 8:50:16 PM Please see: https://www.zigbee2mqtt.io/how_tos/how_to_support_new_devices.html.
  zigbee2mqtt:info 4/29/2019, 8:50:16 PM Device incoming...
  zigbee2mqtt:info 4/29/2019, 8:50:16 PM MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"type":"pairing","message":"device incoming"}'
  zigbee2mqtt:warn 4/29/2019, 8:50:53 PM Failed to configure 0x7cb03eaa00a6f7a3 (0x7cb03eaa00a6f7a3) ('Error: Timed out after 10000 ms') (attempt #1)
  zigbee2mqtt:warn 4/29/2019, 8:50:53 PM This can be ignored if the device is working properly

But the OSRAM plug does not seem to work well with this firmware or software. State changes are not reported and there are these timed out error messages.