Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ANNOUNCEMENT] Zigbee-herdsman: the next zigbee-shepherd #1888

Open
Koenkk opened this issue Aug 23, 2019 · 36 comments

Comments

@Koenkk
Copy link
Owner

commented Aug 23, 2019

As some of you have already noticed, lately I've been putting lots of efforts in refactoring zigbee-shepherd. Zigbee-shepherd is the Javascript Zigbee library were zigbee2mqtt is based up-on.

My biggest reason to do this is that, in my opinion, zigbee-shepherd was not maintainable and extensible. This made it very hard to fix bugs and add new features, which made it hard to take zigbee2mqtt to the next level.

I ended up in completely rewriting zigbee-shepherd from scratch. Because of that I also gave it a different name, zigbee-herdsman. Just to give an idea of the effort that I've put into it, this job took me 100+ hours spread over 2 months.

Zigbee-herdsman will land into the zigbee2mqtt (dev branch) and zigbee-shepherd-converters after zigbee2mqtt 1.6. After this zigbee-shepherd-converters will be renamed to zigbee-herdsman-converters, as zigbee-shepherd-converters and zigbee-herdsman are not compatible.

What will I as a user notice from it?

  • Improved stability and predictability
  • Improved Xiaomi devices pairing (should be similar to the official Xiaomi gateway now, even for Mija devices, bye bye Cannot get node descriptor for... error 😺)
  • Improved logging, especially during pairing devices (earlier shows when a device has been connected and we can now show if a device has been paired successfully)
  • Improved error handling (e.g. unplugging the stick will stop zigbee2mqtt now)
  • (future) Possibility to support other adapters than TI Z-stack adapters
  • (future) Keep track of device bindings

What will I as a developer notice from it?

  • Written in Typescript; makes it easier to detect errors earlier in the development process -> less bugs
  • 100% test coverage; makes it easier to detect errors earlier in the development process -> less bugs
  • Reduced amount of dependencies (and especially not maintained dependencies)
  • Simplified Zigbee API -> adding devices to zigbee-shepherd-converters will be easier
  • Automatic iAS zone enrolment -> no need to do it in zigbee-shepherd-converters anymore -> easier to add support for iAS devices
  • Push down code from zigbee2mqtt into zigbee-herdsman (= less zigbee2mqtt code size and other programs based on zigbee-herdsman are able to benefit from it).
  • Drastically reduced and simplified zigbee-shepherd codebase:
    • From 17231 to 11273 lines (= 35%), but as a big part of the code are long definition files (example), it would be fair to count without these files, resulting in: From 7675 to 3739 lines (=52% code reduction)
@james-fry

This comment has been minimized.

Copy link

commented Aug 23, 2019

Nice!
Looking forward to trying 1.6.

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Aug 23, 2019

Nice!
Looking forward to trying 1.6.

Just to be clear, it will NOT be included in 1.6, but will be included in the dev branch after the 1.6 release. (we first need some time to test before it can be released).

@way2many

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2019

Great job!
Thank you a million times for all the hard work and thank you to everybody involved in this great project.

@santirguez

This comment has been minimized.

Copy link

commented Aug 26, 2019

That sounds amazing!!
Thank you so much for such a great project and work!

@runningman84

This comment has been minimized.

Copy link

commented Aug 26, 2019

What will be the optimal hardware (CC2530, ...) for the new version?

@bochelork

This comment has been minimized.

Copy link

commented Aug 26, 2019

Koen, Thanks for your great work. I still have a love-hate relationship with zigbee. I've quite a large network, about 37 devices, still growing. Stability is increasing because of your hard work. Less devices are getting lost. My network is expanding, 3 floors and a garden. Which never could have happened without zigbee2mqtt! Thanks thanks thanks!

@dirstel

This comment has been minimized.

Copy link

commented Aug 26, 2019

Improved Xiaomi devices pairing (should be similar to the official Xiaomi gateway now, even for Mija devices, bye bye Cannot get node descriptor for... error 😺

Can't wait to test this... I have more than 50 Devices, lots of them Xiaomi/Aqara. 👍 Herd is still growing and pairing Xiaomi has become a pain in the #$§!!

Maybe, a hint to Donation is useful. Come on Guys: Just think about the difference sending for any given device (f.e. hue vs tradfri)...

@bochelork

This comment has been minimized.

Copy link

commented Aug 26, 2019

Maybe, a hint to Donation is useful. Come on Guys: Just think about the difference sending for any given device (f.e. hue vs tradfri)...

That was on my list. Thanks for reminding.

@nashetali837

This comment has been minimized.

Copy link

commented Aug 26, 2019

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2019

zigbee-herdsman has now landed in the dev branch, let the testing begin :)

@santirguez

This comment has been minimized.

Copy link

commented Sep 9, 2019

Hi,

Is there any breaking change? I udated dev branch and now cannot start:
Sep 09 20:21:41 SmartHome systemd[1]: Started zigbee2mqtt.
Sep 09 20:21:42 SmartHome npm[7178]: > zigbee2mqtt@1.6.0 start /opt/zigbee2mqtt
Sep 09 20:21:42 SmartHome npm[7178]: > node index.js
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Logging to directory: '/opt/zigbee2mqtt/data/log/2019-09-09.20-21-42'
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Starting zigbee2mqtt version 1.6.0 (commit #d83085e)
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Starting zigbee-herdsman...
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Error while starting zigbee-herdsman
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Error: Backup is only supported for Z-Stack 3
Sep 09 20:21:44 SmartHome npm[7178]: at Object. (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/backup.js:78:19)
Sep 09 20:21:44 SmartHome npm[7178]: at Generator.next ()
Sep 09 20:21:44 SmartHome npm[7178]: at fulfilled (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/backup.js:4:58)
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Failed to start zigbee
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Exiting...
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! code ELIFECYCLE
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! errno 1
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! zigbee2mqtt@1.6.0 start: node index.js
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! Exit status 1
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR!
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! Failed at the zigbee2mqtt@1.6.0 start script.
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! A complete log of this run can be found in:
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! /home/pi/.npm/_logs/2019-09-09T18_21_44_648Z-debug.log
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Failed with result 'exit-code'.
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Service RestartSec=100ms expired, scheduling restart.
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Scheduled restart job, restart counter is at 2.

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2019

Can you provide the log when running with DEBUG=* npm start? (please post it on pastebin)

@BabyDino

This comment has been minimized.

Copy link

commented Sep 9, 2019

I have some issues with device locking. I'm not sure if it's my setup or not. I have to physically disconnect my USB device and reconnect it before I can restart zigbee2mqtt.

https://pastebin.com/xpPsiJit

@BabyDino

This comment has been minimized.

Copy link

commented Sep 9, 2019

And it looks like I've just pastebinned my network key...

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2019

@BabyDino can you check which processes have the device open? (lsof /dev/ttyUSB2)

@santirguez

This comment has been minimized.

Copy link

commented Sep 9, 2019

Can you provide the log when running with DEBUG=* npm start? (please post it on pastebin)

Sure, here it is: https://pastebin.com/CUF7MBMZ

@BabyDino

This comment has been minimized.

Copy link

commented Sep 9, 2019

@Koenkk Next time it happens I will post the data.

Another issue with an unhandled promise rejection: https://pastebin.com/S6bj9VY5

Koenkk added a commit to Koenkk/zigbee-herdsman that referenced this issue Sep 9, 2019
Koenkk added a commit that referenced this issue Sep 9, 2019
@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2019

@santirguez issue should be fixed in the latest dev branch (make sure to do a rm -rf node_modules && npm install after updating).

@BabyDino I see that you are also at least suffering from the same issue as @santirguez, (Error: Backup is only supported for Z-Stack 3). Should be fixed in latest dev, can you check if it also fixed the other issue. Regarding the resource busy, it probably happens because zigbee2mqtt fails to start (because of the issue) and therefore keeping the port locked. So that should fix itself automatically when we can get zigbee2mqtt to start :)

@way2many

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2019

9/9/2019, 7:38:57 PM - debug: Loaded state from file /app/data/state.json
9/9/2019, 7:38:57 PM - debug: Saving state to file /app/data/state.json
9/9/2019, 7:38:57 PM - info: Starting zigbee2mqtt version 1.6.0 (commit #d83085e)
9/9/2019, 7:38:57 PM - info: Starting zigbee-herdsman...
9/9/2019, 7:38:57 PM - debug: Using zigbee-herdsman with settings: '{"network":{"panID":6754,"extenedPanID":[221,221,221,221,221,221,221,221],"channelList":[11],"networkKey":"HIDDEN"},"databasePath":"/app/data/database.db","backupPath":"/app/data/coordinator_backup.json","serialPort":{"baudRate":115200,"rtscts":true,"path":"/dev/ttyACM0"}}'
9/9/2019, 7:38:59 PM - error: Error while starting zigbee-herdsman

Cannot get more from debug log. Running latest docker container on RPi 3.

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2019

@way2many probably same issue as @santirguez, please update to the latest dev (you should see commit #411f1fa). If you are on docker, wait until https://travis-ci.org/Koenkk/zigbee2mqtt/builds/582844293?utm_source=github_status&utm_medium=notification has been finished. (max 30 minutes from now).

EDIT: docker image is ready now.

@BabyDino

This comment has been minimized.

Copy link

commented Sep 9, 2019

@Koenkk lsof /dev/ttyUSB2:

node    17737   pi   22uW  CHR  188,2      0t0 17006905 /dev/ttyUSB2

/edit: after killing the process the error remains.

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2019

@BabyDino I think you also need to unplug the usb, please see my explanation from #1888 (comment)

@way2many

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2019

Updated to latest docker container and everything is working fine. Thanks @Koenkk

Startup was blazing fast!

@BabyDino

This comment has been minimized.

Copy link

commented Sep 9, 2019

@Koenkk Makes sense. It's running now. I have a lot of Xiaomi stuff and still some pairings to do. Do you want individual issues created in the zigbee-hersman repo or report in this issue for now?

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 9, 2019

@BabyDino yes please create seperate issues in the zigbee2mqtt repo

@BabyDino

This comment has been minimized.

Copy link

commented Sep 9, 2019

@Koenkk I'm not sure what you have done, but this is lightning fast!! I have 2 spots in the hallway and they are turning on simultaneously without delay (instead of one before the other). Everything reacts real-time to commands. Will test stability in the coming days. A IKEA Tradfri lamp just paired flawlessly (I couldn't get it paired on -shepherd).

Thank you for your hard work!

@bochelork

This comment has been minimized.

Copy link

commented Sep 9, 2019

Thanks @Koenkk! First impression is great. It's fast, some devices were impossible to pair and now it's done in an eye wink. Thanks!

@didiht

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2019

Great work @Koenkk
After update I had to unplug the stick to get zigbee-herdsman started, but then the sytem was up for normal operationing very fast. It is very stable, even without a router (which was necessary before)
Thanks!

@wixoff

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2019

Latest-dev is working GREAT for me here, and I'm awe-struck by the speed. Well done! I'm struggling with joining some devices (Cree Connected bulbs, which worked before), and if I can't get that worked out I'll file a separate issue. I have some debug log spelunking to finish before I do.

@santirguez

This comment has been minimized.

Copy link

commented Sep 11, 2019

@santirguez issue should be fixed in the latest dev branch (make sure to do a rm -rf node_modules && npm install after updating).

Hi @Koenkk. It was working perfectly! Now after a couple of day I lost around 4 Xiaomi movement sensors (not reporting state in zigbee2mqtt)
I am trying to remove them to repair them again, but every time I try to remove a sensro I get this error:
PM Failed to remove "XXXXX" (Error: AREQ - ZDO - mgmtLeaveRsp after 10000ms)

Is there any way to remove them without loose everything else?

Thank you for your work!!
Best regards,

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 11, 2019

This makes sense as the device is sleeping its not responding to requests. If you hold the reset button on the device it will remove itself. After that you can repair.

@r1cht3r

This comment has been minimized.

Copy link

commented Sep 15, 2019

Latest-dev is working GREAT for me here, and I'm awe-struck by the speed. Well done! I'm struggling with joining some devices (Cree Connected bulbs, which worked before), and if I can't get that worked out I'll file a separate issue. I have some debug log spelunking to finish before I do.

@wixoff @Koenkk I can confirm this behavior with CREE Connected bulbs. So long as they are already joined they work fine. I had a couple to add to the network, here's what I did:

  • revert to latest docker image
  • join new CREE bulbs like normal
  • change to latest-dev with existing database.db
  • new CREE bulbs work as expected
@bochelork

This comment has been minimized.

Copy link

commented Sep 16, 2019

@r1cht3r same here with GledOpto bulbs. Pairing works fine in 1.5, moving database to 1.6 gives me a full network for a while but loosing contact with some of the bulbs after a few minutes and no way to get it paired again. Last night I ran with 1.5, changing color each 5 minutes and all Gledoptos are still alive.
I changed my scripts to prevent bursting, because this is definately causing problems. (max 1 publish per second now).
@Koenkk which data do you need to see what's wrong, I can same db-files from different stages. Running 1.5, after starting 1.6, after lost contact on 1.6, all logs files?

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 16, 2019

@bochelork Can you provide the log when running with DEBUG=* npm start when pairing the device? Please provide the log on pastebin.com and link it here.

@bochelork

This comment has been minimized.

Copy link

commented Sep 16, 2019

@Koenkk With a working 1.5 database I started with latest dev. After starting I switched off 7 devices with zibee2mqtt with 0.5 seconds interval.

https://pastebin.com/JKd3a9GZ

Here I switch off the 230v power of 6 devices, I switch on after 10 seconds. This should announce the devices.

https://pastebin.com/TmdUkfu4

Switching off by zigbee2mqtt 7 devices again. Only 2 respond.

https://pastebin.com/PSvqdUCi

Reset the bulbs. 5 times 230v power on/off

https://pastebin.com/4HPqieZv

Again switch off by zigbeemqtt, only 1 responds.

https://pastebin.com/wASVKnA3

After this I switched back to 1.5, 5 out of 7 device respond normally. Have to repair the others....

@Koenkk

This comment has been minimized.

Copy link
Owner Author

commented Sep 17, 2019

@bochelork can you try with: https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/Z-Stack_Home_1.2/bin/source_routing? Before executing your tests make sure to turn each device on/off via zigbee2mqtt once. This ensures that the coordinator knows the routes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.