Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ANNOUNCEMENT] Zigbee-herdsman: the next zigbee-shepherd #1888

Closed
Koenkk opened this issue Aug 23, 2019 · 49 comments
Closed

[ANNOUNCEMENT] Zigbee-herdsman: the next zigbee-shepherd #1888

Koenkk opened this issue Aug 23, 2019 · 49 comments
Labels
stale Stale issues

Comments

@Koenkk
Copy link
Owner

Koenkk commented Aug 23, 2019

As some of you have already noticed, lately I've been putting lots of efforts in refactoring zigbee-shepherd. Zigbee-shepherd is the Javascript Zigbee library were zigbee2mqtt is based up-on.

My biggest reason to do this is that, in my opinion, zigbee-shepherd was not maintainable and extensible. This made it very hard to fix bugs and add new features, which made it hard to take zigbee2mqtt to the next level.

I ended up in completely rewriting zigbee-shepherd from scratch. Because of that I also gave it a different name, zigbee-herdsman. Just to give an idea of the effort that I've put into it, this job took me 100+ hours spread over 2 months.

Zigbee-herdsman will land into the zigbee2mqtt (dev branch) and zigbee-shepherd-converters after zigbee2mqtt 1.6. After this zigbee-shepherd-converters will be renamed to zigbee-herdsman-converters, as zigbee-shepherd-converters and zigbee-herdsman are not compatible.

What will I as a user notice from it?

  • Improved stability and predictability
  • Improved Xiaomi devices pairing (should be similar to the official Xiaomi gateway now, even for Mija devices, bye bye Cannot get node descriptor for... error 😺)
  • Improved logging, especially during pairing devices (earlier shows when a device has been connected and we can now show if a device has been paired successfully)
  • Improved error handling (e.g. unplugging the stick will stop zigbee2mqtt now)
  • (future) Possibility to support other adapters than TI Z-stack adapters
  • (future) Keep track of device bindings

What will I as a developer notice from it?

  • Written in Typescript; makes it easier to detect errors earlier in the development process -> less bugs
  • 100% test coverage; makes it easier to detect errors earlier in the development process -> less bugs
  • Reduced amount of dependencies (and especially not maintained dependencies)
  • Simplified Zigbee API -> adding devices to zigbee-shepherd-converters will be easier
  • Automatic iAS zone enrolment -> no need to do it in zigbee-shepherd-converters anymore -> easier to add support for iAS devices
  • Push down code from zigbee2mqtt into zigbee-herdsman (= less zigbee2mqtt code size and other programs based on zigbee-herdsman are able to benefit from it).
  • Drastically reduced and simplified zigbee-shepherd codebase:
    • From 17231 to 11273 lines (= 35%), but as a big part of the code are long definition files (example), it would be fair to count without these files, resulting in: From 7675 to 3739 lines (=52% code reduction)
@james-fry
Copy link

Nice!
Looking forward to trying 1.6.

@Koenkk
Copy link
Owner Author

Koenkk commented Aug 23, 2019

Nice!
Looking forward to trying 1.6.

Just to be clear, it will NOT be included in 1.6, but will be included in the dev branch after the 1.6 release. (we first need some time to test before it can be released).

@way2many
Copy link
Contributor

Great job!
Thank you a million times for all the hard work and thank you to everybody involved in this great project.

@santirguez
Copy link

That sounds amazing!!
Thank you so much for such a great project and work!

@runningman84
Copy link

What will be the optimal hardware (CC2530, ...) for the new version?

@bochelork
Copy link

Koen, Thanks for your great work. I still have a love-hate relationship with zigbee. I've quite a large network, about 37 devices, still growing. Stability is increasing because of your hard work. Less devices are getting lost. My network is expanding, 3 floors and a garden. Which never could have happened without zigbee2mqtt! Thanks thanks thanks!

@dirstel
Copy link

dirstel commented Aug 26, 2019

Improved Xiaomi devices pairing (should be similar to the official Xiaomi gateway now, even for Mija devices, bye bye Cannot get node descriptor for... error 😺

Can't wait to test this... I have more than 50 Devices, lots of them Xiaomi/Aqara. 👍 Herd is still growing and pairing Xiaomi has become a pain in the #$§!!

Maybe, a hint to Donation is useful. Come on Guys: Just think about the difference sending for any given device (f.e. hue vs tradfri)...

@bochelork
Copy link

Maybe, a hint to Donation is useful. Come on Guys: Just think about the difference sending for any given device (f.e. hue vs tradfri)...

That was on my list. Thanks for reminding.

@nashetali837
Copy link

nashetali837 commented Aug 26, 2019 via email

@Koenkk
Copy link
Owner Author

Koenkk commented Sep 9, 2019

zigbee-herdsman has now landed in the dev branch, let the testing begin :)

@santirguez
Copy link

Hi,

Is there any breaking change? I udated dev branch and now cannot start:
Sep 09 20:21:41 SmartHome systemd[1]: Started zigbee2mqtt.
Sep 09 20:21:42 SmartHome npm[7178]: > zigbee2mqtt@1.6.0 start /opt/zigbee2mqtt
Sep 09 20:21:42 SmartHome npm[7178]: > node index.js
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Logging to directory: '/opt/zigbee2mqtt/data/log/2019-09-09.20-21-42'
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Starting zigbee2mqtt version 1.6.0 (commit #d83085e)
Sep 09 20:21:43 SmartHome npm[7178]: zigbee2mqtt:info 9/9/2019, 8:21:43 PM Starting zigbee-herdsman...
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Error while starting zigbee-herdsman
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Error: Backup is only supported for Z-Stack 3
Sep 09 20:21:44 SmartHome npm[7178]: at Object. (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/backup.js:78:19)
Sep 09 20:21:44 SmartHome npm[7178]: at Generator.next ()
Sep 09 20:21:44 SmartHome npm[7178]: at fulfilled (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/backup.js:4:58)
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Failed to start zigbee
Sep 09 20:21:44 SmartHome npm[7178]: zigbee2mqtt:error 9/9/2019, 8:21:44 PM Exiting...
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! code ELIFECYCLE
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! errno 1
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! zigbee2mqtt@1.6.0 start: node index.js
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! Exit status 1
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR!
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! Failed at the zigbee2mqtt@1.6.0 start script.
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! A complete log of this run can be found in:
Sep 09 20:21:44 SmartHome npm[7178]: npm ERR! /home/pi/.npm/_logs/2019-09-09T18_21_44_648Z-debug.log
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Failed with result 'exit-code'.
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Service RestartSec=100ms expired, scheduling restart.
Sep 09 20:21:44 SmartHome systemd[1]: zigbee2mqtt.service: Scheduled restart job, restart counter is at 2.

@Koenkk
Copy link
Owner Author

Koenkk commented Sep 9, 2019

Can you provide the log when running with DEBUG=* npm start? (please post it on pastebin)

@BabyDino
Copy link

BabyDino commented Sep 9, 2019

I have some issues with device locking. I'm not sure if it's my setup or not. I have to physically disconnect my USB device and reconnect it before I can restart zigbee2mqtt.

https://pastebin.com/xpPsiJit

@BabyDino
Copy link

BabyDino commented Sep 9, 2019

And it looks like I've just pastebinned my network key...

@r1cht3r
Copy link
Contributor

r1cht3r commented Sep 15, 2019

Latest-dev is working GREAT for me here, and I'm awe-struck by the speed. Well done! I'm struggling with joining some devices (Cree Connected bulbs, which worked before), and if I can't get that worked out I'll file a separate issue. I have some debug log spelunking to finish before I do.

@wixoff @Koenkk I can confirm this behavior with CREE Connected bulbs. So long as they are already joined they work fine. I had a couple to add to the network, here's what I did:

  • revert to latest docker image
  • join new CREE bulbs like normal
  • change to latest-dev with existing database.db
  • new CREE bulbs work as expected

@bochelork
Copy link

@r1cht3r same here with GledOpto bulbs. Pairing works fine in 1.5, moving database to 1.6 gives me a full network for a while but loosing contact with some of the bulbs after a few minutes and no way to get it paired again. Last night I ran with 1.5, changing color each 5 minutes and all Gledoptos are still alive.
I changed my scripts to prevent bursting, because this is definately causing problems. (max 1 publish per second now).
@Koenkk which data do you need to see what's wrong, I can same db-files from different stages. Running 1.5, after starting 1.6, after lost contact on 1.6, all logs files?

@Koenkk
Copy link
Owner Author

Koenkk commented Sep 16, 2019

@bochelork Can you provide the log when running with DEBUG=* npm start when pairing the device? Please provide the log on pastebin.com and link it here.

@bochelork
Copy link

@Koenkk With a working 1.5 database I started with latest dev. After starting I switched off 7 devices with zibee2mqtt with 0.5 seconds interval.

https://pastebin.com/JKd3a9GZ

Here I switch off the 230v power of 6 devices, I switch on after 10 seconds. This should announce the devices.

https://pastebin.com/TmdUkfu4

Switching off by zigbee2mqtt 7 devices again. Only 2 respond.

https://pastebin.com/PSvqdUCi

Reset the bulbs. 5 times 230v power on/off

https://pastebin.com/4HPqieZv

Again switch off by zigbeemqtt, only 1 responds.

https://pastebin.com/wASVKnA3

After this I switched back to 1.5, 5 out of 7 device respond normally. Have to repair the others....

@Koenkk
Copy link
Owner Author

Koenkk commented Sep 17, 2019

@bochelork can you try with: https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/Z-Stack_Home_1.2/bin/source_routing? Before executing your tests make sure to turn each device on/off via zigbee2mqtt once. This ensures that the coordinator knows the routes.

@lolorc
Copy link
Contributor

lolorc commented Sep 29, 2019

hi there,

giving a try to herdsman/dev, looking very good so far, where are the devices stored ? in states.json ?
because I've restarted the daemon and I had to re-pair everything again.

edit: I think they're stored in states.json, in my z2m start script I used to rm states.json because of #2033. Could you confirm this ? :-)

@Koenkk
Copy link
Owner Author

Koenkk commented Sep 29, 2019

Only states are stored there, devices are stored in database.db

@Imperial-Guard
Copy link

I switched over to try the dev version with "herdsman" but somehow I lost a few devices, they should be connected but don't give any signal and repairing doesn't seem to work.

Any thoughts how I can help to give you more additional info to debug this?

The devices what I'm trying to repair are Ikea E1746 Signal Repeater, Xiaomi WSDCGQ11LM and a Xiaomi MCCGQ01LM

@hdo
Copy link

hdo commented Oct 22, 2019

Update: SOLVED

It seems pairing works best with full batteries :-)

The sensor was sending happily with the old battery but pairing was not possible.


I'm also having problems repairing Xiaomi Temperature Sensor (Xiaomi WSDCGQ11LM) :-(

@erazor666
Copy link

I'm thinking of rebuilding my z2m setup (redoing Home Assistant and the works). How far along is the next release?
Is the dev any good for daily use?
(I have only Xiaomi sensors at the moment)

@L3H0
Copy link

L3H0 commented Oct 28, 2019

@erazor666 i have using it about 2-3 weeks and it is rock solid. Try it and you will not go back to the original shephard.

@wixoff
Copy link
Contributor

wixoff commented Oct 29, 2019

Yes, I have finally eliminated ALL dropped devices using a combination of the latest-dev (post-1.6.0) z2m with zigbee-herdsman and the Zigbee 3.0.x firmware on a CC2531.

As @L3H0 said, it's ROCK SOLID. And much faster!

@Emma-2409
Copy link

Hi everyone,
I am having trouble when pairing a Sylvania device LIGHTIFY LED RGBW A19, immediately after pairing, it shows an Error "Cluster with key '64513' does not exist" and stops the zigbee2mqtt service.

I am new to this whole zigbee, mqtt, and linux enviroment... has anyone experimented with the same error?

@Koenkk
Copy link
Owner Author

Koenkk commented Nov 1, 2019

@Emma-2409 should be fixed in the zigbee2mqtt dev branch (https://www.zigbee2mqtt.io/how_tos/how-to-switch-to-dev-branch.html)

@Emma-2409
Copy link

@Emma-2409 should be fixed in the zigbee2mqtt dev branch (https://www.zigbee2mqtt.io/how_tos/how-to-switch-to-dev-branch.html)

Thanks Koenkk, it worked just fine, you have created something wonderful I must say ;)

@stale
Copy link

stale bot commented Dec 31, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Stale issues label Dec 31, 2019
@electrofloat
Copy link

anti-stale

@stale stale bot removed the stale Stale issues label Dec 31, 2019
@stale
Copy link

stale bot commented Feb 29, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Stale issues label Feb 29, 2020
@stale stale bot closed this as completed Mar 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale Stale issues
Projects
None yet
Development

No branches or pull requests