[SUGESTION] - Documentation on how to use the library #452
Comments
|
Zigpy doesn't depend on HA in any way (other than HA being the biggest project directly utilizing it) so you can definitely use it standalone with the appropriate radio library. Here's an (untested) example: import asyncio
# There are many different radio libraries but they all have the same API
from zigpy_znp.zigbee.application import ControllerApplication
class MainListener:
"""
Contains callbacks that zigpy will call whenever something happens.
Look for `listener_event` in the Zigpy source or just look at the logged warnings.
"""
def __init__(self, application):
self.application = application
def device_joined(self, device):
print(f"Device joined: {device}")
def attribute_updated(self, device, cluster, attribute_id, value):
print(f"Received an attribute update {attribute_id}={value}"
f" on cluster {cluster} from device {device}")
async def main():
app = ControllerApplication(ControllerApplication.SCHEMA({
"database_path": "/path/to/zigbee.db",
"device": {
"path": "/dev/serial/by-id/your-serial-port",
}
}))
listener = MainListener(app)
app.add_listener(listener)
await app.startup(auto_form=True)
# Permit joins for a minute
await app.permit(60)
await asyncio.sleep(60)
# Just run forever
await asyncio.get_running_loop().create_future()
if __name__ == "__main__":
await asyncio.run(main()) |
|
thanks that will be very useful. What about sending a command to a device ? Like On/Off on Cluster 0x0006 or a LevelCOntrol on Cluster 0x0008 N |
|
You would do something like Some places to look:
The |
|
@puddly thanks for all those informations. I'm making progress, as I have been able to get connected to the HW ( RPI Zigate ), and get devices paired. I have those error messages which seems coming when the device is sending messages ( Xiaomi Motion Sensor ) As the Database is concerned, do I have to create it first ? Or will the library will create at the first occasion ? After pairing devices, I still didn't get it ! I also got that message: More debug information available: |
|
Ok going from errors to errors .... Here is my setup: python3.7.7 When switching to dev branch of zigpy-zigate and zigpy on master branche I'm getting
When switching to dev branch of zigpy-zigate and zigpy on dev branche I'm getting an other error |
|
One more comment, I wanted to use the Quirks library but this looks ZHA specific, isn't ? For instance, trying to pair a Legrand Dimer (here are the logs ), I beleive the quirks for the Device was'nt found ! `` |
|
It's not. Depending on what are you trying to do you may not need it. |
Where , how should I install it ? Ultimate goal is got a plugin for Domoticz |
|
Thanks I have it, but still I do not see the library taking it , this is why I was asking if there are any specific config/setup to be done when using it outside of ZHA! And one remark, in the suggested link here is the statement:
|
|
you need to install it and you need to |
|
As @sanyatuning already mentioned to @iftahgi in zigpy/zigpy-cc#61 Be sure to check out the ZHA integration component code by @dmulcahey in Home Assistant at https://github.com/home-assistant/core/tree/dev/homeassistant/components/zha as its code is currently the best reference application for how-to utilize zigpy and its radio libraries inside a other projects/application. @sanyatuning also has a tip to checkout https://github.com/zigpy/zigpy-cc/blob/master/test.py as he uses that to test zigpy-cc without Home Assistant (check git history for this test.py code file in the zigpy-cc project). Also, if I understand it correctly, the zigpy library was originally a refactored fork from the bellows library as it was split into two to separate duties, however even though the bellows was refactored to use zigpy, the parts moved to zigpy is still left in bellows as a legacy. Therefore a tip can be to in addition look for non-HA standalone application projects are still using bellows without zigpy? As unlike the other radio libraries for zigpy, bellows can still (or at least) could be used without the zigpy library. As well as look for non-HA standalone application projects using zigpy of course. I believe that those non-HA standalone application projects include but are might not be limited to ZigCoHTTP by @daniel17903, qzig by @AndreasBomholtz , and the hue-thief by @vanviegen (where ZigCoHTTP by @daniel17903 look to be the most recently updated): https://github.com/daniel17903/ZigCoHTTP https://github.com/Seluxit/qzig https://github.com/vanviegen/hue-thief Again there might also be other non-HA standalone application projects out there as well that use zigpy or bellows as a library. Publicly we can only see in the "used by" dependency graphs those public project that is listed on GitHub as using zigpy or bellows as dependencies: https://github.com/zigpy/bellows/network/dependents?package_id=UGFja2FnZS01MDI3Mjc2Mg%3D%3D https://github.com/zigpy/zigpy/network/dependents?package_id=UGFja2FnZS01MjcyNjUwNg%3D%3D @pvizeli Again FYI, a very similar question was recently posted by @iftahgi here -> zigpy/zigpy-cc#61 so perhaps it might be worth you talking and brainstorming with @iftahgi if he now happens to be working on something similar. |
@pipiche38 Tip on the hardware-side is that you might want to consider buying a Silicon Labs Zigbee adapter like example the latest Elelabs Zigbee USB adapter or probably better today is the new Sonoff ZBBridge with Tasmota (for use with bellows) as an addition to also buying a newer Texas Instruments like the LAUNCHXL-CC26X2R1 dev board or the slaesh CC2652RB stick adapters. Reason for this is that I know ZiGate support is experimental and has only ever had one developer working on it, while there have over the years been more developers working on the bellows library for Silabs support and zigpy-cc + zigpy-znp libraries for TI support. Currently, there might in release be more than one developer each working actively on those as well but there are at least many more developers out there with those two types of hardware than there are those with ZiGate hardware. I think that it awesome that zigpy support multiple adapters from different manufacturers, but if you are going develop a brand new standalone implementation of zigpy then you probably want to be using the same type of hardware as most active developers are. |
For now I'm the developper of the Domoticz plugin for Zigate. I do consider having a zigpy plugin for Domoticz. |
|
@pipiche38 Here's an example that actually works with a Xiaomi motion sensor. Make sure to poke the pairing button every second or so to keep the sensor awake while it is joining. import asyncio
import logging
import coloredlogs
coloredlogs.install(milliseconds=True, level=logging.DEBUG)
import zhaquirks
# There are many different radio libraries but they all have the same API
from zigpy_znp.zigbee.application import ControllerApplication
LOGGER = logging.getLogger(__name__)
class MainListener:
"""
Contains callbacks that zigpy will call whenever something happens.
Look for `listener_event` in the Zigpy source or just look at the logged warnings.
"""
def __init__(self, application):
self.application = application
def device_initialized(self, device, *, new=True):
"""
Called at runtime after a device's information has been queried.
I also call it on startup to load existing devices from the DB.
"""
LOGGER.info("Device is ready: new=%s, device=%s", new, device)
for ep_id, endpoint in device.endpoints.items():
# Ignore ZDO
if ep_id == 0:
continue
# You need to attach a listener to every cluster to receive events
for cluster in endpoint.in_clusters.values():
# The context listener passes its own object as the first argument
# to the callback
cluster.add_context_listener(self)
def attribute_updated(self, cluster, attribute_id, value):
# Each object is linked to its parent (i.e. app > device > endpoint > cluster)
device = cluster.endpoint.device
LOGGER.info("Received an attribute update %s=%s on cluster %s from device %s",
attribute_id, value, cluster, device)
async def main():
app = await ControllerApplication.new(
config=ControllerApplication.SCHEMA({
"database_path": "/tmp/zigbee.db",
"device": {
"path": "/dev/cu.usbmodemL1100H861",
}
}),
auto_form=True,
start_radio=True,
)
listener = MainListener(app)
app.add_listener(listener)
# Have every device in the database fire the same event so you can attach listeners
for device in app.devices.values():
listener.device_initialized(device, new=False)
# Permit joins for a minute
await app.permit(60)
await asyncio.sleep(60)
# Run forever
await asyncio.get_running_loop().create_future()
if __name__ == "__main__":
asyncio.run(main())@Adminiuga Do you think there's a benefit to hard-coding the node descriptor responses into quirks based solely on the device model name, like the Xiaomi gateway does? It'd make the joining process very fast for battery-powered Xiaomi devices that are known to have only one firmware version available. |
|
hrm, this would require overriding |
|
Can I suggested to all (zigpy) developers to also start using some kind of inline code documentation in the source code files? That way the bulk of any code documentation does not have to be maintained separately (which would normally quickly be outdated) and can instead be generated live at any time from the latest code three using documentation generator tools. You would really only have to agree to follow a standard for inline code documentation., which should not be too hard as Doxygen is the de facto standard and do support Python. Doxygen, is a documentation system that supports most programming languages today. It can generate html docs documenting a projects source code, by either extracting special tags from the source code (put there by people wanting to make use of doxygen), or doxygen will otherwise attempt to build the documentation from the existing source code. https://www.doxygen.nl/index.html CodeDocs will then generate and publish documentation from Doxygen for you, (free to setup with public GitHub repositories). For example, you can check out the Kodi code documentation generated on codedocs.xyz https://codedocs.xyz/AlwinEsch/kodi/ There are of course many different standards for inline code documentation and also many documentation generation tools, ex: |
Maybe scripts like that should be added to the zigpy repo as "examples" for new applications independent from Home Assistant? That is, create a new directory for "examples" and put scripts like that there? ...and maybe also a separate new directory for "docs"? |
|
Making progress, but now facing abig issue with Domoticz integration. Getting a Segmentation fault The interesting thing, is that Creation of the DB works fine, so the execute method is correctly executed with the CREATE Table statments, but as soon as we have the SELECT statement, it crashs |
|
What platform and Python distribution are you running? You shouldn't be getting segfaults in a pure Python project, especially in the sqlite module. |
I don't expect that the issue is on the zigpy / sqlite3 side. Domoticz is not as HA. In that context and to coexist with the embedded python framework, I have to do the zigpy part in a dedicated thread (which is not a problem) From my current investigation the segmentation fault occurred on the execute() method and only when doing the SELECT * from devices statements and other tables as well. But the Table creation and Index creation works without any issues. To answer your question on platform:
|
|
@puddly , I'm making progress. I have been able to duplicate the Domoticz / plugin issue with the appdb part, by extracting only the sqlite3 related class/method. If Domoticz guru's troubleshoot and fix the issue, I think that I'm on the good path to develop a Domoticz plugin while using zigpy library. |
|
That's excellent news. I don't have much experience with embedded Python but one possible thing to check is making sure the version of SQLite that Python might be dynamically linked against is compatible with what you have installed. |
Found the zigpy statement causing the segmentation violation. Line 118 in appdb.py I'll be doing more investigations during the day, PS/ The code is fully compiled on my system, so is using the right shared libs |
…rojects Update README.md based on feedback regarding integration into other projects/applications that is not ZHA in Home Assistant, see: zigpy/zigpy#452 General deedback is that missing documentation on how to use this and the zigpy library.
|
@pipiche38 FYI, see #459 as a start. |
Yep, I saw it and that is great move. |
|
@pipiche38 Also checkout new PR #460 by @puddly |
|
Just a short update, for now I'm stuck as the appdb.py implementation hangs with the embedded python framework in Domoticz. The funny things is that it creates the Database (all CREATE, INDEXES and PRAGMA statements are correctly executed), but at the first SELECT statement it crashes in the sqlite3.so with a segmentation violation. I suspect some conflict between the embedded Python and the Domoticz which are both doing SQLITE. I'm thinking to migrate the zigpy/appdb.py from sqlite3 to dict with json. If by chance anyone has already done some work on that matter I would be interested. |
|
@pipiche38 This might be helpful. It's not feature-complete or particularly good code but it implements enough stuff to work for standalone zigpy applications: import json
import pathlib
import zigpy.types
from zigpy.zdo import types as zdo_t
class JSONPersistingListener:
def __init__(self, database_file, application):
self._database_file = pathlib.Path(database_file)
self._application = application
self._db = {'devices': {}}
def device_joined(self, device):
self.raw_device_initialized(device)
def device_initialized(self, device):
# This is passed in a quirks device
pass
def device_left(self, device):
self._db['devices'].pop(str(device.ieee))
self._dump()
def raw_device_initialized(self, device):
self._db['devices'][str(device.ieee)] = self._serialize_device(device)
self._dump()
def device_removed(self, device):
self._db['devices'].pop(str(device.ieee))
self._dump()
def attribute_updated(self, cluster, attrid, value):
self._dump()
def _serialize_device(self, device):
obj = {
'ieee': str(device.ieee),
'nwk': device.nwk,
'status': device.status,
'node_descriptor': None if not device.node_desc.is_valid else list(device.node_desc.serialize()),
'endpoints': [],
}
for endpoint_id, endpoint in device.endpoints.items():
if endpoint_id == 0:
continue # Skip zdo
endpoint_obj = {}
endpoint_obj['id'] = endpoint_id
endpoint_obj['status'] = endpoint.status
endpoint_obj['device_type'] = getattr(endpoint, 'device_type', None)
endpoint_obj['profile_id'] = getattr(endpoint, 'profile_id', None)
endpoint_obj['output_clusters'] = [cluster.cluster_id for cluster in endpoint.out_clusters.values()]
endpoint_obj['input_clusters'] = [cluster.cluster_id for cluster in endpoint.in_clusters.values()]
obj['endpoints'].append(endpoint_obj)
return obj
def _dump(self):
devices = []
for device in self._application.devices.values():
devices.append(self._serialize_device(device))
existing = self._database_file.read_text()
new = json.dumps({'devices': devices}, separators=(', ', ': '), indent=4)
# Don't waste writes
if existing == new:
return
self._database_file.write_text(new)
def load(self):
try:
state_obj = json.loads(self._database_file.read_text())
except FileNotFoundError:
logger.warning('Database is empty! Creating an empty one...')
self._database_file.write_text('')
state_obj = {'devices': []}
for obj in state_obj['devices']:
ieee = zigpy.types.named.EUI64([zigpy.types.uint8_t(int(c, 16)) for c in obj['ieee'].split(':')][::-1])
assert obj['ieee'] in repr(ieee)
device = self._application.add_device(ieee, obj['nwk'])
device.status = zigpy.device.Status(obj['status'])
if 'node_descriptor' in obj and obj['node_descriptor'] is not None:
device.node_desc = zdo_t.NodeDescriptor.deserialize(bytearray(obj['node_descriptor']))[0]
for endpoint_obj in obj['endpoints']:
endpoint = device.add_endpoint(endpoint_obj['id'])
endpoint.profile_id = endpoint_obj['profile_id']
device_type = endpoint_obj['device_type']
try:
if endpoint.profile_id == 260:
device_type = zigpy.profiles.zha.DeviceType(device_type)
elif endpoint.profile_id == 49246:
device_type = zigpy.profiles.zll.DeviceType(device_type)
except ValueError:
pass
endpoint.device_type = device_type
endpoint.status = zigpy.endpoint.Status(endpoint_obj['status'])
for output_cluster in endpoint_obj['output_clusters']:
endpoint.add_output_cluster(output_cluster)
for input_cluster in endpoint_obj['input_clusters']:
cluster = endpoint.add_input_cluster(input_cluster)
# Use this in place of your radio's ControllerApplication
class JSONControllerApplication(ControllerApplication):
def __init__(self, config):
super().__init__(self.SCHEMA(config))
# Replace the internal SQLite DB listener with our own
self._dblistener = JSONPersistingListener(self.config['json_database_path'], self)
self.add_listener(self._dblistener)
self._dblistener.load()
self._dblistener._dump()Get rid of the {
"devices": [
{
"ieee": "00:15:8d:00:01:b7:b0:42",
"nwk": 22045,
"status": 2,
"node_descriptor": [
2,
64,
128,
55,
16,
127,
100,
0,
0,
0,
100,
0,
0
],
"endpoints": [
{
"id": 1,
"status": 1,
"device_type": 24321,
"profile_id": 260,
"output_clusters": [
0,
4,
6
],
"input_clusters": [
0,
3
]
}
]
}
]
} |
|
@puddly, you save me a lot of time and you made my day Device is paired under Domoticz via ... It so confirmed :
This sound pretty great. Questions: 1/ Can we have the JSONPersistingListener supported in the zigpy stack. We can imagine that at a point of time we can select JSON or SQLITE3 as the persistent storage mode ? 2/ This question is more related to Zigbee devices. After almost 3 years in the Zigbee world developing the Zigate plugin for Domoticz, I saw quiet a large number of devices not fully standard. Even if they are ZB3.0 compliant they do not behave as the norm expect. Xiaomi is one brand, Legrand , Livolo, Konke, Tuya are other examples do you plan to support and manage all specifics behavours ? Where to so see the border between what is done on the zigpy stack and the quirk libraries and what is expected to be done on the above layer (like a gateway application would do ) ? The first device paired with Domoticz <-> zigpy <-> zigpy-zigate <-> zigate |
@pipiche38 Do you think that a Domoticz developer can now be convinced to fix SQLite issues inside Domoticz so it can be used?
@dmulcahey I understand this is what quirks / device-handlers is for or are there also workarounds in HA's ZHA as well? https://github.com/zigpy/zha-device-handlers/blob/dev/CONTRIBUTING.md https://github.com/zigpy/zha-device-handlers/blob/dev/README.md |
|
@puddly sorry to come with basic question, but as now, I have paired and receiving messages, I'm not sure on how to process such messages provided by
In other terms how can I decode : |
|
|
Thanks. on how to manipulate the bitmap that is ok. My question was more does that mean, that I need at the upper level to know what is the Attribute type in order to manipulate it ? Now , I understand the issue that zigpy have with the Konke switch where on 0x0006/0x0000 it reports something else than a Bool (the zigpy stack expect a Bool from the standard, but Konke decided otherwise). thanks for the cluster info |
|
If I'm understanding your question correctly, zigpy doesn't do any high-level translation like pulling apart a bitmap into individual bits (a lot of sensors are more complicated than they seem at first glance, like the IAS However, naming the bitmap fields is something I'd like to do in the future to make them easier to work with: In [1]: import zigpy.types as t
In [2]: class OccupancyAttribute(t.bitmap8):
... Occupancy = 0x01
... Reserved2 = 0x02
... Reserved3 = 0x04
... Reserved4 = 0x08
... Reserved5 = 0x10
... Reserved6 = 0x20
... Reserved7 = 0x40
... Reserved8 = 0x80
...
In [3]: attr = OccupancyAttribute(0b10000001)
In [4]: attr
Out[4]: <OccupancyAttribute.Reserved8|Occupancy: 129>
In [5]: attr & OccupancyAttribute.Occupancy
Out[5]: <OccupancyAttribute.Occupancy: 1>
In [6]: attr & OccupancyAttribute.Reserved2
Out[6]: <OccupancyAttribute.0: 0>
In [7]: OccupancyAttribute._member_map_
Out[7]:
{'Occupancy': <OccupancyAttribute.Occupancy: 1>,
'Reserved2': <OccupancyAttribute.Reserved2: 2>,
'Reserved3': <OccupancyAttribute.Reserved3: 4>,
'Reserved4': <OccupancyAttribute.Reserved4: 8>,
'Reserved5': <OccupancyAttribute.Reserved5: 16>,
'Reserved6': <OccupancyAttribute.Reserved6: 32>,
'Reserved7': <OccupancyAttribute.Reserved7: 64>,
'Reserved8': <OccupancyAttribute.Reserved8: 128>}This might make it possible for you to not have to do all of the parsing within your application code. |
Extend the ZHA device handlers contributing introduction with example puddly posted to pipiche38 in zigpy/zigpy#452 Feedback was from pipiche38 that he did not understand from README.md what ZHA device handlers for or that it could be used with only zigpy as stand-alone without the ZHA integration implementation in Home Assistant.
|
Just a quick update on my end. The good news. Based on the PersistentDD provided by @puddly , I have now a "working" POC based on a xiaomi Motion sensor When the plugin is started from Domoticz, it then create the corresponding widget in Domoticz, and then will update it when there is a motion detected and/or a Lux variation is detected
The so not good news, nobody at Domoticz side is interested so far to dive into the SQLITE3 problem. Here is the thread on the Domoticz forum: I have open an issue on Domoticz side: |
Copied this suggestion to its own issue here -> #469 |
As per definition the zigpy is implemented as Library for Zigbee devices.
Whoever all documentation I found is very focussed on HA.
I was under the impression that I could use zigpy as a pure python Library , maybe I'm wrong !
The text was updated successfully, but these errors were encountered: