Sources and external partners with existing third-party databases/repositories that could be be scraped/harvested into new device database?

[Hedda]

Open letter question related to #27

What sources or external partners already have with existing third-party databases/repositories/libraries/projects with data that could potentially be be scraped and harvested into the new device database if given permission or allowed without asking if within the license/terms-of-service?

FYI, here is a list with a few potential sources and potential external partners with third-party device databases/libraries that I know of:

• https://devices.esphome.io/

  • https://github.com/esphome/esphome-devices

• https://www.zigbee2mqtt.io/supported-devices/

  • https://github.com/Koenkk/zigbee2mqtt.io/tree/master/docs/devices

• https://csa-iot.org/csa-iot_products/?p_type=14#products

• https://products.z-wavealliance.org/products

• https://devices.zwave-js.io/

• https://opensmarthouse.org/zwavedatabase/

• https://github.com/openhab/org.openhab.ui.habmin/wiki/Z-Wave-Product-Database

  • https://github.com/openhab/openhab1-addons/tree/main/bundles/binding/org.openhab.binding.zwave/database

• https://github.com/OpenZWave/open-zwave/tree/master/config

• https://templates.blakadder.com/

  • https://github.com/blakadder/templates

• https://zigbee.blakadder.com/

  • https://github.com/blakadder/zigbee

• https://matterdevices.io/

• https://www.matterdatabase.com/

  • https://github.com/matterdatabase/matterdatabase

• https://ch-bas.github.io/cctv-camera-database/

  • https://github.com/ch-bas/cctv-camera-database

• https://usb-ids.gowdy.us/index.html

  • https://usb-ids.gowdy.us/read/UD/

  • https://usb-ids.gowdy.us/read/UC/

• https://www.sortabase.com/databases?category=Smart%20Home

• https://www.theverge.com/23568091/matter-compatible-devices-accessories-apple-amazon-google-samsung

• https://www.domoticaeconomica.com/en/collaborative-database/

• https://www.smarthome-database.eu/en

• https://partners.smartthings.com/supported-devices

• https://www.samsung.com/se/smartthings/compatible-devices/

• https://homey.app/en-us/apps/browse/

• https://www.lg.com/global/business/smart-home

• https://www.home-connect.com

I am unsure which (if any) of them will allow you to automatically scrape and havest to collect all their data without explicit permission? The TOU might also be very different if end users later manually copy the data on a per device basis instead of OHF developers doing it programmatically?

Anyway, above links are primarly focusing specifically on getting context from high-value sources to collect information needed for generating automations (via purpose specific triggers and conditions) and pre-purchasing research before buying smart home devices made purposely for home automation (i.e. smart plugs, smart lights/bulbs/lamps, switches, covers, fans, buttons, sensors, locks, cameras, thermostats and HVAC systems, etc., and also compatible connected appliances, like example robot vacuums, washing machines, dryers, dishwashers, etc. as well as different alarm devices).

[mkerstner]

Thanks for the list, there are further ones we also consider like https://smarthomeindex.com but as you've mentioned we also want to clarify the TOU / license part first.

[mkerstner]

Update

• https://www.reddit.com/r/homeassistant/s/0H1rrj6qBH

[Hedda]

HA-Battery-Notes project has a library with specs on different battery types and amount of batteries for many battery-powered smart devices:

• https://github.com/andrew-codechimp/HA-Battery-Notes

  • https://github.com/andrew-codechimp/HA-Battery-Notes/blob/main/library.md

    • https://github.com/andrew-codechimp/HA-Battery-Notes/blob/main/library/library.json

PS: The battery library is a JSON document at library/library.json and they have a form that let anyone to easily submit new devices:

• https://github.com/andrew-codechimp/HA-Battery-Notes/issues/new?template=new_device_request.yaml&title=%5BDevice%5D%3A+

[Hedda]

Zigbee4Domoticz Certified Devices package (a.k.a. z4d config files used by Zigbee for Domoticz) contain Zigbee devices clusters and attributes:

• https://github.com/zigbeefordomoticz/z4d-certified-devices

Very nice JSON formatted data in individual documents per device that is sorted per manufacturer/brand:

• https://github.com/zigbeefordomoticz/z4d-certified-devices/tree/main/z4d_certified_devices/Certified

This detailed Zigbee device data could if combined with unformatted data collected from Zigbee2MQTT's pages of Z2M support devices:

• https://www.zigbee2mqtt.io/supported-devices/

  • https://github.com/Koenkk/zigbee2mqtt.io/tree/master/docs/devices

All this context could be useful for users of the ZHA integration and allow LLM's to evaluate device capabilities and generate automations.

For reference, Zigbee clusters are the logical building blocks that define the specific functionality of a Zigbee device. Each cluster belongs to a functional domain (e.g., HVAC, Lighting) and contains a specific set of commands and attributes. FYI, Matter devices also uses very similar logic. These have all the supported features and endpoint mappings. Zigbee or Matter application can have multiple endpoints with each endpoint containing a device configuration made up of clusters on that end-point. Zigbee and Matter devices all need to follow the standard Zigbee Cluster Library (ZCL) specification, however cluster groups differ slighly depending on whether it is a Zigbee or a Matter device.

Additional data about clusters could maybe also be extracted from ZHA Device Handlers (zha-quirks) and Z2M's zigbee-herdsman-converters(?):

• https://github.com/zigpy/zha-device-handlers

  • https://github.com/zigpy/zha-device-handlers/tree/dev/zhaquirks

• https://github.com/Koenkk/zigbee-herdsman-converters

  • https://github.com/Koenkk/zigbee-herdsman-converters/tree/master/src/devices

[Hedda]

Matter Survey (matter-survey.org) web inteface does a nice job at exposing what devices can do (presumably based on supported endpoints):

• https://matter-survey.org/

  • https://matter-survey.org/device/eve-energy-4874-80

PS: Apparently it also uses data contributed by Matter Binding Helper users to show allowed bindings too:

• https://github.com/cedricziel/ha-matter-binding-helper

  "Telemetry: Optional anonymous device telemetry to improve Matter ecosystem understanding"