Add-on suggestion finder for USB devices

[andrewfg]

This is a service to suggest USB device related add-ons.

For example Z-Wave, Zigbee, or EnOcean sticks.

Signed-off-by: Andrew Fiddian-Green software@whitebear.ch

---

Note: after merging this PR we also need to add the discovery methods to the respective bindings' addon.xml files. For example as follows..

// Zigbee
<discovery-methods>
	<discovery-method>
		<service-type>usb</service-type>
		<match-properties>
			<match-property>
				<name>product</name>
				<regex>(?i).*zigbee.*</regex>
			</match-property>
		</match-properties>
	</discovery-method>
</discovery-methods>

// Z-Wave
<discovery-methods>
	<discovery-method>
		<service-type>usb</service-type>
		<match-properties>
			<match-property>
				<name>product</name>
				<regex>(?i).*z-wave.*</regex>
			</match-property>
		</match-properties>
	</discovery-method>
</discovery-methods>

// EnOcean
<discovery-methods>
	<discovery-method>
		<service-type>usb</service-type>
		<match-properties>
			<match-property>
				<name>manufacturer</name>
				<regex>(?i)enocean.*</regex>
			</match-property>
		</match-properties>
	</discovery-method>
</discovery-methods>

[andrewfg]

@mherwege this is a first draft of a finder for USB device related bindings. => And comments?

[andrewfg]

Obviously javax.usb is not part of OH core, so karaf complains. I need to figure out how to import that module.

[mherwege]

Obviously javax.usb is not part of OH core, so karaf complains. I need to figure out how to import that module.

The knx binding has it as an (optional) dependency, probably transitive because I don’t see it appear in the code directly.
javax.usb is just the api, not the implementation. There are some concept implementations for it, but not covering all platforms (https://javax-usb.sourceforge.net/). I believe the most common implementation is http://usb4java.org/index.html (last updated in 2018). Another one I found is https://github.com/KeyBridge/lib-javax-usb3.
You can try if you get it working with any of these.

Be aware there may be plaform issues. E.g. Linux may need write permissions to the device, Windows would need a device driver.
To my knowledge none of the add-ons uses USB. They all use serial and USB sets up a gateway to a serial port. We need to validate that this can discover these devices and will not interfere with operations.

[andrewfg]

@mherwege OK I understand. The serial port stuff is useless because serial ports a) have no data about the source, and b) would need to be configured before they could be discovered. I tried to go the USB route because 1) who does serial by any other means than USB these days?, and 2) the stick can report its product, manufacturer, and serial number data (which could be used to check for matches before the serial is actually set up).

[mherwege]

@andrewfg And your idea may very well work. But it will need a lot of testing. It would be useful to find a list of USB product an manufacurer characteristics to math on so not all sticks for zwave and zigbee that can be used with OH need to be tried individually to find the match criteria.

[andrewfg]

would be useful to find a list of USB product an manufacurer characteristics to math on so not all sticks for zwave and zigbee that can be used with OH need to be tried individually to find the match criteria

AFAICT the usb devices only support very limited properties — namely manufacturer, product id, and serial number; so probably in the first step we would just suggest on the manufacturer name. If we need to filter on more specific data, we could add that later.

EDIT: alternatively simply match on ‘Z-‘ resp ‘Zi’ in the product names.. :) ??

@mherwege further to my above post, I realise that the core usbSerial discovery classes do have a strong basis for the discovery part as well as the serial part; so I think I can benefit from that after all.

More tomorrow..

[mherwege]

I realise that the core usbSerial discovery classes do have a strong basis for the discovery part as well as the serial part; so I think I can benefit from that after all.

I had a look at it as well. As far as I can understand, this will only work on linux. Embedding one of the libraries I mentioned could potentially make it work for multiple platforms. Apart from that, I think what is there in the code will also have the same constraints I cited above on linux (device must have write permission). This can be documented, en in openhabian it could be made part of the setup scripts.

[andrewfg]

As far as I can understand, this will only work on linux.

Indeed. So I am going back to plan A (javax.usb)

I found the following post in the community, which may help me to import the java4usb module without causing Karaf issues. It seems that this technique has been used in o.o.c.audio and o.o.c.io.transport.modbus ..

[mherwege]

Here is the original PR that introduced the functionality in core: eclipse-archived/smarthome#5315

I wonder if it would make sense to work on the core functionality to improve platform support. I am not sure where the core functionality is currently used.

For the dependency: I would make this a feature (like mDNS and UPnP finders) that can be installed/uninstalled. In that case, you should be able to add this dependency to the feature. It would only be installed when the feature is enabled. Wouldn't that be enough?

[mherwege]

Did a search, I can only find the Enocean binding as a consumer of the usb discovery service.

[wborn]

The issue with most USB sticks is that they all use the same FTDI chips which results in discovery false positives. :-/

[wborn]

It would also be better to reuse the existing USB discovery mechanism instead of creating a new one. That way you can also suggest add-ons based on ser2net discovery results (#2519).

[andrewfg]

@mherwege today's latest commit is now able to find the USB devices (at least the six devices on my Windows system .. listed below).

vendorId:0bc2 (vendorName:null) / productId:2322 (productName:null)
vendorId:0a5c (vendorName:null) / productId:21e8 (productName:null)
vendorId:047d (vendorName:null) / productId:00f2 (productName:null)
vendorId:045e (vendorName:null) / productId:082a (productName:null)
vendorId:045e (vendorName:null) / productId:00db (productName:null)
vendorId:046d (vendorName:null) / productId:0826 (productName:null)

As you can see, the issue is that the device data returns vendor id codes and product id codes, rather than strings. The vendor id string is available via online lookup (and in the attached pdf). It seems that only certain usb devices include the strings within the device itself.

vendor_ids051920_0.pdf

issue with most USB sticks is that they all use the same FTDI chips

Sorry but I don't understand.

the Enocean binding as a consumer of the usb discovery service

I did look into that. .. ..

[andrewfg]

It would also be better to reuse the existing USB discovery mechanism instead of creating a new one

@wborn it would be nice to do that. But apparently it only works on Linux? And AFAICT it does not read the USB stick vendor name and product name. So it would not be much use in determining if the stick supports a certain functionality, or not. => But if you think that is not the case, please let me know.

[wborn]

Sorry but I don't understand.

openhab/openhab-addons#4421

But apparently it only works on Linux?

There can be multiple UsbSerialDiscovery implementations so you can add another one like linuxsysfs that works on Windows or other platforms.

it does not read the USB stick vendor name and product name

Does it really read it from the device or does it have some internal database to return names for certain vendor/product IDs?

[andrewfg]

does it have some internal database to return names for certain vendor/product IDs?

I dunno. Still learning.

[andrewfg]

would also be better to reuse the existing USB discovery mechanism instead of creating a new one

There can be multiple UsbSerialDiscovery implementations so you can add another one

Ok. I think I could mis-use the existing disco mechanism. And add a finder for windows too. However the existing disco is for creating things, so the mis-use would be to not create a thing, but that method could create a suggestion instead.

[andrewfg]

@mherwege Aargh!! I really don't understand how Karaf works. The documentation is really rubbish! So I wonder if you can kindly help me to eliminate the current CI build error due to the following two dependencies that this finder requires?

    <dependency>
      <groupId>javax.usb</groupId>
      <artifactId>usb-api</artifactId>
      <version>1.0.2</version>
    </dependency>
    <dependency>
      <groupId>org.usb4java</groupId>
      <artifactId>usb4java-javax</artifactId>
      <version>1.3.0</version>
    </dependency>

[mherwege]

I often struggle with this as well. I don't think the problem is with your pom file. The error is in the karaf feature verification:

[ERROR] Failed to execute goal org.apache.karaf.tooling:karaf-maven-plugin:4.4.4:verify (karaf-feature-verification) on project org.openhab.core.features.karaf.openhab-core: Feature resolution failed for [openhab-core-config-discovery-addon-usb/4.1.0.SNAPSHOT]
[ERROR] Message: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity=openhab-core-config-discovery-addon-usb; type=karaf.feature; version=4.1.0.SNAPSHOT; filter:="(&(osgi.identity=openhab-core-config-discovery-addon-usb)(type=karaf.feature)(version>=4.1.0.SNAPSHOT))" [caused by: Unable to resolve openhab-core-config-discovery-addon-usb/4.1.0.SNAPSHOT: missing requirement [openhab-core-config-discovery-addon-usb/4.1.0.SNAPSHOT] osgi.identity; osgi.identity=org.openhab.core.config.discovery.addon.usb; type=osgi.bundle; version="[4.1.0.202312141745,4.1.0.202312141745]"; resolution:=mandatory [caused by: Unable to resolve org.openhab.core.config.discovery.addon.usb/4.1.0.202312141745: missing requirement [org.openhab.core.config.discovery.addon.usb/4.1.0.202312141745] osgi.wiring.package; filter:="(osgi.wiring.package=org.usb4java)"]]

So it must be somewhere in the feature definition, where it cannot find org.usb4java. That's why it will work in Eclipse (where you manually include these bundles in your bndrun file) and not in the distro with karaf.

[mherwege]

Does it really read it from the device or does it have some internal database to return names for certain vendor/product IDs?

I don't think it has an internal DB. The usb4java (http://usb4java.org/index.html) library uses libusb (https://libusb.info/) underneath. This is a native libary that talks to the USB devices, written in C. Multiple platform versions are packaged with usb4java. Looking at teh API for libusb, it has methods to retrieve that info from the device, and I assume that's also what is being used in usb4java.

[mherwege]

I think trying to detect USB serial devices may be opening a can of worms here. Some agreement needs to be reached on path forward. Here are a few observations:

Current USB serial discovery

• As it stands, we have an API for thing discovery in core. This is installed by default. It only has an implementation that works for linux and ser2net (which itself will detect serial interfaces in the network based on mDNS, so a 'server' component needs to be installed on the machine with the actual serial interface).
• The API offers thing discovery methods in the context of thing discovery. It doesn't have methods that fit the add-on finders at the moment. This can of course be extended.
• The linux detection is based on sysfs, so using the linux file system to detect devices.
• There is no easy mechanism to detect USB serial devices on Windows, which I assume would be the other platform that should be supported as a minimum. A google search did not reveal much that could be used directly.
• I don't know if the current mechanism for linux would also work for Mac.
• The linuxsysfs discovery is available and running on any platform but will identify when it cannot perform a scan and not do anything.
• Only the enocean binding uses this for thing discovery.

javax.usb

• javax.usb (https://javax-usb.sourceforge.net/) is an interface (and concept implementation for linux) for USB device management. It is a standard, so could be used, but the included implementations are old and not maintained.
• So using the interface is relevant, but I don't think we should use any of the provided implementations.

usb4java

• usb4java (http://usb4java.org/index.html#) is a library wrapping a native library, libusb (https://libusb.info/), for USB device management and access. It implements its own interface (direct wrapper around the native library) and a javax.usb compliant implementation.
• It wraps the native libraries for linux, Windows and Mac on a number of architectures.
• It is available as a maven package.
• A new USB serial discovery implementation could be created using this library. However, it may lead to conflicts with the existing linuxsysfs discovery as both would discover the same device on linux.
• We are not sure we can get relevant vendor and product info using this library yet.
• The libary does not look to be maintained much (last changes seem to be from 5 years ago). Documentation is good though. All source code is available.
• There already are forks, some with continuous development. But I haven't seen one with a maven package. See for example: https://github.com/KeyBridge/lib-javax-usb3
• The libusb native libraries do have a better maintainance track record. The latest versions of this are not included in usb4java. It may be a simple drop-in replacement, but it would mean we need to start maintaining a fork.
• usb4java uses Apache commons (and it seems to be for only 2 calls), which increases the package size a lot. We could probably eliminate it, but again, it would mean a fork to maintain ourselves.

I don't see a perfect solution here at the moment. But before going too far in this, I would like to see opinions and consensus being developed. Otherwise we risk spending a lot of time and not getting anywhere.

@andrewfg @wborn @kaikreuzer What are your views on this?

[mherwege]

And AFAICT it does not read the USB stick vendor name and product name.

From the code, I think it does (or at least it tries to find it):

openhab-core/bundles/org.openhab.core.config.discovery.usbserial.linuxsysfs/src/main/java/org/openhab/core/config/discovery/usbserial/linuxsysfs/internal/SysfsUsbSerialScanner.java

Line 247 in 4b92db3

private UsbSerialDeviceInformation createUsbSerialDeviceInformation(Path usbDevicePath, Path usbInterfacePath,

[andrewfg]

@mherwege thanks for your many lengthy posts. Just a few short answers..

1. The existing OH USB discovery comprises two sorts of classes. Namely several scanners which parse the USB device tree and read their data. And a discovery service that uses inputs from those scanners, and calls discovery participants with that data.
2. OH currently does not have a scanner for Windows.
3. It is possible that I could write such a scanner for Windows, that would plug into the discovery service above. But it might have fewer features than the other scanners. However it would provide some data anyway.
4. It is possible that I can write an addon finder service that uses the above mentioned existing and new scanners. But is different than the above mentioned disco service resp disco participants.
5. So in summary I would create two modules, one of which would enhance existing USB thing discovery on other platforms, and the other which would use the scanners to find potential addons.
6. I hope that today I will find a way to get my current PR to build in Karaf, and when that works I will push a new commit with the abovementioned two modules.

[mherwege]

@andrewfg Your understanding is correct.

[andrewfg]

@mherwege I cannot complete the rebase and have the CI build succeed, because due to the migration from v4.1 to v4.2 somebody changed the spotless formatter parameters which means that it reformats all 138 pom.xml files, and since nobody has yet reformatted the pom.xml files on github main, that means that committing from this PR now would show include all 150 'changed' files into this PR. So I shall have to wait until someone else sorts that mess out.

EDIT it has been fixed via #3953

[andrewfg]

I need to change from having vendorId and productId as separate properties, to having a combined chipId property..

[mherwege]

I need to change from having vendorId and productId as separate properties, to having a combined chipId property..

What was the reason for this? After all, you could do that already by providing both in the discovery parameters.

[andrewfg]

What was the reason for this?

A correct match is an AND of the combination of vendorId and productId. If you look at the following Zigbee example you can see for your self how difficult it would be to code match properties for OR of multiple stick types each having an AND..

https://github.com/openhab/org.openhab.binding.zigbee/blob/360000619924b4190b575fc25545220f19fbdb55/org.openhab.binding.zigbee/src/main/resources/OH-INF/addon/addon.xml

[wborn]

Thanks for the nice enhancement!