Termux (Android) support

[Querela]

Hi,

In Termux you only receive a USB file descriptor (number bound to single process).
Any operations like usb.core.find(..) won't work.
I wrote a workaround/fix that extends the libusb1 interface and provides the Device object given the file descriptor.
See here: https://github.com/Querela/termux-usb-python/blob/e6166ded4555d988365e6609b91dec4f5ebb8fcd/usblib.py#L94

def device_from_fd(fd):
    # setup library
    backend = libusb1.get_backend()
    lib = backend.lib
    ctx = backend.ctx

    # extend c wrapper with android functionality
    lib.libusb_wrap_sys_device.argtypes = [
        libusb1.c_void_p,
        libusb1.c_int,
        libusb1.POINTER(libusb1._libusb_device_handle),
    ]

    lib.libusb_get_device.argtypes = [libusb1.c_void_p]
    lib.libusb_get_device.restype = libusb1._libusb_device_handle

    # get handle from file descriptor
    handle = libusb1._libusb_device_handle()
    libusb1._check(lib.libusb_wrap_sys_device(ctx, fd, libusb1.byref(handle)))

    # get device (id?) from handle
    devid = lib.libusb_get_device(handle)

    # device: devid + handle wrapper
    class DummyDevice:
        def __init__(self, devid, handle):
            self.devid = devid
            self.handle = handle
    dev = DummyDevice(devid, handle)

    # create pyusb device
    device = usb.core.Device(dev, backend)
    device._ctx.handle = dev

    # device.set_configuration()
    return device

I only tested it on the most current Android (10) and Termux version ...
Usage examples are in my Repo.

Can we implement this in your library?

Enumerating devices would maybe work with subprocesses but working with a specific device is only possible in a process(?) callback and does not seem feasable. I guess ..?

[jonasmalacofilho]

This seems reasonable.

Besides ctypes bindings for the libusb_wrap_sys_device (added in libusb 1.0.23) and libusb_get_device APIs, I think we should have a PyUSB interface that abstracts these details.

Of the top of my mind I'm not sure where to place or how to name this interface. But are you willing to give it a shot and send me a patch/open a pull request?

---

Archiving: original discussion on the libusb mailing list.

[Querela]

Can do.

But I was also not sure where to place this, as it is a different approach to the standard enumerate devices → select device to work with sequence.

Would this be good:

• usb.core.[device_]from_fd(fd: int) -> usb.core.Device
• + extend libusb1 (other backends are not on Android)

I am also not sure whether this is only for Android or if we can natively get USB file descriptors on other systems from other programs? to work with...

I have to check later but I think dev.set_configuration() did not work for me on Termux.

[Querela]

Do we need more detailed lib functions?

class _LibUSB(usb.backend.IBackend):
    # ...
    # fd -> handle
    @methodtrace(_logger)
    def wrap_sys_device(self, sys_dev):
        handle = _libusb_device_handle()
        _check(self.lib.libusb_wrap_sys_device(self.ctx, sys_dev, byref(handle)))
        return handle

    # handle -> devid
    @methodtrace(_logger)
    def get_device(self, handle):
        return self.lib.libusb_get_device(handle)

Or how it most probably will only be used?

class _FileDescriptorDeviceHandle(object):
    def __init__(self, fd, ctx):
        # get handle from file descriptor
        self.handle = _libusb_device_handle()
        _check(_lib.libusb_wrap_sys_device(ctx, fd, byref(self.handle)))

        # get device (id?) from handle
        self.devid = _lib.libusb_get_device(self.handle)

class _LibUSB(usb.backend.IBackend):
    # ...
    # fd --> dev (like _DeviceHandle)
    @methodtrace(_logger)
    def get_device_from_fd(self, fd):
        return _FileDescriptorDeviceHandle(fd, self.ctx)

---

I have not found a mention about the need to free the handles etc. Android libusb1 doc only mentions the creation of a temporary device...

---

Working on it: master...Querela:master

[Querela]

On my device:

git clone https://github.com/Querela/pyusb.git && cd pyusb
python3 -m venv venv
source venv/bin/activate
pip install -e .

termux-usb -l
termux-usb -r -e ./tools/termux_device_info.py /dev/bus/usb/001/002

DEVICE ID 10c4:ea60 on Bus 001 Address 002 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x110 USB 1.1
 bDeviceClass           :    0x0 Specified at interface
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x0
 bMaxPacketSize0        :   0x40 (64 bytes)
 idVendor               : 0x10c4
 idProduct              : 0xea60
 bcdDevice              :  0x100 Device 1.0
 iManufacturer          :    0x1 Silicon Labs
 iProduct               :    0x2 CP2102 USB to UART Bridge Controller
 iSerialNumber          :    0x3 0001
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 100 mA ==================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x20 (32 bytes)
   bNumInterfaces       :    0x1                                     bConfigurationValue  :    0x1
   iConfiguration       :    0x0
   bmAttributes         :   0x80 Bus Powered
   bMaxPower            :   0x32 (100 mA)
    INTERFACE 0: Vendor Specific ===========================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x2
     bInterfaceClass    :   0xff Vendor Specific
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x2 CP2102 USB to UART Bridge Controller
      ENDPOINT 0x81: Bulk IN ===============================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x2 Bulk
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x0
      ENDPOINT 0x1: Bulk OUT ===============================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :    0x1 OUT
       bmAttributes     :    0x2 Bulk
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x0

[mcuee]

Last time I tested Termux and it does work with libusb (requires root) on my Android TV boxes. So yes it will be nice to see pyusb working under Termux.

[Querela]

The output above works without root on my device (under Android 10).
I wrote a simple serial (uart?) wrapper around this and I can catch serial usb2ttl output, e. g. from my DSO138mini. gh:Querela/termux-usb-python

[mcuee]

Great. I will try this when I have some time, probably over the weekend.

Just notice that libusb was moved from termux-root-packages to the standard packages repo three months ago.
https://github.com/termux/termux-packages/tree/master/packages/libusb

[Querela]

Yes, I saw it only afterwards but I kind of remember that I saw the Termux issue some time ago when it was not yet implemented.
(With root this sys_device wrapping may not even be neccessary but as non-root user, we can only request a USB file descriptor from Android.)

Experiments etc.:
pySerial doesn't work for me because it requires device paths/names to open the USB descriptor. Maybe I can patch it up so that it will work with PyUSB but it natively tries to use hidapi. (Since the device handles should be the same, it may be interchangeable? I'm not sure.)
I also have a problem where programs try to set termios controls/settings and that will not work with the PyUSB handle. My research was also not successful, so I'm not sure how to do this. One user in the Termux issue (?) suggested remserial but I kind of doubt it works. And socat was not that newbie friendly ...
Since I wrote my own UART (?) serial with PyUSB + my patch, I know it works but I'm stuck on getting the Android USB FD to work with other tools ...

[normanr]

@Querela could you publish tools/termux_device_info.py somewhere. I'm interested in adding support for --usb-fd for a cli that I've written for a tiqiaa IR blaster.

[normanr]

😊 oops nm, I found it in your PR.

[normanr]

Also note that if termux/termux-api#349 is fixed then no special handling for termux will be required in pyusb as the termux build of libusb will handle everything for you.

[mcuee]

Yes I can build libusb and pyusb under termux now without any changes. Running the test suite without real hardware seems to indicate no issues.

But I have not tested with any real hardware devices yet.

[mcuee]

Related pull request: #287

[normanr]

Yes I can build libusb and pyusb under termux now without any changes. Running the test suite without real hardware seems to indicate no issues.

But I have not tested with any real hardware devices yet.

ack. The core issue is that the default selinux configuration on newer Android versions doesn't allow direct access to /dev/bus/usb/ and a custom API has to be used to open usb devices (but once open it allows access to the file descriptor, which can be passed into libusb1 to use the device as normal).

[mcuee]

Please also refer to here about using current libusb Android support.
libusb/libusb#830

[mcuee]

And another one.
libusb/libusb#874

[normanr]

ack, it looks like both of those only refer to accessing usb devices from within a java process that links to libusb. pyusb isn't going to be used in that way. I don't think pyusb is the right place to add support for Android, and that libusb should "do-it-all" (although some workarounds could be added in pyusb, until the time at which libusb fully supports Android).

[mcuee]

@normanr You are of course right. I think libusb/libusb#830 and libusb/libusb#874 are the right direction for libusb Android support. Please give them a try. The method in libusb pull request 830 is already usable now (it is just a README updates with an example).

As for this particular patch, it covers a niche use cases for Termux users. I am actually using Termux myself and I do not really need to use the normal Android libusb support at all as I do now know Java/Android development myself and have no interests to know about either of them. (I am not a programmer myself but I know a little bit about C and Python).

[mcuee]

Unfortunately I tried https://github.com/Querela/pyusb/commits/master and it does not seem to work under my Huawei P20 running Android 10.

The following two commands failed (hang). I simply do not have access to "/dev/bus/usb/" directory.

termux-usb -l
termux-usb -r -e ./tools/termux_device_info.py /dev/bus/usb/001/002

If I do not use the above two commands, it still failed.

~ $ git clone https://github.com/Querela/pyusb pyusb_termux
Cloning into 'pyusb_termux'...
remote: Enumerating objects: 2207, done.
remote: Total 2207 (delta 0), reused 0 (delta 0), pack-reused 2207
Receiving objects: 100% (2207/2207), 1.02 MiB | 5.07 MiB/s, done.
Resolving deltas: 100% (1281/1281), done.

~ $ cd pyusb_termux/

~/pyusb_termux $ python3 -m venv venv

~/pyusb_termux $ source venv/bin/activate

(venv) ~/pyusb_termux $ pip install -e .
Obtaining file:///data/data/com.termux/files/home/pyusb_termux
Installing collected packages: pyusb
  Running setup.py develop for pyusb
Successfully installed pyusb-1.0.2
WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.
You should consider upgrading via the '/data/data/com.termux/files/home/pyusb_termux/venv/bin/python3 -m pip install --upgrade pip' command.

(venv) ~/pyusb_termux $ python ./tools/termux_device_info.py
Traceback (most recent call last):
  File "/data/data/com.termux/files/home/pyusb_termux/./tools/termux_device_info.py", line 38, in <module>
    fd = int(sys.argv[1])
IndexError: list index out of range

[mcuee]

I tried my Mi Box S Android TV box running Android TV OS 9.0 as well and "termux-usb -l" just does not list anything. So it seems to me that I do not have the right Android device to carry out the test.

[mcuee]

Then I used another Android 7 Box with root option (H96 Max RK3328 based box, need to run as root to have the write permissions), plain libusb/pyusb work as expected. However, termux-usb or tools/termux_device_info.py (from https://github.com/Querela/pyusb/blob/master/tools/termux_device_info.py) does not work.

The error message is the same with root or without root.

(venv) ~/pyusb_termux $ python ./tools/termux_device_info.py /dev/bus/usb/005/007
Traceback (most recent call last):
  File "/data/data/com.termux/files/home/pyusb_termux/./tools/termux_device_info.py", line 38, in <module>
    fd = int(sys.argv[1])
IndexError: invalid literal for int() with base 10: /dev/bus/usb/005/007

[mcuee]

@normanr It seems you have done a lot of research for termux usb and pyusb, so do you think #287 is worth merging or it is no longer necessary?

Ref: termux/termux-packages#5831
Ref: termux/termux-api#349

[mcuee]

All in all, it seems to me for #287 to work, "termux-usb -l" should work and it seems to me that means the right permission and existence of "/dev/bus/usb" which does not seem to be the case for my Android devices (non rooted).

And this bridge seems to rely on a libusb fork. Technically you do not need to use a libusb fork as the latest libusb version should have the stuff in place already.
https://github.com/normanr/termux-libusb-bridge/blob/main/termux-libusb-bridge.py
termux/termux-api#349

But I think fundamentaly right now libusb needs a user provided helper (java) to work under Android non rooted device.
libusb/libusb#830

The following libusb pull request is supposed to get rid of that user helper.
libusb/libusb#874

[normanr]

termux_device_info.py accepts a file descriptor, not a path. When it's run via termux-usb the -e option will execute the specified command with a file descriptor referring to the device as its argument. You can not see the files because selinux does not allow it, only the java api's have access to the usb devices (and can be used open and return a file descriptor for them).

So if you want to test without termux-usb (like on a non-android device), you need to do something like: ./termux_device_info.py 9 9</dev/bus/usb/005/007 (i.e. use a shell redirect to open the usb device and pass the fd number to the process). You can see the effect by running termux-usb -e echo /dev/bus/usb/005/007 and seeing that it prints out a file descriptor number.

The latest libusb does not have the right-stuff in place yet (as far as I know) to support opening usb devices on non-rooted devices without having to change the source code of the binary that you're using. #287 has the same problem (the source code of the binary needs to be changed to explicitly support reading the fd from the command line and opening it and giving it to libusb).

libusb/libusb#874 doesn't get rid of the helper it just proposes to add some of the java glue required to get libusb working with non-rooted devices into libusb itself. It still requires the program that's opening the usb device to link to java and call the right init functions to make everything work. That only works for programs you have the source code to, and you're willing to change to make it work. The nice thing about AnotherTerm's helper is that it "just-works" for existing binaries.

It could be nice to expose the wrap_sys_device API in pyusb, but it does require code changes to make use of it. I have some pending changes in a working tree, mostly the suggestions I made on #287, if you want I can look at committing and pushing them somewhere (not as a full PR, but so that they're not 100% lost).

[mcuee]

It could be nice to expose the wrap_sys_device API in pyusb, but it does require code changes to make use of it. I have some pending changes in a working tree, mostly the suggestions I made on #287, if you want I can look at committing and pushing them somewhere (not as a full PR, but so that they're not 100% lost).

@normanr
Yes that would be good so that your work is not lost. Thanks.

[mcuee]

Not so sure if the patched libusb version will help. I will try it under Termux and unrooted Android.

Ref: https://github.com/green-green-avk/AnotherTerm-scripts/blob/master/install-patched-libusb-1.0.23.sh

[nathanielatom]

On my device:

git clone https://github.com/Querela/pyusb.git && cd pyusb
python3 -m venv venv
source venv/bin/activate
pip install -e .

termux-usb -l
termux-usb -r -e ./tools/termux_device_info.py /dev/bus/usb/001/002

DEVICE ID 10c4:ea60 on Bus 001 Address 002 =================
 bLength                :   0x12 (18 bytes)
 bDescriptorType        :    0x1 Device
 bcdUSB                 :  0x110 USB 1.1
 bDeviceClass           :    0x0 Specified at interface
 bDeviceSubClass        :    0x0
 bDeviceProtocol        :    0x0
 bMaxPacketSize0        :   0x40 (64 bytes)
 idVendor               : 0x10c4
 idProduct              : 0xea60
 bcdDevice              :  0x100 Device 1.0
 iManufacturer          :    0x1 Silicon Labs
 iProduct               :    0x2 CP2102 USB to UART Bridge Controller
 iSerialNumber          :    0x3 0001
 bNumConfigurations     :    0x1
  CONFIGURATION 1: 100 mA ==================================
   bLength              :    0x9 (9 bytes)
   bDescriptorType      :    0x2 Configuration
   wTotalLength         :   0x20 (32 bytes)
   bNumInterfaces       :    0x1                                     bConfigurationValue  :    0x1
   iConfiguration       :    0x0
   bmAttributes         :   0x80 Bus Powered
   bMaxPower            :   0x32 (100 mA)
    INTERFACE 0: Vendor Specific ===========================
     bLength            :    0x9 (9 bytes)
     bDescriptorType    :    0x4 Interface
     bInterfaceNumber   :    0x0
     bAlternateSetting  :    0x0
     bNumEndpoints      :    0x2
     bInterfaceClass    :   0xff Vendor Specific
     bInterfaceSubClass :    0x0
     bInterfaceProtocol :    0x0
     iInterface         :    0x2 CP2102 USB to UART Bridge Controller
      ENDPOINT 0x81: Bulk IN ===============================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :   0x81 IN
       bmAttributes     :    0x2 Bulk
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x0
      ENDPOINT 0x1: Bulk OUT ===============================
       bLength          :    0x7 (7 bytes)
       bDescriptorType  :    0x5 Endpoint
       bEndpointAddress :    0x1 OUT
       bmAttributes     :    0x2 Bulk
       wMaxPacketSize   :   0x40 (64 bytes)
       bInterval        :    0x0

Trying this, I get:

Traceback (most recent call last):                File "/data/data/com.termux/files/home/source/pyusb/./tools/termux_device_info.py", line 31, in <module>
    main(fd)
  File "/data/data/com.termux/files/home/source/pyusb/./tools/termux_device_info.py", line 19, in main
    device = usb.core.device_from_fd(fd)                     
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/home/source/pyusb/usb/core.py", line 1298, in device_from_fd
    raise NoBackendError('No backend available')
usb.core.NoBackendError: No backend available

termux-usb -l gives a list with /dev/bus/usb/001/002 as expected.

Any idea why import usb.backend.libusb1 as libusb1; libusb1.get_backend() would be returning None? I tried with both pkg install libusb which is currently at version 1.0.26-1 and uninstalling that and using pip install libusb==1.0.23b7 as that's the version mentioned near the start of this issue, but neither seem to provide a backend that's not None.

Any ideas?

Also were there any blockers for #287?

[mcuee]

You may need to put libusb-1.0.0.so in the right place in order for pyusb to work. Termux does not use the default location like /usr/local/lib or /usr/lib.

[mcuee]

Also were there any blockers for #287?

I do not think #287 is ready. Please try it and post your test results. By right it should work with stock libusb-1.0.26 from termux and not a third party fork of libusb-1.0.23.

And I think the right way to use libusb under Android is using the documented ways from libusb. I am not so sure how it can be adopted by Termux though as I do not know much about Android.
https://github.com/libusb/libusb/tree/master/android/examples

There is also an on-going improvement for libusb under Android.

• JNI Android Support on Unrooted devices libusb/libusb#1164

[nathanielatom]

You may need to put libusb-1.0.0.so in the right place in order for pyusb to work. Termux does not use the default location like /usr/local/lib or /usr/lib.

Thanks, yeah I tried stuff like https://stackoverflow.com/a/71102974/2561747 to specify the location to no avail.

But compiling stock libusb-1.0.23 worked, following this procedure termux/termux-packages#1714 (comment) but using the official release for version 1.0.23. I guess there may have been a backwards-incompatible change or regression since that version. But now the PR works and I can print device info, and read and write small bits of data.

I keep getting timeout errors when writing large amounts of data (even with very long timeouts), but I'm not sure if that's from the pyusb fork or the application code.