Dynamic USB descriptors

[dhalbert]

Implements dynamic USB descriptors, including HID.
Fixes #4351, fixes #1015, fixes #4242, fixes #2087.
Not yet included: ability to change interface names (#4433), boot keyboard (#1136).
#620 (touchpad) would implemented as a custom HID device and a library.
This will break the existing VENDOR WebUSB implementation and that will have to be redone.

API Summary [edited to reflect final API after PR was merged]

All configuration is done in boot.py.

• MSC and MIDI are simple on/off choices:

storage.disable_usb_device()    # Turn off MSC device
usb_midi.disable()   # Turn off MIDI device

• CDC: you can enable/disable REPL or data channel (API changed from 6.x):

# Turn on both REPL and data channels; default is just REPL.
usb_cdc.enable(console=True, data=True) 

• HID: default devices are usb_hid.Device.MOUSE, .KEYBOARD, and .CONSUMER_CONTROL. Gamepad, sys_control, etc, are all dropped, but could be supplied by the user. You can choose any combination of these, or roll your own.

my_device = usb_hid.Device(report_descriptor=bytes(...), , usage_page=1, page=2, in_report_length=5, report_id_index=2)
usb_hid.enable((usb_hid.KEYBOARD, my_device))

• All the configuration routines use the general configure_usb(), rather than just, say, enabled()) so that in the future they might take more arguments other than just bools or a list (e.g. interface names, multiple HID devices for a separate boot keyboard, etc.)

Implementation Details

The RAM needed is allocated dynamically according to size, when possible. A few things are small fixed arrays when it wasn't practical or worth it to make it dynamic. The boolean enabled/disabled choices are just remembered in static storage. The defaults are set before boot.py runs.

For the HID device list, it's trickier. The default list is set, and may be changed in boot.py. When boot.py finishes, before the VM goes away, the HID report descriptor is built. Since it might not be safe to leave it on the heap, it's copied up to the main.c stack temporarily, then the VM is stopped, then it's put back in a storage_allocation before the VM and heap are re-created for the REPL or code.py. Before the REPL or code.py VM is started, the other descriptors are built and put in storage_allocations or in static storage if they are fixed length. Then USB is initialized.

The HID devices are remembered in a short static array (up to 8 devices) between VM runs.

We could free the storage_allocations after USB starts, but it's hard to tell when, and I'm not sure the RAM can be made use of anyway, since it's just in a a random place. Perhaps more work could be done on that later, but we're talking a few hundred bytes.

The Python scripts that built the descriptors before are no longer needed. The descriptor fixups that used to be done in Python (put this interface number here and here, etc.) are now in C. The usb_descriptors submodule could be removed but requires some git surgery which I need to look up.

Tested on a Metro M4:

• Descriptors compared and vetted: they match the previous fixed descriptors.
• CIRCUITPY can be turned on and off
• MIDI tested: midi_simpletest.py works and MIDI can be disabled
• CDC: REPL only, data only, both
• HID: device list varied, and copy of mouse descriptor used for a user-created device and tested

I expect build failures due to size.

Sample boot.py [EDIT: black mangled the comments in the first version of this rather badly. Fixed.]

import storage
import usb_cdc
import usb_midi
import usb_hid

# storage.disable_usb_device()

# usb_cdc.enable(console=True, data=True)

# usb_midi.disable()

# This is the same as the regular mouse descriptor
my_mouse = usb_hid.Device(
    report_descriptor=bytes(
        # Regular mouse
        (0x05, 0x01)  # 0,1 Usage Page (Generic Desktop)
        + (0x09, 0x02)  # 2,3 Usage (Mouse)
        + (0xA1, 0x01)  # 4,5 Collection (Application)
        + (0x09, 0x01)  #   6,7 Usage (Pointer)
        + (0xA1, 0x00) #   8,9 Collection (Physical)
        + (0x85, 0xFF)   #   10,11 Report ID  [11 is SET at RUNTIME]
        + (0x05, 0x09)  #     Usage Page (Button)
        + (0x19, 0x01)  #     Usage Minimum (0x01)
        + (0x29, 0x05)  #     Usage Maximum (0x05)
        + (0x15, 0x00)  #     Logical Minimum (0)
        + (0x25, 0x01)  #     Logical Maximum (1)
        + (0x95, 0x05)  #     Report Count (5)
        + (0x75, 0x01)  #     Report Size (1)
        + (0x81, 0x02)  #     Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
        + (0x95, 0x01) #     Report Count (1)
        + (0x75, 0x03)  #     Report Size (8)
        + (0x81, 0x01)  #     Input (Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position)
        + (0x05, 0x01)  #     Usage Page (Generic Desktop Ctrls)
        + (0x09, 0x30)  #     Usage (X)
        + (0x09, 0x31)  #     Usage (Y)
        + (0x15, 0x81)  #     Logical Minimum (-127)
        + (0x25, 0x7F)  #     Logical Maximum (127)
        + (0x75, 0x08)  #     Report Size (8)
        + (0x95, 0x02)  #     Report Count (2)
        + (0x81, 0x06)  #     Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position)
        + (0x09, 0x38)  #     Usage (Wheel)
        + (0x15, 0x81)  #     Logical Minimum (-127)
        + (0x25, 0x7F)  #     Logical Maximum (127)
        + (0x75, 0x08)  #     Report Size (8)
        + (0x95, 0x01)  #     Report Count (1)
        + (0x81, 0x06)  #     Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position)
        + (0xC0,)  #   End Collection  
        + (0xC0,) # End Collection
    ),
    usage_page=1,
    usage=2,
    in_report_length=4,
    report_id_index=11,
)

usb_hid.enable((usb_hid.Device.KEYBOARD, my_mouse))

[ThomasAtBBTF]

I like this approach very much.
Will start to test as soon as it is available in a beta build.

[Red-M]

Does this allow setting multiple keyboard HID descriptors and their order?
If it does this would also allow a 6+ KRO keyboard HID and a method to allow BIOS mode keyboards.

[dhalbert]

Does this allow setting multiple keyboard HID descriptors and their order?
If it does this would also allow a 6+ KRO keyboard HID and a method to allow BIOS mode keyboards.

No, multiple HID device descriptors are not yet supported.

[tannewt]

Here are my first API thoughts. I haven't looked at the code yet. Thanks for working on this!

• All the configuration routines use the general configure_usb(), rather than just, say, enabled()) so that in the future they might take more arguments other than just bools or a list (e.g. interface names, multiple HID devices for a separate boot keyboard, etc.)

I'm not a fan of the common configure_usb design because it makes them equally vague. I think it'd be better to have interface specific functions like storage.disable_usb_drive(), usb_midi.disable(), and usb_cdc.configure_interfaces(). I think this will make the boot.py code more readable.

Another thing to consider is whether this will work in the longer term where some interfaces are disabled and others are enabled. It might be better to set them all at once so you have one source for errors. What would happen if someone tries to enable too many things at once?

We could free the storage_allocations after USB starts, but it's hard to tell when, and I'm not sure the RAM can be made use of anyway, since it's just in a a random place. Perhaps more work could be done on that later, but we're talking a few hundred bytes.

I don't think you can because USB can restart at any time.

[jerryneedell]

@dhalbert I hate to keep bringing the pirkey_m0 up , but this removes pulseio from it again. Was that intentional? Should I give up on it...

[jerryneedell]

Ah -- it's more than just that -- I put pulseio back in -- now get an IOError: USB Busy -- I'll look into that but it may be time to move on...

[dhalbert]

Ah -- it's more than just that -- I put pulseio back in -- now get an IOError: USB Busy -- I'll look into that but it may be time to move on...

The removal of pulseio was unintentional; I will put it back. If you are getting USB errors, that is something else, and may reflect a larger problem. Try a Trinket M0 or similar; I did test, but not after I did a lot of the build rework.

[jerryneedell]

I opened #4724

[deshipu]

Works like a charm: https://hackaday.io/project/179496-chocolad-keyboard-hacking/log/192866-dynamic-usb-descriptors

[dglaude]

Works like a charm: https://hackaday.io/project/179496-chocolad-keyboard-hacking/log/192866-dynamic-usb-descriptors

Based on the above, I have been trying to improve my mouse jiggler to make the presence of CircuitPython stealth.
But I fail with two error: https://gist.github.com/dglaude/aad3d9e19858b062306b2a514434fd6b

No sure exactly what I did wrong (more than one thing likely)... I tried my best to upgrade my CPX to the latest CP, upgrade all the mpy and run this code as code.py... but It still fail. When using boot.py nothing seems to occurs.

[Red-M]

From my understanding, that slide position check and the disabling of USB descriptors needs to be in boot.py after you re-plug the board in.

[dglaude]

From my understanding, that slide position check and the disabling of USB descriptors needs to be in boot.py after you re-plug the board in.

So indeed boot.py is needed, and I was confused because the output of boot.py are in boot_out.txt.

It is very hard to troubleshoot and understand what is happening... but I also need to convince Windows 10 that the filesystem is clean. Any ejection when CIRCUITPY is visible make this trick stop to work for me and the drive keep reappering. Is the mass storage temporary visible to windows, and once the popup appear it is not possible to hide the drive???

Here is my minimal boot.py file for CPB board:

import board
import digitalio
import storage
import usb_cdc

switch = digitalio.DigitalInOut(board.SLIDE_SWITCH)
switch.direction = digitalio.Direction.INPUT

if switch.value:
    storage.disable_usb_drive()
    usb_cdc.disable()

[dhalbert]

That code looks correct. You should be able to get a clean unmount in Windows by doing an "Eject" from the USB icon in the taskbar. But that should not affect whether this code runs.

boot.py only runs after a hard reset (power-cycle, reset button press, or similar). It does not run after auto-reload due to editing code.py or another file, or if you type ctrl-D in the REPL.

[ThomasAtBBTF]

Hi,

this code:

import usb_cdc
import storage
import digitalio
import microcontroller

buttona = digitalio.DigitalInOut(microcontroller.pin.GPIO15)
buttona.direction = digitalio.Direction.INPUT
buttona.pull = digitalio.Pull.UP
thisstat = not buttona.value
buttona.deinit()

if thisstat:
  print("disable_usb_drive")
  storage.disable_usb_drive()
  print("usb_cdc.enable(console=False, data=True)")
  usb_cdc.enable(console=False, data=True)`
else:
  print("nothing changed")
print("boot.py done")

works for me on a MagTag.

What other USB-devices (end-points) can I deactivate to get:
usb_cdc.enable(console=True, data=True)
working?

If I try this the boot process is stuck in a loop!

(I think because "safe-mode" is not working on a ESP32-S2 !?

[dhalbert]

Do usb_hid.disable(). I think that will get you the extra endpoint pair you need.

The safe mode problem has an issue at #4766.

(If you put your entire code in a pair of triple-backquotes it will be marked as a single block of code.)

EDIT: and if you add py after the first set of triple-backquotes, it will syntax-color the code as Python. I did that above.

[bitboy85]

I somehow broke the boot.py file which makes the file system read-only.

Adafruit CircuitPython 7.0.0-alpha.2-584-g25a44cb77 on 2021-05-15; Raspberry Pi Pico with rp2040
boot.py output:
OSError: [Errno 5] Input-/Output error

I think it should be the other way round: If something is wrong with boot.py, the filesystem should be always writeable so you are able to fix the error.

Renaming boot.py via REPL didn't work but formating the drive resets it completly.

[dhalbert]

I think it should be the other way round: If something is wrong with boot.py, the filesystem should be always writeable so you are able to fix the error.

If the filesystem is corrupted, then the metadata is probably damaged (e.g., the directory entries). It can't be recovered, so we or the host computer make it read-only so no worse damage will happen.

Read about the best way to edit files and avoid corruption here:
https://learn.adafruit.com/welcome-to-circuitpython/creating-and-editing-code#editing-code-2977443-18

[bitboy85]

Ok, because the error got written to boot_out.txt it looked liked an internal behaviour of circuitpython.

Is there an updated sample boot.py from your first post?

[dhalbert]

Is there an updated sample boot.py from your first post?

What is your boot.py? That would be the easiest. Also see the documentation: https://circuitpython.readthedocs.io/en/latest/shared-bindings/index.html.

I will be writing a Learn Guide shortly. This is all very new. I am working on boot HID devices at the moment.

[ladyada]

FYI tested this last nite with RP2040, very successful!

[bitboy85]

What is your boot.py? That would be the easiest.

I've just copied the sample you gave in this thread in the first post #4689 (comment)

but it outputs an error in boot_out.txt

Adafruit CircuitPython 7.0.0-alpha.2-584-g25a44cb77 on 2021-05-15; Raspberry Pi Pico with rp2040
boot.py output:
back trace (jüngste Aufforderung zuletzt):
file "boot.py", line 48, in
ValueError: out_report_length must be 1-255

Edit: According to this doc, out_report_length is allowed to be 0 https://circuitpython.readthedocs.io/en/latest/shared-bindings/usb_hid/

[ThomasAtBBTF]

I somehow broke the boot.py file which makes the file system read-only.

Adafruit CircuitPython 7.0.0-alpha.2-584-g25a44cb77 on 2021-05-15; Raspberry Pi Pico with rp2040
boot.py output:
OSError: [Errno 5] Input-/Output error

I think it should be the other way round: If something is wrong with boot.py, the filesystem should be always writeable so you are able to fix the error.

Renaming boot.py via REPL didn't work but formating the drive resets it completly.

I ran also into errors in boot.py during coding....
To get back a working system I simply downgraded to CP6.2 where these new libraries are not available.
Sure I got errors in boot.py but nothing locked up! And I could edit boot.py

[todbot]

I tested disabling CDC and MSD on a QTPy M0, using touchio pins. Works great, thank you!
Gist here: https://gist.github.com/todbot/707e4e3d393313cf31cdab56bf9d4255

[Billfred]

I think I'm missing a memo here. Between the information at the top of this pull request and @todbot's Gist from 5/17/21, I got to Billfred/Serpente-iPad-HID@affb16e for boot.py on my Serpente R2. Reset the board, and boot_out.txt reads:

Adafruit CircuitPython 7.0.0-alpha.4 on 2021-07-08; Serpente with samd21e18
boot.py output:
Traceback (most recent call last):
  File "boot.py", line 19, in <module>
AttributeError: 'module' object has no attribute 'KEYBOARD'

Mass storage is also remounting, which is an even clearer sign that something is going off the rails since it sure looks like I'm disabling storage in boot.py. Any insight in what I'm doing wrong?

[dhalbert]

@Billfred Please see https://learn.adafruit.com/customizing-usb-devices-in-circuitpython and https://circuitpython.readthedocs.io/en/latest/README.html for a more complete version of the latest information> if you have trouble please ask in https://adafru.it/discord or https://forums.adafruit.com.