rp2040: ep 0 out was already available

[DatanoiseTV]

Operating System

Linux

Board

Custom RP2040 board

Firmware

https://github.com/rppicomidi/midi2usbhost custom version

What happened ?

I've built a custom board which has USB Type-C for Firmware Uploads (and during boot) and a USB Type-A which is switched to the D+/D- Pins with a FSUSB42 (and which power is controlled via MOSFET).

The select pin of the FSUSB42 is pulled low, so the device starts up with the USB-C being active for firmware upload.
Then the firmware puts SEL high, calls board_init() / tusb_init() as usual and enables VBUS. The USB-C is then only used for supplying the board with power.

Unfortunately and unexpectedly, the Type-A won't work because TinyUSB says the EP 0 out was already claimed.

I've tried a workaround with calling a usb reset before calling tusb, but that won't help.

    // set device to USB Host mode
    gpio_put(PIN_USB_MUX_SEL, 1);

        // Reset usb controller
    reset_block(RESETS_RESET_USBCTRL_BITS);
    unreset_block_wait(RESETS_RESET_USBCTRL_BITS);

    // Clear any previous state in dpram just in case
    memset(usb_dpram, 0, sizeof(*usb_dpram)); // <1>

    gpio_init(PIN_VUSB_EN);
    gpio_set_dir(PIN_VUSB_EN, GPIO_OUT);
    gpio_put(PIN_VUSB, 1);

How to reproduce ?

1. Clone the midi2usbhost repo
2. Try to switch from a device being connected on board to a host being connected

Debug Log as txt file

ep 0 out was already available
WARN: starting new transfer on already active ep 0 out
assertion "!(buf_ctrl & USB_BUF_CTRL_FULL)" failed: file "/home/x/dev/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/rp2040_usb.c", line 219, function: sync_ep_buffer

Screenshots

No response

I have checked existing issues, dicussion and documentation

• I confirm I have checked existing issues, dicussion and documentation.

[DatanoiseTV]

On further investigation, this seems to happen with some MIDI devices and unrelated to my USB MUX. For example, I cannot get a Novation Remote Zero SL to enumerate.

[hathach]

move to discussion since it is not clear too me if the bug is caused by tinyusb stack or user error

[d-frey]

Not sure if my issue is related, the title of this discussion fits well, but I have a different setup. I'm using two Pi Picos and I cloned the SDK and the pico-examples repositories, then compiled the examples with CMake on an Ubuntu 22.04LTS VM. The pico-examples contain the examples from tinyusb, so I programmed the example with

openocd -f interface/picoprobe.cfg -f target/rp2040.cfg -c "program usb/host/tinyusb_host_examples/cdc_msc_hid/tinyusb_host_cdc_msc_hid.elf verify reset exit"

and connected to the Pico with minicom. I'm seeing this when connecting a keyboard to the Pico:

TinyUSB Host CDC MSC HID Example
A device with address 5 is mounted

*** PANIC ***

ep 0 out was already available

I tried using the "develop" branch of the pico-sdk/-examples, but that doesn't really help. Things get weirder: Sometimes I see

ep %d %s was already available

I don't even understand how it prints the format string without replacing any values. I could give the exact sequence of commands I used to clone and build the examples, if that helps...

[AntonHermann]

I've got the same problem, using TinyUSB on a custom rp2040 board and I don't even understand what the error message "ep 0 out was already available" means. If the endpoint is available, doesn't that mean that everything is good? Or should it be in use and not available.

It's also not clear to me how to reproduce the bug consistently, sometimes my program works fine for a while, then suddenly panics with this message. I thought it was solved by upgrading the PicoSDK to version 1.5.0, after which the error disappeared for a while, but now its back and I have to clue how to approach it

[phryniszak]

The same on my side.
Program is working on Pico but fails on XIAO...

Bellow logs from the same program with incorrect and correct USB startup:

Welcome to app!
  Queue EP 00 with 1 bytes ...
  Prepare BufCtrl: [0] = 0x5401  [1] = 0x0000
wrote 1 bytes to USB
BUS RESET
USBD Bus Reset : Full Speed

USBD Setup Received 80 06 00 01 00 00 40 00 
  Get Descriptor Device
  Queue EP 80 with 18 bytes ...
  Prepare BufCtrl: [0] = 0xf412  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6012  [1] = 0x0000
  Short packet on buffer 0 with 18 bytes
Completed transfer of 18 bytes on ep 0 in
BUS RESET
USBD Xfer Complete on EP 80 with 18 bytes
  Queue EP 00 with 0 bytes ...
WARN: starting new transfer on already active ep 0 out
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000

*** PANIC ***

ep 0 out was already available

------------------------------


Welcome to app!
BUS RESET
USBD Bus Reset : Full Speed
BUS RESET
USBD Bus Reset : Full Speed

USBD Setup Received 80 06 00 01 00 00 40 00 
  Get Descriptor Device
  Queue EP 80 with 18 bytes ...
  Prepare BufCtrl: [0] = 0xf412  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6012  [1] = 0x0000
  Short packet on buffer 0 with 18 bytes
Completed transfer of 18 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 18 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
BUS RESET
USBD Xfer Complete on EP 00 with 0 bytes
USBD Bus Reset : Full Speed

USBD Setup Received 00 05 0E 00 00 00 00 00 
  Set Address
  Prepare BufCtrl: [0] = 0xf400  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 0 bytes

USBD Setup Received 80 06 00 01 00 00 12 00 
  Get Descriptor Device
  Queue EP 80 with 18 bytes ...
  Prepare BufCtrl: [0] = 0xf412  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6012  [1] = 0x0000
  Short packet on buffer 0 with 18 bytes
Completed transfer of 18 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 18 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 80 06 00 02 00 00 09 00 
  Get Descriptor Configuration[0]
  Queue EP 80 with 9 bytes ...
  Prepare BufCtrl: [0] = 0xf409  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6009  [1] = 0x0000
  Short packet on buffer 0 with 9 bytes
Completed transfer of 9 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 9 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 80 06 00 02 00 00 20 00 
  Get Descriptor Configuration[0]
  Queue EP 80 with 32 bytes ...
  Prepare BufCtrl: [0] = 0xf420  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6020  [1] = 0x0000
  Short packet on buffer 0 with 32 bytes
Completed transfer of 32 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 32 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 80 06 00 03 00 00 FF 00 
  Get Descriptor String[0]
  Queue EP 80 with 4 bytes ...
  Prepare BufCtrl: [0] = 0xf404  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6004  [1] = 0x0000
  Short packet on buffer 0 with 4 bytes
Completed transfer of 4 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 4 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 80 06 02 03 09 04 FF 00 
  Get Descriptor String[2]
  Queue EP 80 with 16 bytes ...
  Prepare BufCtrl: [0] = 0xf410  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6010  [1] = 0x0000
  Short packet on buffer 0 with 16 bytes
Completed transfer of 16 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 16 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 80 06 01 03 09 04 FF 00 
  Get Descriptor String[1]
  Queue EP 80 with 26 bytes ...
  Prepare BufCtrl: [0] = 0xf41a  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x601a  [1] = 0x0000
  Short packet on buffer 0 with 26 bytes
Completed transfer of 26 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 26 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 80 06 03 03 09 04 FF 00 
  Get Descriptor String[3]
  Queue EP 80 with 34 bytes ...
  Prepare BufCtrl: [0] = 0xf422  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6022  [1] = 0x0000
  Short packet on buffer 0 with 34 bytes
Completed transfer of 34 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 34 bytes
  Queue EP 00 with 0 bytes ...
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000
buf_status = 0x00000002
  Sync BufCtrl: [0] = 0xe000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 out
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 00 09 01 00 00 00 00 00 
  Set Configuration
  Open EP 02 with Size = 64
  Allocated 128 bytes at offset 0x180 (0x50100180)
  Open EP 81 with Size = 64
  Allocated 128 bytes at offset 0x200 (0x50100200)
  Queue EP 02 with 64 bytes ...
  Prepare BufCtrl: [0] = 0x5440  [1] = 0x0000
  VENDOR opened
  Bind EP 02 to driver id 0
  Bind EP 81 to driver id 0
  Queue EP 80 with 0 bytes ...
  Prepare BufCtrl: [0] = 0xf400  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6000  [1] = 0x0000
  Short packet on buffer 0 with 0 bytes
Completed transfer of 0 bytes on ep 0 in
USBD Xfer Complete on EP 80 with 0 bytes
USBD init on controller 0
sizeof(usbd_device_t) = 83
sizeof(tu_fifo_t) = 20
sizeof(tu_edpt_stream_t) = 40
VENDOR init
sizeof(hw_endpoint_t) = 32

------------------------------


Welcome to app!
  Queue EP 00 with 1 bytes ...
  Prepare BufCtrl: [0] = 0x5401  [1] = 0x0000
wrote 1 bytes to USB
BUS RESET
USBD Bus Reset : Full Speed

USBD Setup Received 80 06 00 01 00 00 40 00 
  Get Descriptor Device
  Queue EP 80 with 18 bytes ...
  Prepare BufCtrl: [0] = 0xf412  [1] = 0x0000
buf_status = 0x00000001
  Sync BufCtrl: [0] = 0x6012  [1] = 0x0000
  Short packet on buffer 0 with 18 bytes
Completed transfer of 18 bytes on ep 0 in
BUS RESET
USBD Xfer Complete on EP 80 with 18 bytes
  Queue EP 00 with 0 bytes ...
WARN: starting new transfer on already active ep 0 out
  Prepare BufCtrl: [0] = 0x7400  [1] = 0x0000

*** PANIC ***

ep 0 out was already available

[phryniszak]

Small updated. I cloned tinyusb from repository instead version from rp2040 and this error is gone.

[gemarcano]

I'm seeing this happen on a personal project with latest master as of 2024-02-14, but only while disconnecting and reconnecting USB using tud_disconnect and tud_connect, and only sometimes (really looks like a race condition). I am using FreeRTOS with SMP enabled (although all of my tasks are on core 2, by fixing affinity), and have a task dedicated to running tud_task. I have a separate "CLI" task that takes commands, and it is from this second task that I initiate a USB disconnect. Specifically, I call tud_disconnect, then call vTaskDelay(100) to give some time for the task with tud_task to run, then I call tud_connect.

Usually this process works fine. But sometimes I get a panic on core 2.

Here's the backtrace:

#0  _exit (status=status@entry=1) at /home/gabriel/Projects/pico-sdk/src/rp2_common/pico_runtime/runtime.c:187
#1  0x10019dd8 in panic (fmt=0x100688ec "ep %d %s was already available") at /home/gabriel/Projects/pico-sdk/src/rp2_common/pico_runtime/runtime.c:325
#2  0x20000718 in _hw_endpoint_buffer_control_update32 (ep=ep@entry=0x200077ac <hw_endpoints+160>, and_mask=and_mask@entry=0, or_mask=or_mask@entry=37888)
    at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/rp2040_usb.c:115
#3  0x2000075e in _hw_endpoint_buffer_control_set_value32 (value=37888, ep=0x200077ac <hw_endpoints+160>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/rp2040_usb.h:122
#4  hw_endpoint_start_next_buffer (ep=ep@entry=0x200077ac <hw_endpoints+160>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/rp2040_usb.c:212
#5  0x1001e7e0 in hw_endpoint_xfer_start (ep=0x200077ac <hw_endpoints+160>, buffer=<optimized out>, total_len=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/rp2040_usb.c:244
#6  0x1001e6f0 in hw_endpoint_xfer (total_bytes=<optimized out>, buffer=<optimized out>, ep_addr=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/dcd_rp2040.c:186
#7  dcd_edpt_xfer (rhport=<optimized out>, ep_addr=<optimized out>, buffer=<optimized out>, total_bytes=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/dcd_rp2040.c:529
#8  0x1001f224 in usbd_edpt_xfer (rhport=<optimized out>, rhport@entry=0 '\000', ep_addr=<optimized out>, buffer=buffer@entry=0x20006298 <_cdcd_itf+264> "Current controllers: 1\r\nog 4: 5.682810 - Connected with IP addre! ", 
    total_bytes=total_bytes@entry=24) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.c:1243
#9  0x1001f566 in tud_cdc_n_write_flush (itf=itf@entry=0 '\000') at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/class/cdc/cdc_device.c:208
#10 0x1001f864 in cdcd_xfer_cb (rhport=<optimized out>, ep_addr=<optimized out>, result=<optimized out>, xferred_bytes=2) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/class/cdc/cdc_device.c:471
#11 0x1001eaaa in tud_task_ext (timeout_ms=<optimized out>, in_isr=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.c:548
#12 0x10000396 in tud_task () at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.h:54
#13 usb_device_task () at /home/gabriel/Projects/snes_controllers_to_usb/firmware/src/main.cpp:156
#14 0x1001ab6c in xPortSysTickHandler () at /home/gabriel/Projects/FreeRTOS-Kernel/portable/ThirdParty/GCC/RP2040/port.c:759
#15 0xa5a5a5a4 in ?? ()

Some additional debugging information:

(gdb) p/x *ep

$27 = {configured = 0x0, rx = 0x0, ep_addr = 0x0, next_pid = 0x1, endpoint_control = 0x0, buffer_control = 0x0, hw_data_buf = 0x0, user_buf = 0x20006298, remaining_len = 0x18, xferred_len = 0x0, wMaxPacketSize = 0x0, 
  active = 0x1, transfer_type = 0x0, pending = 0x0}

Huh, that's... interesting. Looks like buffer_control is null. I just noticed that. Just happens to be that there's the rp2040 ROM at 0x0, so there's some arbitrary data there. On my pico, that first byte of memory looks like a stack pointer for ROM, specifically 0x20041f00. That's why the panic is happening, because the buffer available bit in ROM here just happens to be set.

At a higher level, what should happen if buffer_control is null? Is it just a bug? Or should hw_endpoint_start_next_buffer or some other function catch that and bail?

So somehow, at least for my case, something must be zero-ing out the buffer_control pointer. Now it's just a matter of figuring out what. Since it looks like a race, it might be difficult to trigger with the slowdown incurred by gdb conditional breaks...

[gemarcano]

Interestingly enough, I set up the rp2040's MPUs on both cores to deny access to the first 256 bytes of memory, and I'm seeing tinyUSB fault very early on in my overall program, while my linux computer is trying to assign it a device address. Specifically, tinyUSB is trying to deference a null pointer again, but this time it's endpoint_control.

Here's that stack trace:

#0  isr_hardfault () at /home/gabriel/Projects/pico-sdk/src/rp2_common/pico_standard_link/crt0.S:98
#1  <signal handler called>
#2  0x2000072c in hw_endpoint_start_next_buffer (ep=ep@entry=0x2000772c <hw_endpoints+32>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/rp2040_usb.c:175
#3  0x1001e810 in hw_endpoint_xfer_start (ep=0x2000772c <hw_endpoints+32>, buffer=<optimized out>, total_len=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/rp2040_usb.c:244
#4  0x1001e720 in hw_endpoint_xfer (total_bytes=<optimized out>, buffer=<optimized out>, ep_addr=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/dcd_rp2040.c:186
#5  dcd_edpt_xfer (rhport=<optimized out>, ep_addr=<optimized out>, buffer=<optimized out>, total_bytes=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040/dcd_rp2040.c:529
#6  0x1001f254 in usbd_edpt_xfer (rhport=<optimized out>, rhport@entry=0 '\000', ep_addr=<optimized out>, buffer=<optimized out>, total_bytes=total_bytes@entry=18)
    at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.c:1243
#7  0x1001f2be in _data_stage_xact (rhport=rhport@entry=0 '\000') at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd_control.c:106
#8  0x1001f35e in tud_control_xfer (rhport=rhport@entry=0 '\000', request=request@entry=0x20016610, buffer=0x10069ee0 <desc_device>, len=len@entry=18)
    at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd_control.c:125
#9  0x1001eed2 in process_get_descriptor (p_request=0x20016610, rhport=0 '\000') at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.c:991
#10 process_control_request (p_request=0x20016610, rhport=0 '\000') at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.c:717
#11 tud_task_ext (timeout_ms=<optimized out>, in_isr=<optimized out>) at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.c:516
#12 0x10000396 in tud_task () at /home/gabriel/Projects/pico-sdk/lib/tinyusb/src/device/usbd.h:54
#13 usb_device_task () at /home/gabriel/Projects/snes_controllers_to_usb/firmware/src/main.cpp:157
#14 0x1001ab9c in xPortSysTickHandler () at /home/gabriel/Projects/FreeRTOS-Kernel/portable/ThirdParty/GCC/RP2040/port.c:759

and here's ep:

(gdb) p *ep
$3 = {configured = false, rx = false, ep_addr = 128 '\200', next_pid = 1 '\001', endpoint_control = 0x0, buffer_control = 0x50100080, hw_data_buf = 0x50100100 "", user_buf = 0x200063ac <_usbd_ctrl_buf> "\022\001", 
  remaining_len = 18, xferred_len = 0, wMaxPacketSize = 64, active = true, transfer_type = 0 '\000', pending = 0 '\000'}

[gemarcano]

Yeah, there looks to be a (different?) bug here with endpoint_control. According to comments in src/portable/raspberrypi/rp2040/dcd_rp2040.c in function hw_endpoint_init:

  if ( num == 0 )
  {
    // EP0 has no endpoint control register because the buffer offsets are fixed
    ep->endpoint_control = NULL;

    // Buffer offset is fixed (also double buffered)
    ep->hw_data_buf = (uint8_t*) &usb_dpram->ep0_buf_a[0];
  }

This explicitly sets the endpoint_control pointer to null... and the code in other parts of the rp2040 port doesn't check anywhere if it's null or if the endpoint number is 0 and ignore it, etc. I'm going to try to wing it on my end to see if I can add those checks.

[gemarcano]

I managed to get past the endpoint_control null pointers. Just required two simple checks in the device code.

As for the buffer_control nonsense, it's turning out to be an interesting issue at least. Taking a further look as to what pointer ep is pointing to, the memory is stored in an array of endpoint control structures in hw_endpoints. Even though *ep reports that it's address is 0, the physical location was &hw_endpoints[2][1]. Or more specifically:

(gdb) p *ep

$27 = {configured = false, rx = false, ep_addr = 0 '\000', next_pid = 1 '\001', endpoint_control = 0x0, buffer_control = 0x0, hw_data_buf = 0x0, 
  user_buf = 0x20006338 <_cdcd_itf+264> "Current controllers: 1\r\nted with IP address 192.168.5.216\r\n> th ! ", remaining_len = 24, xferred_len = 0, wMaxPacketSize = 0, active = true, transfer_type = 0 '\000', 
  pending = 0 '\000'}
...
(gdb) p ep

$29 = (struct hw_endpoint *) 0x2000784c <hw_endpoints+160>
...
(gdb) p &hw_endpoints[2][1]

$39 = (struct hw_endpoint *) 0x2000784c <hw_endpoints+160>

Placing a gdb watch on the ep_addr entry shows that only two places modify that variable:

hw_endpoint_init
reset_non_control_endpoints

reset_non_control_endpoints gets called from USBCTRL_IRQ, which I'm guessing is happening in my case because I'm using tud_disconnect to disconnect the USB device so I can update the USB configuration (following the suggestion from #1904).

I haven't confirmed this yet, but my money is on some race condition between some ongoing tud_task transmission and this IRQ firing due to the disconnection.

[gemarcano]

I found something in the function hw_endpoint_xfer_start. There is a locking function here... but when I checked the implementation, this is what I found:

TU_ATTR_ALWAYS_INLINE static inline void hw_endpoint_lock_update(__unused struct hw_endpoint * ep, __unused int delta) {
  // todo add critsec as necessary to prevent issues between worker and IRQ...
  //  note that this is perhaps as simple as disabling IRQs because it would make
  //  sense to have worker and IRQ on same core, however I think using critsec is about equivalent.
}

So... the locking is not implemented. That definitely explains the bug I'm seeing.