Nonfunctional Launch of gbdtui, crashing and inability to debug at all (OCD-835)

[dmglogowskiOS]

Development Kit

Custom

Module or chip used

ESP-32-S3-WROOM-2

Debug Adapter

board/esp32s3-builtin.cfg

OpenOCD version

v0.12.0-esp32-20230419

Operating System

Linux / Windows / Docker Container

Using an IDE ?

VSCode and Terminal

OpenOCD command line

openocd "-f board/esp32s3-builtin.cfg"

JTAG Clock Speed

default

ESP-IDF version

V5.1.1

Problem Description

We are not sure how to reproduce the issue without our hardware configuration.

However this issue persists throughout all debugging envoirements we attempted:

• Windows 10,
• Docker Container running on Windows 10
• Linux(Ubuntu 22.04)
• Docker Container running on Linux(Ubuntu 22.04)

Whether attempting to debug from a terminal launched by the system or via the terminal launched from Visual Studio Code, the behavior is the same.

Debug Logs

Open OCD:

https://gist.github.com/dmglogowskiOS/1c13a2b92253944fa739f8ab4f7887a9

GDB:

Cannot find/Cannot create the log file

Expected behavior

Launch of a functional gbdtui instance and ability to debug.

Screenshots

Screencast.from.21.09.2023.13.04.24.webm

Video showing the experienced Behavior.

[igrr]

Error: 950 10 libusb_helper.c:179 jtag_libusb_open(): libusb_open() failed with LIBUSB_ERROR_ACCESS
Error: 951 10 esp_usb_jtag.c:646 esp_usb_jtag_init(): esp_usb_jtag: could not find or open device!

This usually indicates that OpenOCD cannot find the USB device of the built-in USB Serial/JTAG adapter of the ESP32-S3.

Could you check in Windows Device Manager, which device (VID:PID) appears in the system when you plug in your board? Or on Linux, run lsusb with and without the board plugged in.

[dmglogowskiOS]

Under Linux the following appears:

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 046d:c534 Logitech, Inc. Unifying Receiver
Bus 001 Device 002: ID 303a:1001 Espressif USB JTAG/serial debug unit
Bus 001 Device 004: ID 8087:0033 Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And the device is recognized and can be flashed/monitored via /dev/ttyACM0

In Windows the behavior is the same , the device appears, is recognized and can be flashed/monitored via it's associated COM Port.

[igrr]

Thanks, a couple more things to check:

1. On Linux, have you added the udev rule for this device?

   On Linux adding OpenOCD udev rules is required and is done by placing the following udev rules file in the /etc/udev/rules.d folder.

2. On Windows, do you have the driver installed? Could you please also try running Zadig to confirm the device is configured with "WinUSB" as a driver, and not some other driver.

[dmglogowskiOS]

Under Linux the Udev rules have been copied correctly. I had already thought that might have been an issue and triple checked that before I opened the issue.

On Windows the WinUSB Driver is selected for the device.

[dmglogowskiOS]

Additionally:
Surprisingly enough it occasionally does work somewhat under a Docker Container on Windows 10, however it seems to get stuck when trying to continue past the first break point, and not as in "it crashes and an error message comes up" but as in "it stops and I cannot even exit it manually."

However that only happens upon a fresh rebuild of the container then never again on the same container, it happened once on Linux and never on Windows.

[igrr]

I see, thanks for the extra details!

@erhankur do you have any clue what else can lead to the device not being found? (libusb_open() failed with LIBUSB_ERROR_ACCESS)

[erhankur]

@dmglogowskiOS To run OpenOCD on Linux, a user must be a member of the plugdev group. If you try to run it with a root user, it can help determine if the user requires special privileges to execute the program.

About Windows, If you repeatedly attempt to install the driver, it could lead to problems. My suggestion is to uninstall until you revert to the default driver and then reinstall WinUSB with Zadig once more. Additionally, before testing, it's a good idea to flash the "hello world" example to rule out any application-related issues.

[dmglogowskiOS]

@erhankur
Thank you for the possible solutions.
Currently we are focusing on getting the debugger to work on linux.
We attempted to run openocd both with my user(after ensuring it was in the plugdev group) and with a root user.

In both cases the behavior what we experienced was exactly the same as the behavior I already documented earlier in this issue.

[erhankur]

From the documentation:

USB-JTAG: GPIO 19 and 20 are used by USB-JTAG by default. In order to use them as GPIOs, USB-JTAG will be disabled by the drivers.

Can be the case for you?

We didn't get any user issue for the S3 but worth to check this link. #241

You said somehow it worked on windows so can not be relevant but it is important to make sure you didn't burn the efuse to use jtag through 4 pins.By default usb-jtag is enabled. You can see the latest state of the fuses by getting the summary.

And last thing i can suggest, please check your usb cable. I remember similar failures with different cables.

[dmglogowskiOS]

The only things connected to GPIO19 and GPIO20 is the USB-JTAG through some noise protection. So that shouldn't be an issue.
It did occasionally work on windows, yes. And in all our envoirements we are capable of flashing and monitoring, just never able to debug.

I will have to try my way through potential solutions in #241 and I will be back with any additional info.

Regarding USB Cables:
We used multiple different cables, 2 different PCBs and 3 different Computers, so it is unlikely that it is relating to that.

[dmglogowskiOS]

Regarding the possible solutions in #241 :

• Attempting the various commands didn't do anything of not
• However there is potential of one of the GPIO Strapping pins being an issue as we use that one for our CS for onboard ETH, I will look into that with our Hardware DesignerUnlikely to be an issue.

Our HW Designer / Dev Team Lead suggested the following ideas which we tried:

• We tried to debug the hello_world example and the same behavior occured
• We tried to reinstall the esp-idf toolchain (Notes taken and attached) and retried debugging the hello_world example and still the same behavior occured.
  esp32s3-debugging.md

Also we have a module with PSRAM, could that be an issue?

[erhankur]

In the log you shared before we just saw usb error. Can you also share the log you have during debugging?

[dmglogowskiOS]

I still haven't figured out where gbd saves the log, if at all sadly.

However in the meantime I tried again and it now seems reproduceable for a session to at least partially succeed to the first break point but not further.
Here a log from the openOCD session of the hello_world example:
https://gist.github.com/dmglogowskiOS/2cad64800f88f681fc77d2a6d8bdca70
And here the log of the openOCD session of our internal Code:
https://gist.github.com/dmglogowskiOS/174c38253b00491b617e7c929595976f

This behavior seems now reproducable and mirrors the behavior we experienced on fresh containers under Windows and attempting to debug under Windows directly.

The only change between earlier today and now was a simple reboot of the linux machine.