Pasting large amounts of text to REPL on ItsyBitsy ESP32 results in dropped data

[RetiredWizard]

CircuitPython version

Adafruit CircuitPython 9.1.0 on 2024-07-10; Adafruit ItsyBitsy ESP32 with ESP32

Code/REPL

None

Behavior

When pasting more than about 500 characters into the REPL, either normal mode or CTRL-E paste mode, some of the characters being pasted get dropped.

Description

I started trouble shooting this in issue #9362 by pasting a 12.5K python script into the REPL using the Ctrl-E paste mode. The basic observation was:

When I paste code to a Ctrl-E repl of the ItsyBitsy ESP32 it consistently starts dropping characters after 255 characters. After skipping about 275 characters, another 124 characters are correctly received and then 937 characters are dropped followed by another 152 properly transmitted characters, etc.....

Additional information

I was able to improve the size of the file that could be pasted without error by decreasing the UART baud rate or increasing the console_uart_rx_buf size.

By commenting out the console echo of the pasted data, I was able to successfully paste the entire 12.5K script without error.

I also made a modification to common_hal_busio_uart_write module in ports/espressif/common-hal/busio/UART.c which I believe prevented the routine from waiting for the UART to become available when transmitting (echoing to the display) a character. This mod resulted in intermittent characters not being echo'd, however the REPL did receive the entire file without error. Increasing the SOC_UART_FIFO_LEN from 128 to 250 had the same result, I'm guessing this prevented the uart_ll_get_txfifo_len function from ever reporting the transmit buffer as full so the uart_writes were not held up.

After a lot of hacking and staring my current theory about what's happening is that the UART rx interrupts are coming in so fast and frequently that the UART is not able to echo the characters back out to the console. I think this results in the FIFO receive buffer filling up which causes ESP-IDF to disable the uart interrupts. That allows the transmit buffer to empty and display the first buffer full of characters, however while the interrupts are disabled, any characters being pasted are lost. The ESP-IDF then re-enables the interrupts when there is FIFO buffer space again and the cycle repeats.

Without the ability to get flow control to the CH9102 chip, I can't think of a great solution to this issue especially since it seems to be limited to the ESP32 boards. It should be noted that the issue doesn't impact RAW REPL mode, presumably because it doesn't echo data and therefore attempt to simultaneously use the UART for RX and TX functions.

[tannewt]

I'm not sure what we can do about this. We could increase the buffer but there will always be a limit to how much we can buffer. We could dynamically buffer but that'd add code size and complexity.

[dhalbert]

We should see if MicroPython has the same problem, and if not, what are they doing differently.

[RetiredWizard]

I was thinking that I could add a little "slip" into the UART write so that when it got backed up, it dropped output. I think that might be a better result than turning off the rx interrupts and dropping input data. I'll see if I can work up the idea in a PR so you can see what I'm thinking. If you don't like that option, marking this as a limitation of boards that don't support UART flow control might be the only option 😁

[RetiredWizard]

We should see if MicroPython has the same problem, and if not, what are they doing differently.

Micropython pastes a little more data before dropping input but behaves the same way. MP may use a slightly larger buffer, or perhaps the console output is slightly more efficient (doesn't check Webworkflow, BLE, etc streams).

[dhalbert]

code.circuitpython.org is being changed to to use Raw Mode instead of Raw Paste mode. Raw Mode does not echo. We talked a bit about this internally, and kind of came to the conclusion that it's not going to work well without delays, and no echo is the way to go if you have to do that.

[RetiredWizard]

In cases like code.circuitpython.org where you have control of the serial transmitter, you can insure the speed is controlled and gain some transmission confidence by confirming that each character transmitted echos back what you sent, before transmitting the next character. In that mode however, you completely negate any speed advantage from the internal uart/Circuitpython buffers.

The Raw Mode is probably a more speed efficient approach, although it adds the slight annoyance of sometimes leaving the board in Raw Mode, requiring an Alt-b (or power cycle) to make the board responsive again.

[dhalbert]

As I mentioned in #9433, this paste mode feature is not even available in CPython, and I don't think we necessarily have to accommodate boards like this with poor or no flow control. I think the easiest solution for an end user is to rate-limit the paste externally, for example, by using the inter-character or inter-line delays available in tio. If you use paste mode a lot, you can set up your terminal program to include enough delay.

from tio man page:

      -o, --output-delay <ms>

              Set output delay [ms] inserted between each sent character (default: 0).

       -O, --output-line-delay <ms>

              Set output delay [ms] inserted between each sent line (default: 0).

[RetiredWizard]

Yep, I'm good closing this. Thanks for putting up with my visit to Alice's wonderland 😁