Can't flash ESP32 over RFC2217 (ESPTOOL-243)

[ferrarimarco]

Operating system

macOS 13.3.1
Linux (Debian)

Python version

3.6.9 (via the esp-idf Docker container)

What Chip

ESP32

What development board or other hardware is the chip attached to

Adafruit huzzah 32

Is anything else attached to the development board, except for the serial flasher connections?

No.

Are you running esptool.py from an IDE such as Arduino or Eclipse?

No IDE. Running the esp-idf Docker container.

Full esptool.py command line that was run:

cd /opt/esp/idf/components/esptool_py && /opt/esp/tools/cmake/3.16.4/bin/cmake -D IDF_PATH="/opt...s" -D WORKING_DIRECTORY="/workspace/build" -P /opt/esp/idf/components/esptool_py/run_serial_tool.cmake
esptool.py esp32 -p rfc2217://localhost:3000?ign_set_control -b 460800 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 40m --flash_size 4MB 0x8000 partition_table/partition-table.bin 0x1000 bootloader/bootloader.bin 0x10000 example-project.bin

Full output from esptool.py

esptool.py v3.1-dev
Serial port rfc2217://localhost:3000?ign_set_control
Connecting....
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, Coding Scheme None
Crystal is 40MHz
MAC: 30:ae:a4:19:ee:44
Uploading stub...
Running stub...
Traceback (most recent call last):
  File "/opt/esp/idf/components/esptool_py/esptool/esptool.py", line 4535, in <module>
    _main()
  File "/opt/esp/idf/components/esptool_py/esptool/esptool.py", line 4528, in _main
    main()
  File "/opt/esp/idf/components/esptool_py/esptool/esptool.py", line 4046, in main
    esp = esp.run_stub()
  File "/opt/esp/idf/components/esptool_py/esptool/esptool.py", line 793, in run_stub
    p = self.read()
  File "/opt/esp/idf/components/esptool_py/esptool/esptool.py", line 377, in read
    return next(self._slip_reader)
StopIteration
CMake Error at run_serial_tool.cmake:50 (message):
  /opt/esp/python_env/idf4.3_py3.6_env/bin/python
  /opt/esp/idf/components/esptool_py/esptool/esptool.py --chip esp32 failed


FAILED: CMakeFiles/flash
cd /opt/esp/idf/components/esptool_py && /opt/esp/tools/cmake/3.16.4/bin/cmake -D IDF_PATH="/opt/esp/idf" -D SERIAL_TOOL="/opt/esp/python_env/idf4.3_py3.6_env/bin/python /opt/esp/idf/components/esptool_py/esptool/esptool.py --chip esp32" -D SERIAL_TOOL_ARGS="--before=default_reset --after=hard_reset write_flash @flash_args" -D WORKING_DIRECTORY="/workspace/build" -P /opt/esp/idf/components/esptool_py/run_serial_tool.cmake
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

Do you have any other information from investigating this?

The command always errors.

My setup is the following:

1. ESP32 board connected to a MacBook.

2. Exposing the serial port with the esptool RFC2217 server: python3 .env/esp_rfc2217_server.py -p 3000 /dev/cu.SLAB_USBtoUART

3. (in another shell) Remote SSH forward the local 3000 to a remote host: ssh -R 3000:localhost:3000 remote.host.com

4. Try flashing the board with idf.py:

   docker run --rm -it \
   	--network host \
   	-v "$(pwd)":/workspace \
   	-w="/workspace" \
   	espressif/idf:release-v4.3 \
   	idf.py \
   	-p rfc2217://localhost:3000?ign_set_control \
   	flash

Is there any other information you can think of which will help us reproduce this problem?

Using the "plain" pyserial rfc2217_server.py makes no difference.

Can you help me debug this? The stack trace just points me to that line, and I cannot deduce what's happening.

Thanks!

[kumekay]

@ferrarimarco Thank you for the issue,
I've tried to reproduce the issue, but unfortunately wasn't able to get the same error.

Do I understand correctly that you are running Docker on the remote Linux machine (because --host network mode only works on Linux hosts and not implemented in docker-for-mac or docker-for-windows)? Just to be sure, is the network connection was fast enough, i.e. at least a few Mbits/s to reliably deliver data at 460800 (with all transport overhead)?
Did you try to flash with a lower bitrate, say 115200? Or, because in general communication between the device works (mac and size are detected correctly) and the problem only occurs after uploading the stub, could you try to flash using esptool.py directly with --no-stub flag?

[ferrarimarco]

Hi @kumekay !

Thanks for your support. See my replies below :)

Do I understand correctly that you are running Docker on the remote Linux machine (because --host network mode only works on Linux hosts and not implemented in docker-for-mac or docker-for-windows)?

Correct

Just to be sure, is the network connection was fast enough, i.e. at least a few Mbits/s to reliably deliver data at 460800 (with all transport overhead)?

Yup, more than hundreds of Mbit/s downstream and upstream.

Did you try to flash with a lower bitrate, say 115200? Or, because in general communication between the device works (mac and size are detected correctly) and the problem only occurs after uploading the stub, could you try to flash using esptool.py directly with --no-stub flag?

I tried with a lower bitrate as well (tested with 115200, same result).

Didn't try by flashing using esptool.py directly, only via idf.py. What is the --no-stub flag doing? Just to be sure that I invoke esptool.py correctly.

Thanks!

[kumekay]

Stub is a tiny flasher that is loaded to the chip memory on every flash to add extra features that ROM bootloader doesn't have, for example flashing on baud rates higher than 115200. Since it fails after stub upload, maybe the failure somehow related to it.

[ferrarimarco]

Thanks for the support @kumekay . I tried many times, but could not reproduce the issue. Closing for now. If it happens again, I'll reopen.

[cchaz003]

I have this exact same issue. Were you ever able to resolve it? Same idea, running through a docker container on a macbook. Was working just fine until the I reset my docker and it had to pull the newest image of the container. Not sure exactly what changed between docker images but it no longer flashes and I receive the exact same error as shown above.

[kumekay]

@cchaz003 What is the version of docker image/idf are you using? Maybe something else has changed? Did you try to run esptool.py directly with --no-stub option?

[KlausPopp]

I have the same problem. My client is in the internet (actually in a gitpod), my server is in the office, so the connection is via internet. The problem appeared when our internet connection had some performance issues.
Adding --no-stub solves the problem, however, the programming is then very very slow.
Any hint how to get it working without --no-stub?

[KlausPopp]

I traked down the problem and saw that there is a very short timeout of 0.05s applied after esptool.py output Stub running....

This causes problems on slow connections. I worked around it as follows:

esp/idf/components/esptool_py/esptool/esptool.py:

    def command(self, op=None, data=b"", chk=0, wait_response=True, timeout=DEFAULT_TIMEOUT):
        saved_timeout = self._port.timeout
        new_timeout = min(timeout, MAX_TIMEOUT)
 added>>>       new_timeout = max(timeout, 1)