arduino-esp32 2.0.4 breaks esp-web-tools

[jameszah]

I was trying to implement this (as I have done before) when kept getting these errors after the esp32 reboot.

23:28:30.382 -> ets Jun  8 2016 00:22:57
23:28:30.382 -> 
23:28:30.382 -> rst:0x1 (POWERON_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
23:28:30.382 -> configsip: 0, SPIWP:0xee
23:28:30.382 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
23:28:30.382 -> mode:QIO, clock div:2
23:28:30.382 -> load:0x3fff0030,len:1344
23:28:30.382 -> load:0xf01d020c,len:-2118480896
23:28:30.430 -> 1162 mmu set 00010000, pos 00010000
23:28:30.430 -> 1162 mmu set 00020000, pos 00020000
23:28:30.430 -> 1162 mmu set 00030000, pos 00030000
23:28:30.477 -> 1162 mmu set 00040000, pos 00040000
23:28:30.477 -> 1162 mmu set 00050000, pos 00050000
23:28:30.524 -> 1162 mmu set 00060000, pos 00060000
23:28:30.524 -> 1162 mmu set 00070000, pos 00070000
23:28:30.570 -> 1162 mmu set 00080000, pos 00080000
23:28:30.570 -> 1162 mmu set 00090000, pos 00090000
23:28:30.570 -> 1162 mmu set 000a0000, pos 000a0000
23:28:30.616 -> 1162 mmu set 000b0000, pos 000b0000
23:28:30.616 -> 1162 mmu set 000c0000, pos 000c0000
23:28:30.662 -> 1162 mmu set 000d0000, pos 000d0000
23:28:30.662 -> 1162 mmu set 000e0000, pos 000e0000
23:28:30.709 -> 1162 mmu set 000f0000, pos 000f0000
23:28:30.709 -> 1162 mmu set 00100000, pos 00100000
23:28:30.709 -> ets Jun  8 2016 00:22:57
23:28:30.709 -> 
23:28:30.709 -> rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
23:28:30.755 -> configsip: 0, SPIWP:0xee
23:28:30.755 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
23:28:30.755 -> mode:QIO, clock div:2
23:28:30.755 -> load:0x3fff0030,len:1344
23:28:30.755 -> load:0xf01d020c,len:-2118480896
23:28:30.755 -> 1162 mmu set 00010000, pos 00010000
23:28:30.802 -> 1162 mmu set 00020000, pos 00020000

Arduino 1.19 with arduino-esp32 2.04 and 2.03 will both program the esp32 fine, but when I grab the 4 files for esp32-web-tools, then the 2.04 files do not work -- they produce the stuff above.

Arduino produces the same programming strings here:

204 C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\tools\esptool_py\3.3.0/esptool.exe --chip esp32 --port COM7 --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 4MB 0x1000 C:\Users\James\AppData\Local\Temp\arduino_build_686088/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bootloader.bin 0x8000 C:\Users\James\AppData\Local\Temp\arduino_build_686088/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.partitions.bin 0xe000 C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\hardware\esp32\2.0.4/tools/partitions/boot_app0.bin 0x10000 C:\Users\James\AppData\Local\Temp\arduino_build_686088/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bin 
203 C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\tools\esptool_py\3.3.0/esptool.exe --chip esp32 --port COM7 --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 4MB 0x1000 C:\Users\James\AppData\Local\Temp\arduino_build_830660/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bootloader.bin 0x8000 C:\Users\James\AppData\Local\Temp\arduino_build_830660/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.partitions.bin 0xe000 C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\hardware\esp32\2.0.3/tools/partitions/boot_app0.bin 0x10000 C:\Users\James\AppData\Local\Temp\arduino_build_830660/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bin 

The files are slightly different sizes for bootloader.bin and ino.bin, and the other two are the same files.

And I'm using the identical manifest.json

{
  "name": "ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino",
  "new_install_prompt_erase": false,
  "builds": [
    {
      "chipFamily": "ESP32",
      "parts": [
        { "path": "ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bootloader.bin", "offset": 4096 },
        { "path": "ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.partitions.bin", "offset": 32768 },
        { "path": "boot_app0.bin", "offset": 57344 },
        { "path": "ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bin", "offset": 65536 }
      ]
    }
  ]
}

2.0.3 files work and 2.0.4 files do not.

This looks like a similar issue to this one (here):

#278

... and this one over at arduino-esp32

espressif/arduino-esp32#7212

There seems to be discussion about qio, dio, dout programming and the 40/80 flash frequency. The two arduino esptool strings are both dio above, even though Arduino is set to qio. Arduino can program it fine at 40 or 80.

Here is the 2.03 40 listing if there is something in there that a wise-man can see:

C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\tools\esptool_py\3.3.0/esptool.exe --chip esp32 --port COM7 --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size 4MB 0x1000 C:\Users\James\AppData\Local\Temp\arduino_build_830660/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bootloader.bin 0x8000 C:\Users\James\AppData\Local\Temp\arduino_build_830660/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.partitions.bin 0xe000 C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\hardware\esp32\2.0.3/tools/partitions/boot_app0.bin 0x10000 C:\Users\James\AppData\Local\Temp\arduino_build_830660/ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino.bin 
esptool.py v3.3
Serial port COM7
Connecting.....
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 0c:b8:15:f4:14:50
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00001000 to 0x00005fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x0000e000 to 0x0000ffff...
Flash will be erased from 0x00010000 to 0x00109fff...
Compressed 18528 bytes to 12759...
Writing at 0x00001000... (100 %)
Wrote 18528 bytes (12759 compressed) at 0x00001000 in 0.6 seconds (effective 231.4 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 119...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (119 compressed) at 0x00008000 in 0.1 seconds (effective 393.2 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 47...
Writing at 0x0000e000... (100 %)
Wrote 8192 bytes (47 compressed) at 0x0000e000 in 0.2 seconds (effective 419.4 kbit/s)...
Hash of data verified.
Compressed 1021040 bytes to 639310...
Writing at 0x00010000... (2 %)
Writing at 0x0001c720... (5 %)
Writing at 0x00028395... (7 %)
Writing at 0x00031ec1... (10 %)
Writing at 0x0004107c... (12 %)
Writing at 0x00046e7d... (15 %)
Writing at 0x0004d569... (17 %)
Writing at 0x00053360... (20 %)
Writing at 0x00058da9... (22 %)
Writing at 0x0005e522... (25 %)
Writing at 0x00063caf... (27 %)
Writing at 0x000690e3... (30 %)
Writing at 0x0006e444... (32 %)
Writing at 0x00073733... (35 %)
Writing at 0x00078abf... (37 %)
Writing at 0x0007dda1... (40 %)
Writing at 0x00083353... (42 %)
Writing at 0x000885a7... (45 %)
Writing at 0x0008d4f0... (47 %)
Writing at 0x00092cfe... (50 %)
Writing at 0x00099976... (52 %)
Writing at 0x0009ec10... (55 %)
Writing at 0x000a45dc... (57 %)
Writing at 0x000a9aea... (60 %)
Writing at 0x000aecaf... (62 %)
Writing at 0x000b4054... (65 %)
Writing at 0x000b94e1... (67 %)
Writing at 0x000be9d2... (70 %)
Writing at 0x000c42ec... (72 %)
Writing at 0x000ca090... (75 %)
Writing at 0x000cfa4c... (77 %)
Writing at 0x000d5400... (80 %)
Writing at 0x000ddea4... (82 %)
Writing at 0x000e7502... (85 %)
Writing at 0x000ec67c... (87 %)
Writing at 0x000f3249... (90 %)
Writing at 0x000f8930... (92 %)
Writing at 0x000fe05e... (95 %)
Writing at 0x00103465... (97 %)
Writing at 0x001092b4... (100 %)
Wrote 1021040 bytes (639310 compressed) at 0x00010000 in 15.2 seconds (effective 539.0 kbit/s)...
Hash of data verified.

Any ideas ... other than stick with 2.03?

[balloob]

What version of ESP Web Tools are you using?

[jameszah]

I knew I left out something -- 9.0.3

The setup is over here: https://github.com/jameszah/ESP32-CAM-RocketCam

<h2>One Click Installer 2.0.4 (doesn't work)</h2>

<script type="module" src="https://unpkg.com/esp-web-tools@9.0.3/dist/web/install-button.js?module"></script>   
<esp-web-install-button manifest="installer/manifest.json"></esp-web-install-button>   

<h2>One Click Installer 2.0.3 </h2>

<script type="module" src="https://unpkg.com/esp-web-tools@9.0.3/dist/web/install-button.js?module"></script>   
<esp-web-install-button manifest="installer203/manifest.json"></esp-web-install-button>   

[balloob]

Can you try flashing it with https://espressif.github.io/esptool-js/ ? If that doesn't work, it's an upstream bug in https://github.com/espressif/esptool-js/

[jameszah]

When I flash on the just 2.0.4 bootloader.bin at 0x1000 onto a esp32 flashed with 2.0.3 version , it fails with the

21:16:05.532 -> rst:0x1 (POWERON_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
21:16:05.532 -> configsip: 0, SPIWP:0xee
21:16:05.532 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
21:16:05.579 -> mode:QIO, clock div:2
21:16:05.579 -> load:0x3fff0030,len:1344
21:16:05.579 -> load:0xf01d020c,len:-2118480896
21:16:05.579 -> 1162 mmu set 00010000, pos 00010000
21:16:05.579 -> 1162 mmu set 00020000, pos 00020000

Then I flash the 2.0.3 bootloader.bin, and it works again.
Then flash the 2.0.4 ino.bin at 0x10000 (still with the 2.0.3 bootloader), and it works.
Then flash 2.0.4 bootloader.bin, and it fails again.

2.0.3 bootloader is 18,560 bytes May 4, 2022
2.0.4 bootloader is 18,880 bytes Jul 6, 2022

So esptool (esptool_py\3.3.0/esptool.exe) that Arduino uses knows how to do it, but not esptool_js?

[TD-er]

Can you flash it to an ESP node using the IDE and then read the flashed data back from it using esptool.py.
With that file, can you try to use the webtools to flash another node?

If that does work, then it is very likely the same bug/issue as is being discussed in various issues when searching for "qio" or any of the other flash modes.

In short:
The flash mode, flash size and flash speed will be altered using flash tools like esptool.py while flashing.
However the web-tool does not.
On top of that, the flash mode and the SPI mode for PSRAM are probably different too, even though you might not even use PSRAM.
By setting the flash mode in the IDE to QIO, you might get a working system again... as long as all pins needed for QIO are connected to the flash on your board. (have not yet seen ESP32 boards that haven't)

[Jason2866]

Flashing individual files with esp-web-tools for ESP32x is always a more or less wrong way. Doing this way you just have one correct variant for flash mode, flash size and flash mode!!. Since core 2.0.4. does not have set any default value for any of these, it will fail if you try to flash. You have to create a combined (merged) firmware. Esptool.py can be used to do this.
To be clear not a issue from web esp tools, nor can it solve that, since web esp tools cant know what setting are needed!

[balloob]

@Jason2866 we should add some guidance to our README maybe. Should we just always encourage using a single firmware that is a merged file?

[Jason2866]

@balloob Yes, a single firmware file is the only way to get predictable working installs.

[jameszah]

So I switched back to 2.0.4, re-flashed it with Arduino,

... and then used this to read it off the flash.

C:\Users\James\esp\esp-idf>C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\tools\esptool_py\3.3.0/esptool.exe --chip esp32 --port COM7 --baud 460800 read_flash 0 0x400000 my_4mb.img
esptool.py v3.3
Serial port COM7
Connecting.......
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 0c:b8:15:f4:14:50
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
4194304 (100 %)
4194304 (100 %)
Read 4194304 bytes at 0x0 in 100.2 seconds (335.0 kbit/s)...
Hard resetting via RTS pin...

... which seemed to work. But when I used esp-web-tools to flash that big file using this:

{
  "name": "ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino",
  "new_install_prompt_erase": false,
  "builds": [
    {
      "chipFamily": "ESP32",
      "parts": [
        { "path": "my_4mb.img", "offset": 0 }
      ]
    }
  ]
}

... it would always timeout at 98%. And it did the same thing when using https://espressif.github.io/esptool-js/.

I tried it all again with a 3mb file -- I'm using 3mb Huge App, and not using the spiffs of littlefs. But it times out again at 98%.

Although the program does run correctly. Although I was re-flashed to a previously working chip. But not elegant.

Is there a merge procedure for bootloader, partitiion,s ino.bin plus boot_app0.bin into a single file?

Or any advice on this timeout business?

[TD-er]

This is what I use to generate a single bin file using PlatformIO: https://github.com/letscontrolit/ESPEasy/blob/mega/tools/pio/post_esp32.py

For example for my 16M module: (DIO mode 40 MHz, which is set in both the bootloader file and DIO flash mode)

Using esptool.py arguments: --chip esp32 merge_bin -o C:\GitHub\TD-er\ESPEasy\.pio\build\max_ESP32_16M8M_LittleFS_ETH/ESP_Easy_mega_20220905_max_ESP32_16M8M_LittleFS_ETH.factory.bin --flash_mode dio --flash_freq 40m --flash_size 16MB 0x1000 C:\Users\gijsn\.platformio\packages\framework-arduinoespressif32\tools\sdk\esp32\bin\bootloader_dio_40m.bin 0x8000 C:\GitHub\TD-er\ESPEasy\.pio\build\max_ESP32_16M8M_LittleFS_ETH\partitions.bin 0xe000 C:\Users\gijsn\.platformio\packages\framework-arduinoespressif32\tools\partitions\boot_app0.bin 0x10000 C:\GitHub\TD-er\ESPEasy\.pio\build\max_ESP32_16M8M_LittleFS_ETH/ESP_Easy_mega_20220905_max_ESP32_16M8M_LittleFS_ETH.bin

In short, it does generate a command line for esptool.py with explicitly setting the flash parameters and using the merge_bin option.

But this might still be different from flashing to a node and reading it back as initially the flash settings changed on the fly using esptool (or ArduinoIDE) are based on what the flash tool detects. The combined bin file is based on what you're setting.
So you might need to experiment.
But I think setting to qio and 40 MHz is probably the best chance for success.

Oh and you can still split your raw bin file if it is giving timeouts. Just use the offset positions at for example 0, 1M, 2M, 3M and split your bin file in 4 parts of 1M each.

[Jason2866]

But I think setting to qio and 40 MHz is probably the best chance for success.

No, it is not. It has to be here dio The flash mode which is read from the 2nd stage bootloader is set in the firmware app.bin during compile. It is a safe bet to use dio here.
And if you want qio you have to use the precompiled bootloader qio. If not used the device will boot. It is not a show stopper.

Thats why @TD-er uses for Espeasy this script and we (Tasmota) a similar one to create the one file firmware.

The ESP32-S3 with OPI flash introduces new issues. Only the 100% correct combination of Flash and PSRAM type will result in a booting S3 with PSRAM.
OPI Flash needs to be flashed with option flash_mode set to dout ;-)

[jameszah]

Hi, thanks for the info - I had never touched that qio/dio/... setting before. The speed/method of writing the flash would depend on hardware, but I would have thought that what gets written would end up the same.

My workaround was to use the the traditional 4 file esp-web-tools method, but replace the bootloader with the version generated by 2.0.3. It will flash and run correctly.

The issue seems to be on the table over at arduino-esp32.

Thanks.

[Jason2866]

My workaround was to use the the traditional 4 file esp-web-tools method, but replace the bootloader with the version generated by 2.0.3. It will flash and run correctly.

I would not do this. You probably run in issues. You have fixed magic bytes settings which will not work for all devices which do differ in flash size and flash frequency.
It is just wrong. Thats the reason why changed in 2.0.4. It is a bug fix!!

In IDF this problem does not occur since the bootloader is compiled WITH the project settings together with the project source. And the needed flash size correction is done via esptool.py when flashing.

[jameszah]

ok, that wasn't too hard

esptool command line with the 4 files from the 2.0.4 compile

C:\ArduinoPortable\arduino-1.8.19\portable\packages\esp32\tools\esptool_py\3.3.0/esptool.exe --chip esp32 merge_bin -o srt.4.ino.img --flash_mode dio --flash_freq 40m --flash_size 4MB 0x1000 srt.4.ino.bootloader.bin 0x8000 srt.4.ino.partitions.bin 0xe000 boot_app0.bin 0x10000 srt.4.ino.bin

And drop the output for esp-web-tools to write as 1 file

{
  "name": "ESP32-CAM-Video-Recorder-junior-60x.4.7.srt.4.ino",
  "new_install_prompt_erase": false,
  "builds": [
    {
      "chipFamily": "ESP32",
      "parts": [
        { "path": "srt.4.ino.img", "offset": 0 }
      ]
    }
  ]
}

[balloob]

Given that esptool.js, aims to be a port of esptool.py, I would still consider this a bug upstream 👍 If esptool.py can figure it out, so should esptool.js

[TD-er]

@balloob Please keep in mind that esptool.py when run from the IDE does have other bootloader bin files at its proposal.
Not sure if the tool does actually fetch other files, or that it may patch them on the fly, but it might be something to be aware of.

[Jason2866]

@balloob @TD-er is right. When compiling Arduino firmware there all variants of the precompiled bootloader files forr the different MCU types in this folders These files are Arduino specific! If a firmware is compiled with IDF the files will be different and builded new with the specific options set in IDF. There are NO bootloaders precompiled at all in IDF. Esptool.py "just" uses thes files to flash the device correctly (and does header patching). But when the wrong file is provided header patching is useless!
In short, user has to take care to provide the correct files or the combined firmware.

[balloob]

I’ve opened a PR for the README #298