It's too hard to flash using Pre-Generated Host Keys when a ESP32 is in Encrypted Firmware Release Mode (ESPTOOL-566)

[chipweinberger]

Related Issues

related issue: espressif/vscode-esp-idf-extension#853
related issue: espressif/esp-idf#10340

Edit: Sorry for all the issues. This issue might not require esptool.py changes, I think. Each issue has unique information in it, however.

Edit2: Actually, I think esptool.py needs to expose a secure_download_status command, in order to determine if a chip is in Secure Download Mode.

Problem

After you turn on Firmware Encryption (Release), it becomes very annoying to use esp_tool.py and idf.py.

There must be a simpler way!

Re-Flashing a device in Firmware Encryption (Development) Mode or Normal Mode:

idf.py flash

Re-Flashing a device in Firmware Encryption (Release) Mode:

espsecure.py encrypt_flash_data --aes_xts --keyfile /Users/charlesweinberger/Desktop/flash_encryption_keys/my_flash_encryption_key.bin --address 0x00000 --output /Users/charlesweinberger/Desktop/encrypted_bins/bootloader.bin ./build/bootloader/bootloader.bin

espsecure.py encrypt_flash_data --aes_xts --keyfile /Users/charlesweinberger/Desktop/flash_encryption_keys/my_flash_encryption_key.bin --address 0x300000 --output /Users/charlesweinberger/Desktop/encrypted_bins/jJamApp.bin ./build/jJamApp.bin

espsecure.py encrypt_flash_data --aes_xts --keyfile /Users/charlesweinberger/Desktop/flash_encryption_keys/my_flash_encryption_key.bin --address 0xf000 --output /Users/charlesweinberger/Desktop/encrypted_bins/partition-table.bin ./build/partition_table/partition-table.bin

espsecure.py encrypt_flash_data --aes_xts --keyfile /Users/charlesweinberger/Desktop/flash_encryption_keys/my_flash_encryption_key.bin --address 0x10000 --output /Users/charlesweinberger/Desktop/encrypted_bins/ota_data_initial.bin ./build/ota_data_initial.bin

/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf4.4_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py -p /dev/cu.SLAB_USBtoUART -b 460800 --before default_reset --after no_reset --chip esp32s3 --no-stub write_flash --flash_mode dio --flash_freq 80m --flash_size 8MB 0x0 /Users/charlesweinberger/Desktop/encrypted_bins/bootloader.bin 0x300000 /Users/charlesweinberger/Desktop/encrypted_bins/jJamApp.bin 0xf000 /Users/charlesweinberger/Desktop/encrypted_bins/partition-table.bin 0x10000 /Users/charlesweinberger/Desktop/encrypted_bins/ota_data_initial.bin

Describe the solution you'd like

idf.py flash! It should just do the right thing!

1. The partition-table is already in ./build.

2. We also know ifd.py can detect when a device is in `Secure Download Mode. Which it tells us:

Connecting....
Chip is ESP32-S3 in Secure Download Mode
WARNING: Stub loader is not supported in Secure Download Mode, setting --no-stub

3. When Secure Download Mode is detected, idf.py should throw up an error forcing us the specify if we want to flash raw or flash encrypted. Error msg:When Secure Download Mode is detected, you must manually specify if you want to flash encrypted or flash raw. Flags: --apply-encryption-key /path/to/key.bin or --flash-raw

4. to automate further, we could have a KConfig for specifying the location of the keyfile, similar to custom partition tables. A keyfile_path.txt will then be written into the build folder. idf.py flash would then consult this file, when it detects it needs to apply encryption. This would make idf.py flash work seamlessly despite the mode of the device.

Describe alternatives you've considered

Writing my own scripts to automate this...

[dobairoland]

Hi @chipweinberger. We are open for suggestions but all these "idf.py ... should do" type of suggestions cannot be implemented on the esptool side. And since you've opened an issue on the ESP-IDF side as well, makes no sense to keep it open. I'm sorry.

The official recommendation since I can remember was to follow https://docs.espressif.com/projects/esp-idf/en/latest/esp32/security/flash-encryption.html step-by-step and not go around and run commands intuitively. I think it makes sense to keep open and consider the suggestion on the ESP-IDF side.

Actually, I think esptool.py needs to expose a secure_download_status command,

There is already a command for that. The status can be read out by espefuse.py. It is not practical to expose every field as a separate command. One should argue that there should be another command which would set it, etc....

[chipweinberger]

There is already a command for that.

Which command? get_security_info throws an uncaught exception for me. But yes, I assume we could report the info in that command, if it worked in Firmware Release Mode?

charlesweinberger@Chips-Macbook-Pro-2 jamcorder_app % /Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf4.4_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py -p /dev/cu.SLAB_USBtoUART -b 460800 --before default_reset --after no_reset get_security_info
esptool.py v3.3.2
Serial port /dev/cu.SLAB_USBtoUART
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32-S3
Chip is ESP32-S3
Features: WiFi, BLE
Crystal is 40MHz
Traceback (most recent call last):
  File "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py", line 5482, in <module>
    _main()
  File "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py", line 5475, in _main
    main()
  File "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py", line 4795, in main
    read_mac(esp, args)
  File "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py", line 4305, in read_mac
    mac = esp.read_mac()
  File "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py", line 2161, in read_mac
    mac0 = self.read_reg(self.MAC_EFUSE_REG)
  File "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py", line 733, in read_reg
    val, data = self.command(self.ESP_READ_REG, struct.pack('<I', addr), timeout=timeout)
  File "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py", line 505, in command
    raise UnsupportedCommandError(self, op)
__main__.UnsupportedCommandError: Invalid (unsupported) command 0xa

[chipweinberger]

Ah, apparently you must always specify --no-stub.

Is there a reason we don't default to --no-stub for the get_security_info command?

Perhaps we should default to --no-stub for get_security_info, and have a flag to add the stub --force-stub....if that was ever needed.

/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf4.4_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py -p /dev/cu.SLAB_USBtoUART -b 460800 --before default_reset --after no_reset --chip esp32s3 --no-stub get_security_info
esptool.py v3.3.2
Serial port /dev/cu.SLAB_USBtoUART
Connecting....
Chip is ESP32-S3 in Secure Download Mode
Changing baud rate to 460800
Changed.
Enabling default SPI flash mode...
Flags: 0x00000684 (0b11010000100)
Flash_Crypt_Cnt: 0x7
Key_Purposes: (7, 2, 3, 0, 0, 0, 12)
Chip_ID: 9
Api_Version: 0
Staying in bootloader.

[mahavirj]

@chipweinberger

Ah, apparently you must always specify --no-stub.

If you have Secure DL mode enabled then executing arbitrary code is not possible through UART DL mode on the target. Hence, one must pass --no-stub option in the presence of the Secure DL mode on the target.

[radimkarnis]

This should happen automatically, the --no-stub argument is activated if secure download mode is detected:

...
Connecting....
Chip is ESP32-S2 in Secure Download Mode
WARNING: Stub loader is not supported in Secure Download Mode, setting --no-stub
Enabling default SPI flash mode...
...

[mahavirj]

Thanks for confirming @radimkarnis

[chipweinberger]

This should happen automatically

Yes, I think in this instance because I did not specify -chip esp32s3 the automatic --no-stub detection did not work. But I am not sure. For best results, I need to specify both, but sometimes it automatically works. I haven't fully figured it out.

[radimkarnis]

@chipweinberger what's weird is that in your case it didn't detect the Secure Download Mode at all, otherwise you would see Chip is ESP32-S3 in Secure Download Mode. That's why esptool tried to upload the stub and failed