esp_ota + encrypted firmware example (IDFGH-4334)

[kshyshkin]

Hello. Have anybody example how make ota update with encrypted firmware using esp_ota_begin(), esp_ota_write(); esp_ota_end in release/v4.2 esp-idf ?

[mahavirj]

@kshyshkin

Default OTA examples shipped with ESP-IDF (ref: https://github.com/espressif/esp-idf/tree/master/examples/system/ota) should work with flash encryption enabled case as well. esp_ota_write uses underlying partition manager API esp_partition_write, which transparently handles encrypted flash writes.

If you are facing any specific problem please feel to post detailed report per github issue template.

[kshyshkin]

@mahavirj Thank you for your reply. Problem with esp_ota_write for pre-encrypted firmware. esp_ota_write check magic byte and etc as in non encrypted firmware. I try esp_ota_write_offset it works, but after in esp_ota_end i got same situation with checking magic byte. I cant find how solve problem with ota update with pre-encrypted firmware. I see in documentation that esp_partition_write_raw function, i think it can help me solve this problem, bu i cant find in which release this function burn. Or i mistake?

[mahavirj]

@kshyshkin Can you please explain more details about your use-case? Do you have flash encryption enabled on device? If yes, any specific reason to flash pre-encrypted firmware during OTA update?

[kshyshkin]

@mahavirj Yes all secure features on controller on flash encryption (RELEASE), secure boot reflashable, all needed fuses on. But i have OTA (not through webserver), with intermediate device. When i need update by OTA, i make project, sighn fw with pem key, encrypt fw by digest from pem key, and send to intermediate device. Intermediate device send pre-encrypted fw to esp32 by custom interface and protocol dividing the firmware by chunks. I need that from my host computer to esp32 fw goes pre-encrypted. Thats why i need store pre-encrypted fw by chunks to OTA partition.

[mahavirj]

@kshyshkin

At this moment, pre-encrypted firmware write is not supported in app_update component.

I see in documentation that esp_partition_write_raw function, i think it can help me solve this problem, bu i cant find in which release this function burn.

This API is part of latest master branch and it will reflect in ESP-IDF v4.3 release.

esp-idf/components/spi_flash/partition.c

Line 423 in 178b122

esp_err_t esp_partition_write_raw(const esp_partition_t* partition,

Probably we can add configuration option to write pre-encrypted firmware image in OTA component, will keep you posted on this.

[kshyshkin]

@mahavirj Thank you for your work. I will be wait this option. It will be awesome, because i see that many issues by this feature.

[AxelLin]

This API is part of latest master branch and it will reflect in ESP-IDF v4.3 release.

esp-idf/components/spi_flash/partition.c

Line 423 in 178b122

esp_err_t esp_partition_write_raw(const esp_partition_t* partition,

Probably we can add configuration option to write pre-encrypted firmware image in OTA component, will keep you posted on this.

Hi @mahavirj
Any progress on this?
esp_partition_write_raw API is available now, but there is no config option to allow esp_https_ota() use it.

[shubhamkulkarni97]

Hi @AxelLin,
We had tried to OTA with pre encrypted firmware using raw_write APIs, however there are more dependencies that should be considered.
For example, if the firmware is encrypted the image magic byte present in the header will be changed due to encryption. In order to verify magic byte, we need to write received data to flash and re-read decrypted data. Writing data to flash without verifying magic byte doesn't seem to be a feasible approach.

We are checking possible issues with this feature in esp_https_ota. I'll keep you posted with the updates.

Thanks,
Shubham

[AxelLin]

any specific reason to flash pre-encrypted firmware during OTA update?

A common use case is the customer's device cannot connect to internet so it cannot reach OTA server.
If it's necessary to provide binary to customers for bug fix, an unencrypted binary is not an option.

[AxelLin]

Hi @AxelLin,
We had tried to OTA with pre encrypted firmware using raw_write APIs, however there are more dependencies that should be considered.
For example, if the firmware is encrypted the image magic byte present in the header will be changed due to encryption. In order to verify magic byte, we need to write received data to flash and re-read decrypted data. Writing data to flash without verifying magic byte doesn't seem to be a feasible approach.

Hi @shubhamkulkarni97
This seems not a big problem.
If you must write the received data to flash for verifying image magic,
it just change the checking image magic from before writting to after writting.

BTW, do you have patch to test updating encrypted firmware even without checking image magic?
I tried replace esp_partition_write() with esp_partition_write_raw() but it does not work.
In esp_ota_end the esp_image_verify failed.

[AxelLin]

@shubhamkulkarni97

Could you take a look what could be wrong in my changes?
https://gist.github.com/AxelLin/fc509e11f1da3b68bad97d6cd68b8777

The changes including replace esp_partition_write with esp_partition_write_raw,
and ignore image header magic checking (for testing purpose and simplify the changes).
The OTA processing seems done, but in esp_ota_end the esp_image_verify failed.

[AxelLin]

Hi @shubhamkulkarni97 @mahavirj

I figure out the problem now.
Encrypt the same image to different flash offset result in different encrypted-binary.
When your current image is running at ota_0, the ota image will be write to ota_1 then you need to use
the encrypted-binary generated for the ota1 offset. If use encrypted-binary generated for the ota0 offset,
in esp_ota_end the esp_image_verify failed.

This is an annoying issue because user needs to know what's current partition to decide which image to download.
(And multiple encrypted-binary for each ota partition offset is required)

[AxelLin]

For example, if the firmware is encrypted the image magic byte present in the header will be changed due to encryption. In order to verify magic byte, we need to write received data to flash and re-read decrypted data. Writing data to flash without verifying magic byte doesn't seem to be a feasible approach.

@shubhamkulkarni97
I re-read the code and don't understand what's your concern.
The flash is erased either in esp_ota_begin() or esp_ota_write().
i.e. flash is erased before checking image header anyway.
For encrypted firmware, you can check the image header after the first call of writing.
e.g.

            ret = esp_partition_write_raw(it->part, it->wrote_size, data_bytes, size);
            if(ret == ESP_OK){
#if 1
               // read back magic byte from flash
               if (it->wrote_size == 0 && it->partial_bytes == 0 && size > 0) {
                       uint8_t magic_byte;
                       ret = esp_partition_read(it->part, it->wrote_size, &magic_byte, 1);
                       if (ret != ESP_OK) {
                               return ret;
                       }

                       if (magic_byte != ESP_IMAGE_HEADER_MAGIC) {
                               ESP_LOGE(TAG, "OTA image has invalid magic byte (expected 0xE9, saw 0x%02x)", magic_byte);
                               return ESP_ERR_OTA_VALIDATE_FAILED;
                       }
               }
#endif

                it->wrote_size += size;
            }

[mahavirj]

@AxelLin

You are right about dependency of flash offset address in flash encryption scheme, it comes due to "key tweaking" as highlighted here.

We have been discussing internally on enabling "pre-encrypted firmware distribution" scheme and so far we are considering this to be entirely different than "platform flash encryption" feature.

• Primarily because, even for devices where flash encryption is not enabled, it could be requirement that firmware image over OTA is still encrypted in nature.
• Additionally for devices where flash encryption is enabled such that key is unique to device and generated by "bootloader" on first boot (recommended for production use-cases), pre-encryption with device key is not possible/feasible approach.

We are still at early discussion stage but probably approach will be to create "pre-encrypted" OTA image format, define encryption scheme, credential storage (use of "DS" peripheral or other approach) etc. This will be format of image for "on-the fly" decryption before writing to device flash, if flash encryption is enabled then device may "re-encrypt" image with platform key before writing to flash.

If you have any feedback on this, please feel free to share.

[AxelLin]

We are still at early discussion stage but probably approach will be to create "pre-encrypted" OTA image format, define encryption scheme, credential storage (use of "DS" peripheral or other approach) etc.

Note ESP32 does not has "DS" peripheral.

[AxelLin]

Hi @mahavirj

For firmware image, it can check the ESP_IMAGE_HEADER_MAGIC to recognize it's a vaild image.
For SPIFFS image, is there a standard way to recognize if a file is spiffs image?

[mahavirj]

@AxelLin

Sorry, I missed your comment.

For SPIFFS image, is there a standard way to recognize if a file is spiffs image?

I was unable to find anything obvious in quick glance through SPIFFS format, looks like successful mount would be required to check integrity of image. @igrr Do you have any other way round this?

[igrr]

I don't think there is an easy way to determine if something is a SPIFFS image just from the header. SPIFFS has a SPIFFS_probe_fs function that ends up reading magic values from 3 filesystem blocks.
If CONFIG_SPIFFS_USE_MAGIC_LENGTH=n (default is =y), then the magic numbers will be the same for all blocks, so one can read the magic value at the end of the 1st page header — see SPIFFS_MAGIC_PADDR macro definition in SPIFFS code for address calculation.

[hmalpani]

@kshyshkin @AxelLin I have created a example which uses encrypted image.
https://github.com/hmalpani/esp-idf/tree/feature/encrypted_image/examples/system/ota/pre_encrypted_binary_ota
Please feel free to take a look. It also contains a tool for creating encrypting image which can be used for OTA updates.

[jacek12345]

So, is there any good news for developer that developed whole project with OTA update (with public supply firmware scenario) and than added secure boot v.2 and encryption and hangs on one month before releasing to production?
I see that there is (officially undocumented) issue with offset parameter used while encryption, dependent of OTA partition number. Happily, It is not big problem to make double image file and choose correct part depended of running partition on device.
But the second issue is a huge challenge for me. I'm talking about magic byte and writing raw to partition.
Is there any chance that it will be available shortly?
@AxelLin
Did You succeed with Your changes with using esp_partition_write_raw() function?

Update:
I used app_update: Add esp_ota_write_raw function to write pre-encrypted data (IDFGH-5583) #7305 from AxelLin.
It works. The only thing that i had to do , increasing CONFIG_BT_BTC_TASK_STACK_SIZE to 4096.

I'm not very familiar with GitHub, but i see that IDFGH-5583 was rejected right? What was the reason?
can this solution pose any threats?

[hmalpani]

Hello @jacek12345,
We are going to use the approach as here: https://github.com/hmalpani/esp-idf/tree/feature/encrypted_image/examples/system/ota/pre_encrypted_binary_ota.
The feature will be available soon in IDF.

[mdmodi]

@hmalpani I Just try this code I see following error while OTA.
I (20010) esp_encrypted_img: Magic Verified
I (20010) esp_encrypted_img: Reading RSA private key
E (20130) esp_encrypted_img: failed
! mbedtls_pk_parse_keyfile returned -0x3d00

E (20130) esp_encrypted_img: Unable to decipher GCM key
E (20140) OTAUpdate: Error in decoding binary
E (20140) OTAUpdate: OTA failed

Let me know if you need more info. Any help will be appreciated

Thanks

[hmalpani]

@hmalpani I Just try this code I see following error while OTA. I (20010) esp_encrypted_img: Magic Verified I (20010) esp_encrypted_img: Reading RSA private key E (20130) esp_encrypted_img: failed ! mbedtls_pk_parse_keyfile returned -0x3d00

E (20130) esp_encrypted_img: Unable to decipher GCM key E (20140) OTAUpdate: Error in decoding binary E (20140) OTAUpdate: OTA failed

Let me know if you need more info. Any help will be appreciated

Thanks

It seems like you are using the wrong RSA private key. Please make sure you are using the correct RSA private key. You can find info about how to generate RSA key in README.md

[mdmodi]

Thanks for the response. I tried the same steps.

I am generating certificate on windows machine. Let me try it again. Let me know if there is any version dependency on Python version.

[mahavirj]

@mdmodi

This feature is now supported in official ESP-IDF and corresponding example is available at https://github.com/espressif/esp-idf/tree/master/examples/system/ota/pre_encrypted_ota

As a first step, I would recommend that you skip the step of "RSA key generation" and use the test key supplied in the example itself. Once we confirm that this setup works on your side, we can chase on key generation problem separately.

Please let us know update on this.

[saidsnanou1]

@mahavirj
i tested this example but i get this error

../main/pre_encrypted_ota.c:44:30: error: unknown type name 'decrypt_cb_arg_t'; did you mean 'esp_decrypt_cfg_t'?
static esp_err_t _decrypt_cb(decrypt_cb_arg_t *args)
^~~~~~~~~~~~~~~~
esp_decrypt_cfg_t
../main/pre_encrypted_ota.c: In function 'pre_encrypted_ota_task':
../main/pre_encrypted_ota.c:108:10: error: 'esp_https_ota_config_t' {aka 'struct '} has no member named 'decrypt_cb'
.decrypt_cb = _decrypt_cb,
^~~~~~~~~~
../main/pre_encrypted_ota.c:108:23: error: '_decrypt_cb' undeclared (first use in this function)
.decrypt_cb = _decrypt_cb,
^~~~~~~~~~~
../main/pre_encrypted_ota.c:108:23: note: each undeclared identifier is reported only once for each function it appears in
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

Note: I didn't change anything in this examples

[hmalpani]

Hello @saidsnanou1,
esp_https_ota component has also been updated to support this feature. I think you need to apply patch 69f51a9 on your local esp-idf repo.

[saidsnanou1]

@hmalpani thank you very much, but when I work on Arduino as an ESP-IDF component I got this error

main.cpp:80:42: error: invalid conversion from 'esp_decrypt_handle_t' {aka 'void*'} to 'void**' [-fpermissive]
ctx = esp_encrypted_img_decrypt_start(&cfg);

[hmalpani]

@saidsnanou1 This must be resolved by espressif/idf-extra-components#17

[infwingit]

Can we set the FLASH_CRYPT_CONFIG to 0x0, then any address offset of pre-encrypt binaries should be ok to read and write.

[mahavirj]

Can we set the FLASH_CRYPT_CONFIG to 0x0, then any address offset of pre-encrypt binaries should be ok to read and write.

Can you please elaborate? Just to note that, pre-encrypted OTA is entirely different scheme and has no co-relation to platform flash encryption feature as such.

[cl-cr]

@mdmodi

This feature is now supported in official ESP-IDF and corresponding example is available at https://github.com/espressif/esp-idf/tree/master/examples/system/ota/pre_encrypted_ota

As a first step, I would recommend that you skip the step of "RSA key generation" and use the test key supplied in the example itself. Once we confirm that this setup works on your side, we can chase on key generation problem separately.

Please let us know update on this.

Hello,

I am also interested in that feature but I would like to stay on the latest stable release (V4.4.X). Is there any luck to see that feature in stable branch v4.4.X in the near future?

Best regards

[mahavirj]

@cl-cr We are not planning to backport this feature to IDF v4.x. It will be part of v5.0 release. However, I had created a branch on top of v4.4.2 release with this feature, please see discussion here https://esp32.com/viewtopic.php?f=13&t=29917#p103742

[cl-cr]

Perfect, that is what I was looking for. I am going to test that. Thanks for the link!

[jci-zimm]

@mdmodi , Any luck with resolving error -0x3d00 ? I'm getting the exact same error and i've tried the key gen process multiple times.

I (40450) esp_encrypted_img: Magic Verified
I (40452) esp_encrypted_img: Reading RSA private key
E (40454) esp_encrypted_img: failed
  ! mbedtls_pk_parse_keyfile returned -0x3d00

E (40455) esp_encrypted_img: Unable to decipher GCM key
E (40456) OTA: OTA_CMD_WRITE Encrypt failed: ABORT ESP_FAIL
E (40457) file_server: Failed writing block update -1

Edit: Issue was due to #4831

Note the 5.0 links in a different flavor of mbedtls ... not sure that issue is applicable in the updated lib given the sample code.

[law-ko]

@mahavirj Is there any update on how to update intermediate device with encrypted firmware? The intermediate device will receive OTA chunks with UART communication protocol.

@mahavirj Yes all secure features on controller on flash encryption (RELEASE), secure boot reflashable, all needed fuses on. But i have OTA (not through webserver), with intermediate device. When i need update by OTA, i make project, sighn fw with pem key, encrypt fw by digest from pem key, and send to intermediate device. Intermediate device send pre-encrypted fw to esp32 by custom interface and protocol dividing the firmware by chunks. I need that from my host computer to esp32 fw goes pre-encrypted. Thats why i need store pre-encrypted fw by chunks to OTA partition.