"idf.py build" fails during app signing with ESP-IDF v4.4.x (IDFGH-8075)

[stevegrau-ayla]

Environment

• Module or chip used: ESP32-WROOM-32, ESP32-C3-WROOM-02
• IDF version: v4.4.1, v4.4.2
• Build System: idf.py
• Operating System: Linux

Problem Description

"idf.py build" fails during app signing with ESP-IDF v4.4.x because "secure_boot_signing_key" is not set when esptool_py/project_include.cmake line 116 executes.

Inserting the following prior to line 112 enables the build to succeed:

get_filename_component(secure_boot_signing_key
"${CONFIG_SECURE_BOOT_SIGNING_KEY}"
ABSOLUTE BASE_DIR "${project_dir}")

The error output looks like this (Note: I have edited pathnames):

Creating esp32c3 image...
Merged 2 ELF sections
Successfully created esp32c3 image.
Generated /build/apa-unsigned.bin
[34/35] Generating signed binary image
FAILED: .signed_bin_timestamp
cd /build && <user_home>/.espressif/python_env/idf4.4_py3.8_env/bin/python <idf_path>/components/esptool_py/esptool/espsecure.py sign_data --version 2 --keyfile -o /build/apa.bin /build/apa-unsigned.bin && /usr/bin/cmake -E echo "Generated signed binary image /build/apa.bin" "from /apa/build/apa-unsigned.bin" && /usr/bin/cmake -E md5sum /build/apa.bin > /build/.signed_bin_timestamp
usage: espsecure sign_data [-h] --version {1,2} --keyfile KEYFILE
[KEYFILE ...] [--append_signatures]
[--output OUTPUT]
datafile
espsecure sign_data: error: argument --keyfile/-k: expected at least one argument

sdkconfig security settings are as follows:

#
# Security features
#
CONFIG_SECURE_SIGNED_ON_BOOT=y
CONFIG_SECURE_SIGNED_ON_UPDATE=y
CONFIG_SECURE_SIGNED_APPS=y
CONFIG_SECURE_BOOT_SUPPORTS_RSA=y
CONFIG_SECURE_TARGET_HAS_SECURE_ROM_DL_MODE=y
CONFIG_SECURE_SIGNED_APPS_RSA_SCHEME=y
CONFIG_SECURE_BOOT=y
CONFIG_SECURE_BOOT_V2_ENABLED=y
CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES=y
CONFIG_SECURE_BOOT_SIGNING_KEY="eng_key_c3.pem"
# CONFIG_SECURE_BOOT_ENABLE_AGGRESSIVE_KEY_REVOKE is not set
CONFIG_SECURE_BOOT_INSECURE=y
CONFIG_SECURE_FLASH_ENC_ENABLED=y
CONFIG_SECURE_FLASH_ENCRYPTION_MODE_DEVELOPMENT=y
# CONFIG_SECURE_FLASH_ENCRYPTION_MODE_RELEASE is not set

Expected Behavior

idf.py build succeeds

Actual Behavior

idf.py build fails

Steps to reproduce

1. configure security as indicated in sdkconfig snippet above
2. "idf.py build"

[dobairoland]

Hi @stevegrau-ayla. Did you by any chance disable the bootloader build?

[stevegrau-ayla]

@dobairoland, The top level CMakeLists.txt does set components to avoid building things that aren't needed. The build worked with pre-4.4 versions of ESP-IDF.

set(COMPONENTS bootloader esptool_py main)

[mahavirj]

@stevegrau-ayla Can you please share minimal sample application to recreate this issue?

[stevegrau-ayla]

I was able to reproduce the problem with the hello_world example in the IDF with the following changes:

Use menuconfig to modify the config as follows:
@@ -70,9 +67,20 @@
#
# Security features
#
-# CONFIG_SECURE_SIGNED_APPS_NO_SECURE_BOOT is not set
-# CONFIG_SECURE_BOOT is not set
+CONFIG_SECURE_SIGNED_ON_BOOT=y
+CONFIG_SECURE_SIGNED_ON_UPDATE=y
+CONFIG_SECURE_SIGNED_APPS=y
+CONFIG_SECURE_BOOT_SUPPORTS_RSA=y
+CONFIG_SECURE_SIGNED_APPS_RSA_SCHEME=y
+CONFIG_SECURE_BOOT=y
+# CONFIG_SECURE_BOOT_V1_ENABLED is not set
+CONFIG_SECURE_BOOT_V2_ENABLED=y
+CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES=y
+CONFIG_SECURE_BOOT_SIGNING_KEY="secure_boot_signing_key.pem"
+# CONFIG_SECURE_BOOT_INSECURE is not set
# CONFIG_SECURE_FLASH_ENC_ENABLED is not set
+# CONFIG_SECURE_DISABLE_ROM_DL_MODE is not set
+CONFIG_SECURE_INSECURE_ALLOW_DL_MODE=y
# end of Security features

#

Edit the top level CMakeLists to restrict the components that get built. (Note: The problem doesn't reproduce without this change.)

diff --git a/examples/get-started/hello_world/CMakeLists.txt b/examples/get-started/hello_world/CMakeLists.txt
index 3772daf2e8..277ef97b11 100644
--- a/examples/get-started/hello_world/CMakeLists.txt
+++ b/examples/get-started/hello_world/CMakeLists.txt
@@ -3,4 +3,5 @@
cmake_minimum_required(VERSION 3.5)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+set(COMPONENTS bootloader esptool_py main)
project(hello_world)

The build failure looks like this:

[2/3] Generating signed binary image
FAILED: .signed_bin_timestamp
cd /home/sgrau/wdl/ada-esp-1/espressif/examples/get-started/hello_world/build && /home/sgrau/.espressif/python_env/idf4.4_py3.8_env/bin/python /home/sgrau/wdl/ada-esp-1/espressif/components/esptool_py/esptool/espsecure.py sign_data --version 2 --keyfile -o /home/sgrau/wdl/ada-esp-1/espressif/examples/get-started/hello_world/build/hello_world.bin /home/sgrau/wdl/ada-esp-1/espressif/examples/get-started/hello_world/build/hello_world-unsigned.bin && /usr/bin/cmake -E echo "Generated signed binary image /home/sgrau/wdl/ada-esp-1/espressif/examples/get-started/hello_world/build/hello_world.bin" "from /home/sgrau/wdl/ada-esp-1/espressif/examples/get-started/hello_world/build/hello_world-unsigned.bin" && /usr/bin/cmake -E md5sum /home/sgrau/wdl/ada-esp-1/espressif/examples/get-started/hello_world/build/hello_world.bin > /home/sgrau/wdl/ada-esp-1/espressif/examples/get-started/hello_world/build/.signed_bin_timestamp
usage: espsecure sign_data [-h] --version {1,2} --keyfile KEYFILE
[KEYFILE ...] [--append_signatures]
[--output OUTPUT]
datafile
espsecure sign_data: error: argument --keyfile/-k: expected at least one argument

Projects that built prior to ESP-IDF 4.4 encounter this problem with 4.4.2.

[AdityaHPatwardhan]

@stevegrau-ayla I was able to reproduce your issue when I set the following In the CMakeListst.txt of the project dir.
set(COMPONENTS bootloader esptool_py main)

But the issue got fixed once I modified it to following:
set(COMPONENTS app_update bootloader esptool_py main)

Please check once.
Also, I dont think hardcoding the required components in the top level CMakeLists.txt is the recommended way. I understand that it could reduce build time, but it may lead to issues similar to above.
Please avoid doing so unless you are absolutely sure about the dependencies for your project.
CC @mahavirj

[stevegrau-ayla]

@AdityaHPatwardhan, Thank you for checking into this. Adding app_update to the components list in the top level CMakeLists.txt does resolve the build problem.

I think this dependency should be handled correctly by the ESP-IDF when app signing is enabled in sdkconfig.

I disagree with your comment about dependencies. The whole point of cmake is to build only what needs to be built taking into account dependencies. If the assumption is dependencies cannot be relied on that degenerates to the only reliable way to build is do a clean build from scratch every time you change anything.

The ESP-IDF documentation suggests trimming the component list and notes that each component will pull in other components it requires. See in the discussion of the COMPONENTS variable: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html#optional-project-variables

[stevegrau-ayla]

Thank you for the fix! I verified it solves the problem for my case.