idf.py build system fails on 4.3 and 4.2 due to missing referece to esptool_py_flash_target function in component/partition_table/CMakeLists.txt (IDFGH-4874)

[marcidy]

Environment

• Module or chip used: ESP32
• IDF version (run git describe --tags to find it): v4.3, v4.2
• Build System: idf.py
• CMake version: 3.13.4
• Operating System: Linux
• Using an IDE?: No

Problem Description

create a project with a component that REQUIRES app_update

idf,py build fails with error:

CMake Error at /toolchains/esp32/esp-idf/components/partition_table/CMakeLists.txt:101 (esptool_py_flash_target):
  Unknown CMake command "esptool_py_flash_target".

upgrading from v4.1 to v4.2 or v4.3 resulted failure to build a simple project. (The project doesn't compile at the moment, though this is not relevant to the build failure, which doesn't get to the compile stage)

I removed the build directory and rebuilt the toolchain a number of times with different versions of the idf, all with the same result.

I deleted esp-idf repo clone, removed ~/.espressif/ directories, cloned from scratch per instructions, and attempted a number of fresh installations (install.sh, export.sh) using a few versions. All failed with the same error.

multiple idf.py commands failed similarly.

Steps to reproduce

run any invocation of idf.py on a fresh installation

Debug Logs

$ idf.py build
Executing action: all (aliases: build)
Running make in directory /simple/build
Executing "make -j 6 all"...
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
-- Project sdkconfig file /simple/sdkconfig
CMake Error at /toolchains/esp32/esp-idf/components/partition_table/CMakeLists.txt:101 (esptool_py_flash_target):
  Unknown CMake command "esptool_py_flash_target".


-- Configuring incomplete, errors occurred!
See also "/simple/build/CMakeFiles/CMakeOutput.log".
make: *** [Makefile:1220: cmake_check_build_system] Error 1
make failed with exit code 2

Possible Solution Indication

esptool_py_flash_target is defined in the esptool_py component, in project_include.cmake.

Inspecting the partition_table CMakeList.txt, I didn't see a reference to the esptool_py component. I changed line 1 in components/partition_table/CMakeLists.txt to:

idf_component_register(PRIV_REQUIRES esptool_py)

And now the build works fine.

other components that require esptool_py* functions appear to include the component in some fashion, e.g. components/bootloader/subprojectCMakeLists.txt:

...

set(COMPONENTS bootloader esptool_py partition_table soc bootloader_support log spi_flash micro-ecc main efuse)

...

Reverting the change causes the failure.

I don't know if that's correct, but it indicates to me the partition_table component is not being passed the reference to functions defined in components/esptool_py/CMakeLists.txt.

Notes

I reproduced this issue after multiple attempts at freshly installing esp-idf in clean directories, including using IDF_TOOLS_PATH to segregate the issue from my home directory or other installations. All attempts reproduced the problem.

I will attempt further investigation as to why this suddenly is an issue for me. I would expect this failure to impact other people on these versions. At this time I cannot localize the issue to something I did, and it's reproducible for me at least.

[marcidy]

This also reproduced on the docker image to build, referenced here:

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/tools/idf-docker-image.html

[marcidy]

My project was using the app_update component. app_update PRIV_REQUIRES partition_table.

When I remove `app_update, the project builds fine on v4.2 and v4.3.

Without the app_update component, i believe bootloader is most likely among the first (alphabetical) component to be built, which also PRIV_REQUIRES partition_table and the build should also fail, but it doesn't.

Interestingly, bootloader also has a subproject/CMakeLists.txt which explicitly references esptool_py.

set(COMPONENTS
    bootloader
    esptool_py
    esp_hw_support
    hal
    partition_table
    soc
    bootloader_support
    log
    spi_flash
    micro-ecc
    main
    efuse
    esp_system)

I dont know if CMake will process the subproject before the PRIV_REQUIRES but if it does, that might explain why the project builds sucessfully without app_update but fails with app_update.

[marcidy]

Here's an example of project that won't build and causes error, but then builds (not compiles!) with proposed change

simple.zip

[projectgus]

Hi @marcidy,

Thanks for the detailed description and the PR. Agree with your conclusion.

The key to triggering the bug is that normally all components are included in the build, but because your sample project has set(COMPONENTS "ota" "esp_http_client"). This should cause the build system to build "these listed components plus all components they depend on" only, and this incorrectly doesn't include esptool_py component.

Will merge the Pull Request ASAP, but in the meantime a workaround that doesn't involve patching ESP-IDF is to add "esptool_py" to the project CMakeLists.txt COMPONENTS variable.