IDF v4.x multi-component application build fails at final linking stage due to "undefined reference to ..." ld errors (IDFGH-5195)

[ul-gh]

This issue I found during the transition from PlatformIO IDE to a pure ESP-IDF setup using multiple components, i.e. arduino-esp32, ESPAsyncWebServer, among others.

After two days of trying, I can't seem to compile using IDF v4x any more.
The complete project code you find here:

ESP-LiveControl

With the current ESP-IDF build system, the compilation of all source files first /succeeds/ (after some fixing of arduino-esp32, see arduino-esp32 issue 5064 but then at the final stage, I get multiple ld linker errors:

(Full build log output attached as a text file):

/ld: /home/ulrich/mysrc/esp32/esp_ajax_if/build/../main/api_server.cpp:242: undefined reference to `AsyncWebServerRequest::send(int, String const&, String const&)'
collect2: error: ld returned 1 exit status

This is although the particular references were definitely successfully built before, as from the same build log output. A full error output I am attaching as a file.

I closely followed the project setup instructions from ESP-IDF build system documentation.

This is the project CMakeLists.txt:

make_minimum_required(VERSION 3.16.0)

# Project built successfully with std=C++20 before
set(CMAKE_CXX_STANDARD 20)

list(APPEND EXTRA_COMPONENT_DIRS
    "components/esp-rainmaker/components"
)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)

project(esp_ajax_if)

This is the main component CMakeLists.txt:

set(requires
    "arduino-esp32"
    "ESPAsyncWebServer"
    #"AsyncTCP" # Already required by ESPAsyncWebServer
    # Not compatible with IDF 4 build, see include_dirs below
    #"ArduinoJson"
    "esp32_ps_pwm"
)

set(sources
    "app_main.cpp"
    "app_state_model.cpp"
    "app_controller.cpp"
    "wifi_configurator.cpp"
    "api_server.cpp"
    "aux_hw_drv.cpp"
    "sensor_kty81_1xx.cpp"
    "esp32_adc_channel.cpp"
    "fs_io.cpp"
)

set(include_dirs
    "include"
    "config"
    "${PROJECT_DIR}/components/esp_idf_build_incompatible/ArduinoJson/src"
)

idf_component_register(
    SRCS "${sources}"
    INCLUDE_DIRS "${include_dirs}"
    REQUIRES "${requires}"
)

This is a component CMakeLists.txt:

set(requires
    "AsyncTCP"
)

# register_component() is now deprecated and will be removed in IDF v.5
# See: https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/build-system.html#no-longer-available-in-cmake
set(sources
    "src/AsyncEventSource.cpp"
    "src/AsyncWebSocket.cpp"
    "src/SPIFFSEditor.cpp"
    "src/WebAuthentication.cpp"
    "src/WebHandlers.cpp"
    "src/WebRequest.cpp"
    "src/WebResponses.cpp"
    "src/WebServer.cpp"
)

idf_component_register(
    SRCS "${sources}"
    INCLUDE_DIRS "src"
    REQUIRES "${requires}"
)

target_compile_definitions(${COMPONENT_TARGET} PUBLIC -DESP32)
target_compile_options(${COMPONENT_TARGET} PRIVATE -fno-rtti)

Am I missing anything or is the build system broken?

Regards,
Ulrich Lukas
build_log.txt
build_log_full.txt.gz

[projectgus]

Hi @ul-gh,

Thanks for the very clear report. I cloned your project master branch from GitHub and using the current ESP-IDF master branch commit I was able to reproduce different linker errors, so something is clearly not quite right!

I think probably the underlying issue is something with transitive dependencies, or at least it looks like it (i.e. some break in the chain of component A depends on component B depends on component C). Or possibly a dependency cycle where the linker command line doesn't end up with the libraries in the right place.

Can I ask exactly which ESP-IDF version you're using? I'll see if I can reproduce the exact error you're seeing by using it.

Also, if you still have the file build/esp_ajax_if.map that matches the logs posted above, could you please attach that?

[ul-gh]

Hi @projectgus!

My current esp-idf version is v4.3-beta3, e9cf9e2 (with added patch of PR6578)

I also tried with master branch head a few days ago, same result.

arduino-esp32 is current master with following added dependencies to make it compile:

ulrich@uhp:~/mysrc/esp32/esp_ajax_if/components/arduino-esp32$ git diff upstream
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f6332f54..11b49f24 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -160,8 +160,9 @@ set(includedirs
 
 set(srcs ${CORE_SRCS} ${LIBRARY_SRCS} ${BLE_SRCS})
 set(priv_includes cores/esp32/libb64)
-set(requires spi_flash mbedtls mdns esp_adc_cal)
-set(priv_requires fatfs nvs_flash app_update spiffs bootloader_support openssl bt arduino_tinyusb main)
+set(rmaker_requires button  esp_rainmaker  esp_schedule  json_generator json_parser  qrcode  rmaker_common  ws2812_led)
+set(requires spi_flash mbedtls mdns esp_adc_cal wifi_provisioning esp_ipc spiffs ${rmaker_requires})
+set(priv_requires fatfs nvs_flash app_update spiffs bootloader_support openssl bt main)
 
 if(NOT CONFIG_ARDUINO_SELECTIVE_COMPILATION OR CONFIG_ARDUINO_SELECTIVE_ArduinoOTA)
   list(APPEND priv_requires esp_https_ota)

[ul-gh]

At last!
This now compiles, and indeed (thanks @projectgus):

It seems there is some cyclic dependency in my components/ESPAsyncWebServer.

I have not yet found out where exactly, but setting the CMake "LINK_INTERFACE_MULTIPLICITY" which defaults to 2 to a value of 3 in that component makes the compilation succeed. (See: GNU LD linker and CMake Link order documentation)

While I am definitely a first-time-user of CMake: This seems to be a non-obvious but very likely issue one might encounter when building with the ESP-IDF build system.

IMO ESP-IDF should not choke on a wrong, automatically generated, link order.
There should be a note in ESP-IDF build system documentation..

Set LINK_INTERFACE_MULTIPLICITY in components/ESPAsyncWebServer/CMakeLists.txt :

set(requires
    "AsyncTCP"
)

# register_component() is now deprecated and will be removed in IDF v.5
# See: https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/build-system.html#no-longer-available-in-cmake
set(sources
    "src/AsyncEventSource.cpp"
    "src/AsyncWebSocket.cpp"
    "src/SPIFFSEditor.cpp"
    "src/WebAuthentication.cpp"
    "src/WebHandlers.cpp"
    "src/WebRequest.cpp"
    "src/WebResponses.cpp"
    "src/WebServer.cpp"
)

idf_component_register(
    SRCS "${sources}"
    INCLUDE_DIRS "src"
    REQUIRES "${requires}"
)

target_compile_definitions(${COMPONENT_TARGET} PUBLIC -DESP32)
target_compile_options(${COMPONENT_TARGET} PRIVATE -fno-rtti)

# Somewhere there seems to be a cyclic dependency (arduino-esp32?) producing linker errors
# without this setting. When dependencies are fixed, following can be omitted.
set_target_properties(${COMPONENT_TARGET} PROPERTIES LINK_INTERFACE_MULTIPLICITY 3)

[projectgus]

Hi @ul-gh,

Well done figuring this out! After a bit of further analysis, I can confirm there is a circular dependency in arduino-esp32. It depends on "main" component, where "main" is the top-level component that depends on all other components. This creates the circular dependency arduino-esp32 -> main -> arduino-esp32, which can cause linker issues with other components that also depend on arduino-esp32 (or their dependencies, in this case it was causing issues because main -> ESPAsyncWebServer -> AsyncTCP -> arduino-esp32 -> main.)

Adding LINK_INTERFACE_MULTIPLICITY to ESPAsyncWebServer works around the issue but at the cost of a slower linker pass.

I'll submit a PR to arduino-esp32 repo to remove the "main" dependency as I don't think it's needed. If it turns out that it is needed for some reason then we can probably add LINK_INTERFACE_MULTIPLICITY in arduino-esp32 to work around the cycles.

IMO ESP-IDF should not choke on a wrong, automatically generated, link order.
There should be a note in ESP-IDF build system documentation..

Agree on both counts. Will add a note about this kind of error, possible workarounds, and some ways to find dependency cycles (as this is also not very simple!) We'll see if we can improve ESP-IDF tooling for this as well.

[projectgus]

remove the "main" dependency as I don't think it's needed

Oh! Of course it's needed, for the case where "Autostart Arduino setup and loop on boot" is enabled. In this case, app_main is in arduino-esp32 and it needs to be able to depend on "main" where setup() and loop() functions are found...

I think we can still find a better fix for it, either with linker tricks or by making a separate libarduino_autostart.a that only contains main.cpp so nothing needs to depend on it...

[diplfranzhoepfinger]

wouldnt it be more ESPish to write it like 

# Somewhere there seems to be a cyclic dependency producing linker errors
# without this setting. When dependencies are fixed, following can be omitted.
set_property(TARGET ${COMPONENT_TARGET} APPEND PROPERTY LINK_INTERFACE_MULTIPLICITY 3)

[diplfranzhoepfinger]

inside ESP-IDF i find it as 

set_property(TARGET ${COMPONENT_LIB} APPEND PROPERTY LINK_INTERFACE_MULTIPLICITY 3)

[selalipop]

I've been hitting this same issue for days many thanks for this! And in hindsight that loop was something I noticed and should have picked up on 😁 For the benefit of anyone using Github's global search (which is how I found this) I was getting linker errors with

• base64_encode_block (libb64)
• fs::File (File.h)
• WifiServer (WifiServer.h)
• various mbedtls imports

But like was noted in the thread, this probably ends up erroring out at a non-deterministic point.

---

Slightly different note but related to the original comment (and what really sent me down the wrong path) is I believe ESPAsyncWebServer has a header access issue when built in esp-idf, not sure if you saw this too @ul-gh and think it might be an issue

It's relying on the Arduino core's libb64 implementation, but that's not a part of the public interface of arduino-esp32. As a result it's listed under PRIV_INCLUDE_DIRS, which according to ESP-IDF is only supposed to be accessible from the current component (arduino-esp32 in this case):

directory paths, must be relative to the component directory, which will be added to the include search path for this component’s source files only

I have a hunch that if the circular reference is fixed this might be an issue, but I haven't had a chance to test

[projectgus]

Glad the workaround is helping. I've pushed the PR linked above to arduino-esp32, this should remove the need for the workaround at all.

Have a pending change also to add LINK_INTERFACE_MULTIPLICITY to the ESP-IDF build system docs.