diff --git a/content/blog/2025/11/advanced-porting-libraries-as-components/featured.webp b/content/blog/2025/11/advanced-porting-libraries-as-components/featured.webp new file mode 100644 index 000000000..b0c61f17d Binary files /dev/null and b/content/blog/2025/11/advanced-porting-libraries-as-components/featured.webp differ diff --git a/content/blog/2025/11/advanced-porting-libraries-as-components/index.md b/content/blog/2025/11/advanced-porting-libraries-as-components/index.md new file mode 100644 index 000000000..29d5fce2a --- /dev/null +++ b/content/blog/2025/11/advanced-porting-libraries-as-components/index.md @@ -0,0 +1,567 @@ +--- +title: "Advanced techniques for porting libraries to ESP-IDF components" +date: "2025-11-24" +summary: "This article follows up on the article 'Porting a library to an ESP-IDF component' and shows some advanced tips and tricks when porting larger libraries into ESP-IDF components." +authors: + - david-cermak +tags: ["component","porting","linux","posix","ESP-IDF"] +--- + +## Introduction + +In the previous article, ["Porting a library to an ESP-IDF component"](/blog/2025/10/porting-external-library-as-component/), we covered the basics of converting an external library into a reusable ESP-IDF component using the simple `tinyexpr` library as an example. We learned how to create a component structure, handle basic compatibility issues, and suppress compiler warnings at the component level. + +However, when porting larger, more complex libraries—especially those originally designed for Linux or other POSIX-compliant systems—you'll encounter additional challenges that require more advanced techniques. This article builds on the fundamentals from the [previous article](/blog/2025/10/porting-external-library-as-component) and explores advanced porting strategies used in real-world library ports such as: + +- [asio](https://github.com/espressif/esp-protocols/tree/master/components/asio) - Asynchronous I/O library +- [mosquitto](https://github.com/espressif/esp-protocols/tree/master/components/mosquitto) - MQTT broker and client library +- [libssh](https://github.com/david-cermak/libssh) - SSH library +- [libwebsockets](https://github.com/espressif/esp-protocols/tree/master/components/libwebsockets) - WebSockets library + +When porting a library, it's helpful to follow a structured approach: + +1. **Assess what to reuse and what to change** - Identify which parts of the original library can be used as-is and which need adaptation +2. **Compose the build system** - Create or adapt the `CMakeLists.txt` to reference the original library sources while integrating port-specific functionality +3. **Create the port folder** - Set up a dedicated `port/` directory structure for adapted functionality that bridges the library and ESP-IDF +4. **Adjust functionality** - Use advanced techniques like header injection, linker wrapping, and the `sock_utils` component to adapt the port layer +5. **Fine-tune** - Manage compiler warnings at the file level and make final adjustments for optimal integration + +Now, we will go through these techniques in detail and look at practical examples from real-world ports to help you successfully port complex libraries to ESP-IDF. + +## Step 1: Assess what to reuse and what to change + +Before writing any port code, decide what can remain unmodified and what needs adaptation for ESP-IDF. Often, you can keep most of the upstream sources intact and adapt behavior via configuration and thin port layers. + +### Compile Definitions and Configuration + +Libraries often use compile-time definitions to enable or disable features, configure behavior, or adapt to different platforms. + +#### Feature Flags and config.h + +Many libraries use a `config.h` file generated by autotools or CMake that contains feature detection results. When porting, you'll need to: + +1. Identify which defines are needed +2. Convert them to `target_compile_definitions()` +3. Make them conditional based on ESP-IDF configuration when appropriate + +The [asio port](https://github.com/espressif/esp-protocols/tree/master/components/asio) demonstrates this pattern: + +```cmake +target_compile_definitions(${COMPONENT_LIB} PUBLIC + SA_RESTART=0x01 + SA_NOCLDSTOP=0x2 + SA_NOCLDWAIT=0x4 + ASIO_DISABLE_SERIAL_PORT + ASIO_SEPARATE_COMPILATION + ASIO_STANDALONE + ASIO_HAS_PTHREADS + OPENSSL_NO_ENGINE + ASIO_DETAIL_IMPL_POSIX_EVENT_IPP) +``` + +Some definitions, which are useful for our component to be configurable by Kconfg system could be defined in `CMake` as conditionals: + +```cmake +if(NOT CONFIG_COMPILER_CXX_EXCEPTIONS) + target_compile_definitions(${COMPONENT_LIB} PUBLIC ASIO_NO_EXCEPTIONS) +endif() + +if(NOT CONFIG_COMPILER_RTTI) + target_compile_definitions(${COMPONENT_LIB} PUBLIC ASIO_NO_TYPEID) +endif() +``` + +This ensures the library adapts to ESP-IDF's configuration, disabling features that aren't available (like C++ exceptions or Run-time Type Information -- RTTI) when they're disabled in the project. + +#### Handling Platform Differences + +Libraries often use `#ifdef HAVE_FEATURE` patterns to conditionally compile code based on detected features. When porting: + +1. Identify which features are available on ESP-IDF +2. Define the appropriate `HAVE_*` macros +3. Provide implementations or stubs for missing features + +For example these macros are used in the mosquitto port: + +```cmake +target_compile_definitions(${COMPONENT_LIB} PRIVATE + HAVE_PTHREADS=1 + HAVE_SOCKETPAIR=1 + HAVE_PIPE=1) +``` + +## Step 2: Compose the build system + +Different libraries use different build systems, and you'll need to adapt them to ESP-IDF's CMake-based build system. + +### Creating Custom CMakeLists.txt + +Most of the time, you'll create your own `CMakeLists.txt` that integrates the library into ESP-IDF's build system. However, there are different approaches depending on the library: + +#### From Scratch + +For libraries without CMake support, or when the original CMake is too complex, create a custom CMakeLists.txt that: +- Lists all source files explicitly +- Sets up include directories +- Configures compile definitions +- Manages dependencies + +The mosquitto port is an example of this approach. + +#### Include Original CMake + +Some libraries already have well-structured CMake that can be included directly. The [`fmt` library](https://github.com/espressif/idf-extra-components/tree/master/fmt) is an example where the original CMake can be included with minimal modifications: + +```cmake +add_subdirectory(fmt) +``` + +#### Include and Modify Original CMake + +For libraries with CMake that needs modifications, you can include the original but override specific settings. The [mbedtls component](https://github.com/espressif/esp-idf/tree/master/components/mbedtls) uses this approach, including the original CMake but customizing it for ESP-IDF. + +### Handling Different Build Systems + +#### GNU Make + +Libraries using GNU Make typically require: +- Converting Makefile variables to CMake variables +- Translating build rules to CMake `add_library()` or `idf_component_register()` +- Handling conditional compilation manually + +#### Autotools + +Autotools-based libraries (using `configure` scripts) are more complex: +- You'll need to replicate the configuration logic in CMake +- Convert `config.h` defines to CMake `target_compile_definitions()` +- Handle feature detection manually + +#### CMake + +Libraries already using CMake are the easiest: +- Include the original CMake if possible +- Or extract the source lists and recreate the build logic + +### Source File Management + +You may need to conditionally include or exclude source files based on ESP-IDF configuration: + +```cmake +set(m_srcs + ${m_lib_dir}/memory_mosq.c + ${m_lib_dir}/util_mosq.c + # ... more sources +) + +if(CONFIG_MOSQ_ENABLE_SYS) + list(APPEND m_srcs ${m_src_dir}/sys_tree.c) +endif() + +idf_component_register(SRCS ${m_srcs} ...) +``` + +You can also replace sources for testing, as shown in the pre-include example earlier, where test versions of source files replace the originals during unit testing. + +## Step 3: Create the port folder + +Create a dedicated `port/` directory for adapted functionality that bridges the library with ESP-IDF. + +**Port-Specific Header Directories** + +A common pattern in ported libraries is to organize port-specific headers in dedicated directories: + +- `port/include/` - Public headers that extend or wrap library functionality +- `port/priv_include/` - Private headers used only within the port implementation + +Both asio and mosquitto use this structure. For example, in the asio port: + +```cmake +idf_component_register(SRCS ${asio_sources} + INCLUDE_DIRS "port/include" "asio/asio/include" + PRIV_INCLUDE_DIRS ${asio_priv_includes} + PRIV_REQUIRES ${asio_requires}) +``` + +This allows the port to provide custom headers that override or extend the original library's headers without modifying the upstream source. + +## Step 4: Adjust functionality (port layer) + +### Header Injection and Pre-inclusion Techniques + +When porting libraries, you often need to inject custom headers or modify include behavior without changing the original source code. This section covers several techniques for achieving this. + +**include_next Directive** + +The `include_next` directive is a GCC/Clang extension (not C++ specific—it works with C too) that allows you to include the next header in the search path. This is useful for: + +- Wrapping or extending standard headers +- Providing compatibility layers +- Adding platform-specific extensions + +For example, if you need to extend a standard header: + +```c +// port/include/sys/socket.h +#include_next + +// Add ESP-IDF specific extensions +int esp_socketpair(int domain, int type, int protocol, int sv[2]); +``` + +The `include_next` directive will find the original `sys/socket.h` in the system include path and include it, then your custom code adds additional definitions. + +**Injecting Headers in the Same Directory** + +Some libraries use old-style include guards that allow you to inject headers in the same directory. This technique works when the library uses patterns like: + +```c +#ifndef __timeval_h__ +#define __timeval_h__ +... +#endif +``` + +By placing a header file with the same name in the same directory (or earlier in the include path), you can intercept the include and provide custom definitions. However, this technique is fragile and only works with old-style guards—modern `#pragma once` headers cannot be intercepted this way. + +**Pre-include Method** + +The pre-include method uses the `-include` compiler flag to automatically include a header before every source file is compiled. This is useful for: + +- Injecting dependency definitions +- Providing platform-specific macros +- Replacing standard library functions + +Here's an example pattern from a unit test implementation that uses pre-inclusion to inject test dependencies: + +```cmake +if(CONFIG_MB_UTEST) + set(dep_inj_dir "${CMAKE_CURRENT_LIST_DIR}/freemodbus/unit_test") + list(APPEND priv_include_dirs "${dep_inj_dir}/include") + foreach(src ${srcs}) + get_filename_component(src_wo_ext ${src} NAME_WE) + if(EXISTS "${dep_inj_dir}/src/${src_wo_ext}_test.c") + list(REMOVE_ITEM srcs ${src}) + list(APPEND srcs "${dep_inj_dir}/src/${src_wo_ext}_test.c") + set_property(SOURCE ${src} APPEND_STRING PROPERTY COMPILE_FLAGS + " -include ${dep_inj_dir}/include/${src_wo_ext}_test.h -include ${dep_inj_dir}/include/dep_inject.h") + endif() + endforeach() +endif() +``` + +This pattern: +1. Checks if a test version of a source file exists +2. Replaces the original source with the test version +3. Pre-includes test headers that provide mock implementations + +### Linker Wrapping + +Linker wrapping is a powerful technique that allows you to override functions without modifying the source code. It uses the `--wrap` linker flag to redirect function calls to wrapper implementations. + +#### How It Works + +When you wrap a function `function_name`, the linker will: +- Redirect calls to `function_name()` to `__wrap_function_name()` +- Make the original function available as `__real_function_name()` + +This allows you to: +- Intercept function calls +- Add ESP-IDF specific behavior +- Call the original function when needed +- Completely replace the function if desired + +**When to use linker wrapping:** + +- Overriding library functions that need ESP-IDF specific behavior +- Intercepting calls to standard library functions (malloc, free, time, etc.) +- When you want to avoid modifying the original source code +- As an alternative to source-level function replacement + +**Limitations:** + +- Only works with functions (not macros or inline functions) +- Requires implementing the wrapper function +- The wrapped function must be linked (not inlined) + +#### Real-World Example: libwebsockets + +The libwebsockets (lws) port uses linker wrapping to override functions that need ESP-IDF specific behavior. Here's the CMakeLists.txt pattern: + +```cmake +set(WRAP_FUNCTIONS mbedtls_ssl_handshake_step + lws_adopt_descriptor_vhost) + +foreach(wrap ${WRAP_FUNCTIONS}) + target_link_libraries(${COMPONENT_LIB} INTERFACE "-Wl,--wrap=${wrap}") +endforeach() +``` + +The wrapped functions are then implemented in `port/lws_port.c`: + +```c +extern int __real_mbedtls_ssl_handshake_step(mbedtls_ssl_context *ssl); + +int __wrap_mbedtls_ssl_handshake_step(mbedtls_ssl_context *ssl) +{ + int ret = 0; + + while (ssl->MBEDTLS_PRIVATE(state) != MBEDTLS_SSL_HANDSHAKE_OVER) { + ret = __real_mbedtls_ssl_handshake_step(ssl); + + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) { + continue; + } + + if (ret != 0) { + break; + } + } + + return ret; +} +``` + +In this example, the wrapper: +1. Calls the original function via `__real_mbedtls_ssl_handshake_step()` +2. Handles `WANT_READ`/`WANT_WRITE` errors by retrying +3. Continues until the handshake is complete + + +### Socket Utilities (sock_utils) - Deep Dive + +When porting Linux/Unix libraries to ESP-IDF, you'll often encounter POSIX socket APIs that aren't directly available. The [`sock_utils` component](https://github.com/espressif/esp-protocols/tree/master/components/sock_utils) provides a compatibility layer that implements common POSIX socket functions using ESP-IDF's `lwIP` and `esp_netif` components. + +`sock_utils` is especially useful when porting libraries that rely on: +- POSIX socket APIs (`socketpair`, `pipe`) +- Network interface enumeration (`ifaddrs`) +- Address resolution (`getnameinfo`, `gai_strerror`) +- Hostname resolution (`gethostname`) + +| API | Description | Limitations | Declared in | +|--------------------|-------------------------------------------------------------|-------------------------------------------------------------------|----------------------------------------| +| `ifaddrs()` | Retrieves interface addresses using `esp_netif` | IPv4 addresses only | `ifaddrs.h` | +| `socketpair()` *) | Creates a pair of connected sockets using `lwIP` loopback stream sockets | IPv4 sockets only | `socketpair.h`, `sys/socket.h` **) | +| `pipe()` *) | Wraps `socketpair()` to provide unidirectional pipe-like functionality | Uses bidirectional sockets in place of true pipes | `socketpair.h`, `unistd.h` ***) | +| `getnameinfo()` | Converts IP addresses to human-readable form using `lwIP`'s `inet_ntop()` | IPv4 only; supports `NI_NUMERICHOST` and `NI_NUMERICSERV` flags only | `getnameinfo.h`, `netdb.h` in ESP-IDF | +| `gai_strerror()` | Returns error code as a string | Simple numeric string representation only | `gai_strerror.h`, `netdb.h` **) | +| `gethostname()` | Returns lwip netif hostname | Not a system-wide hostname, but interface specific hostname | `gethostname.h`, `unistd.h` in ESP-IDF | + +Notes: +- `*)` `socketpair()` and `pipe()` are built on top of `lwIP` TCP sockets, inheriting the same characteristics. For instance, the maximum transmit buffer size is based on the `TCP_SND_BUF` setting. +- `**)` `socketpair()` and `gai_strerror()` are declared in sock_utils header files. From ESP-IDF v5.5 onwards, these declarations are automatically propagated to the official header files. If you're using an older IDF version, you need to manually pre-include the related header files from the sock_utils public include directory. +- `***)` `pipe()` is declared in the compiler's `sys/unistd.h`. + +#### Integration Methods + +For ESP-IDF versions before 5.5, you need to manually ensure the sock_utils headers are included. This can be done by: + +1. Adding sock_utils to your component's `REQUIRES` or `PRIV_REQUIRES`: + +```cmake +idf_component_register(SRCS ${sources} + PRIV_REQUIRES sock_utils) +``` + +2. Pre-including the headers if needed: + +```cmake +target_compile_options(${COMPONENT_LIB} PRIVATE + "-include" "socketpair.h") +``` + +From ESP-IDF v5.5 onwards, the sock_utils declarations are automatically available in the standard headers when you include `sock_utils` as a dependency. Simply add it to your component: + +```cmake +idf_component_register(SRCS ${sources} + PRIV_REQUIRES sock_utils) +``` + +#### Real-World Example: ASIO Using socketpair() + +The asio library uses `socketpair()` from sock_utils for its pipe-interrupter implementation. The pipe-interrupter is used to wake up the event loop when needed. Here's how it's integrated: + +1. The asio component includes sock_utils as a dependency: + +```cmake +set(asio_requires lwip sock_utils) +idf_component_register(SRCS ${asio_sources} + PRIV_REQUIRES ${asio_requires}) +``` + +2. The asio code can then use `socketpair()` directly, and it will resolve to the sock_utils implementation. + +**Dependencies** + +`sock_utils` requires: +- **lwIP** - For socket functionality +- **esp_netif** - For network interface management + +These are automatically pulled in when you add `sock_utils` as a dependency. + +### Override Functions + +When porting libraries, you'll often need to provide implementations for functions that aren't available on ESP-IDF, or override functions to provide ESP-IDF-specific behavior. + +Libraries often need to override standard functions: + +- **malloc/free** - To use ESP-IDF's memory management +- **Time functions** - To use ESP-IDF's time APIs +- **Random number generation** - To use ESP-IDF's RNG + +These can be implemented as: +- Source-level replacements (providing the function in your port code) +- Linker-wrapped functions (as discussed earlier) +- Pre-included headers that define macros + +### Stub Function Implementations + +Some functions may not be needed on ESP-IDF but are required by the library's API. In these cases, you can provide stub implementations. + +The asio port provides a stub for the `pause()` system call: + +```c +extern "C" int pause(void) +{ + while (true) { + ::sleep(UINT_MAX); + } +} +``` + +This stub is placed in `port/src/asio_stub.cpp` and provides a minimal implementation that satisfies the library's requirements. + +## Step 5: Fine-tune (Compiler Warnings) + +### Component-Level Suppression + +As covered in the basic porting article, you can suppress compiler warnings for an entire component using `target_compile_options()`: + +```cmake +target_compile_options(${COMPONENT_LIB} PRIVATE -Wno-char-subscripts) +``` + +This approach works well when the warning applies broadly across the component, but it can be too broad when only specific files trigger warnings. + +### File-Level Suppression + +For more granular control, ESP-IDF allows you to suppress warnings on a per-file basis using `set_source_files_properties()`. This is particularly useful when: + +- Only a few files in a large library trigger warnings +- You want to keep strict warnings enabled for the rest of the component +- Different files need different warning suppressions + +The mosquitto library provides a good example of this technique. Some mosquitto source files unconditionally define `_GNU_SOURCE`, which collides with the ESP-IDF build system and produces redefinition warnings. Instead of suppressing this warning for the entire component, we can target only the offending files: + +```cmake +# Some mosquitto source unconditionally define `_GNU_SOURCE` which collides with IDF build system +# producing warning: "_GNU_SOURCE" redefined +# This workarounds this issue by undefining the macro for the selected files +set(sources_that_define_gnu_source ${m_src_dir}/loop.c ${m_src_dir}/mux_poll.c) +foreach(offending_src ${sources_that_define_gnu_source}) + set_source_files_properties(${offending_src} PROPERTIES COMPILE_OPTIONS "-U_GNU_SOURCE") +endforeach() +``` + +You can also use this technique to suppress specific warnings for individual files: + +```cmake +# Suppress format warnings for specific files +set_source_files_properties(${m_src_dir}/logging.c PROPERTIES COMPILE_OPTIONS "-Wno-format") +``` + +Or combine multiple options: + +```cmake +set_source_files_properties(${m_src_dir}/file.c PROPERTIES + COMPILE_OPTIONS "-Wno-format;-Wno-unused-variable") +``` + +## License Considerations + +When porting external libraries, always consider license compatibility: + +- **Check the library's license** - Ensure it's compatible with your use case +- **Maintain attribution** - Include license files and copyright notices +- **Document modifications** - If you modify the library, document the changes +- **Consider license of dependencies** - Ensure all dependencies are compatible + +Most ported libraries maintain the original license and add Espressif's copyright notice for port-specific code. + +## Real-World Case Study: ASIO Port + +The ASIO (Asynchronous I/O) library port demonstrates several advanced techniques working together. Let's examine the key changes and techniques used. + +The ASIO port switched to using the upstream ASIO library directly, reducing maintenance burden while providing a robust asynchronous I/O implementation for ESP-IDF. + +Key techniques used: + +1. **Switching to Upstream** + +Instead of maintaining a fork, the port now uses the upstream ASIO library directly, making it easier to stay current with updates and bug fixes. + +2. **Using socketpair() from sock_utils** + +ASIO's pipe-interrupter implementation uses `socketpair()` to create a communication channel for waking the event loop. The port uses `sock_utils` to provide this POSIX function: + +```cmake +set(asio_requires lwip sock_utils) +``` + +3. **Pipe-Interrupter Implementation** + +The pipe-interrupter is a critical component for ASIO's event loop. By using `socketpair()` from `sock_utils`, the port provides a working implementation without modifying ASIO's source code. + +4. **Local pause() Stub** + +ASIO requires the `pause()` system call, which isn't available on ESP-IDF. The port provides a stub implementation in `port/src/asio_stub.cpp`: + +```c +extern "C" int pause(void) +{ + while (true) { + ::sleep(UINT_MAX); + } +} +``` + +5. **POSIX Event Customization** + +The port replaces ASIO's default `posix_event` constructor with an ESP-IDF-compatible version that avoids pthread_condattr_t operations that aren't available on all IDF versions: + +```cpp +// This replaces asio's posix_event constructor +// since the default POSIX version uses pthread_condattr_t operations (init, setclock, destroy), +// which are not available on all IDF versions +posix_event::posix_event() + : state_(0) +{ + int error = ::pthread_cond_init(&cond_, nullptr); + asio::error_code ec(error, asio::error::get_system_category()); + asio::detail::throw_error(ec, "event"); +} +``` + +This is implemented in `port/src/asio_stub.cpp` and enabled via the `ASIO_DETAIL_IMPL_POSIX_EVENT_IPP` compile definition. + +For more details on the ASIO port changes, see [ESP-Protocols PR #717](https://github.com/espressif/esp-protocols/pull/717). + +## Conclusion + +Porting complex libraries to ESP-IDF requires a combination of techniques: + +- **Compiler warning management** - Use file-level suppression for granular control +- **Header injection** - Pre-include, include_next, and port directories for seamless integration +- **Linker wrapping** - Override functions without modifying source code +- **sock_utils** - Leverage POSIX compatibility for socket-based libraries +- **Build system adaptation** - Create custom CMake or adapt existing build systems +- **Function overrides and stubs** - Provide ESP-IDF-specific implementations +- **Compile definitions** - Configure libraries to work with ESP-IDF's feature set + +The key is to minimize modifications to the original library source code, making it easier to: +- Stay synchronized with upstream updates +- Maintain the port over time +- Share the port with the community + +By using these advanced techniques, you can successfully port even complex libraries like ASIO, mosquitto, and libwebsockets to ESP-IDF while maintaining clean, maintainable code. + +For more examples, explore the implementations in the [ESP-Protocols](https://github.com/espressif/esp-protocols) repository, which contains many ported libraries demonstrating these techniques in practice.