ddprof_lib_state undefined symbol when loader is dlopen'd with RTLD_GLOBAL

[xroche]

Loading libdd_profiling.so at runtime via dlopen("libdd_profiling.so", RTLD_LAZY | RTLD_GLOBAL) fails:

undefined symbol: ddprof_lib_state

This breaks profiling for applications that dlopen the library instead of linking it at build time (e.g., toggling profiling via a config flag).

Root cause

ddprof_lib_state (a TLS variable added in #490 for fork safety) is defined in the loader and referenced as extern in the embedded .so.

The loader's constructor (loader.c:251) calls my_dlopen(embedded.so, RTLD_LOCAL | RTLD_NOW) during its own initialization. On glibc, RTLD_GLOBAL takes effect after dlopen returns, not while the loaded library's constructors run. The loader's symbols are not yet in global scope when the embedded library tries to resolve ddprof_lib_state.

v0.23.0 worked because the embedded .so had no undefined symbols referencing the loader.

Reproduction

void *h = dlopen("libdd_profiling.so", RTLD_LAZY | RTLD_GLOBAL);
// Constructor fails: embedded .so can't resolve ddprof_lib_state
int rc = ddprof_start_profiling(); // returns -1

[r1viollet]

Delivery of your proposed fix here

[r1viollet]

So the fix is not as trivial because we moved to some initial-exec TLS to ensure that we would never be in a case where the TLS storage is not available. On musl, initial-exec TLS does not work with dlopen. The thought was that dlopen is not how users generally use ddprof, however it seems that this is your use case.
Would it be possible link differently, or is the link through dlopen important to you ?

I reconsidered changing the TLS model to allow dlopen here. However these code paths are not as safe as the current strategy. This was what I had originally proposed. It would require more test & review work.

[xroche]

So the fix is not as trivial because we moved to some initial-exec TLS to ensure that we would never be in a case where the TLS storage is not available. On musl, initial-exec TLS does not work with dlopen. The thought was that dlopen is not how users generally use ddprof, however it seems that this is your use case. Would it be possible link differently, or is the link through dlopen important to you ?

We really don't want to enable the profiler and/or link it all the time due to the augmented surface risk (the code being a little sensitive in term of privileges), we'd really prefer to load it on demand when we need profiling.

And the ddprof wrapper variant doesn't work with our case for various reasons (capabilities, fork...)

[r1viollet]

Thanks for the feedback, PR is up to date with what I feel would address the dlopen use case. However, this has a minor impact on performances & TLS code paths more slightly complex.

[xroche]

Thanks for the feedback, PR is up to date with what I feel would address the dlopen use case. However, this has a minor impact on performances & TLS code paths more slightly complex.

Indeed I was about to suggest to move to pthread API!

Tentative PR on top of this PR to simplify the key handling:
#523