Investigate "Profiler disabled because there was a problem in setting up the libddprof dependency" #2068
Assignees
Labels
profiling
Involves Datadog profiling
Comments
|
The issue seems to be triggered by the dd-trace-rb/ext/ddtrace_profiling_native_extension/extconf.rb Lines 122 to 123 in 625f115 But Ruby does not seem to log a lot of information when that happens. I'm researching a way to properly expose the logs when this failure happens. Will update this ticket once I have more details. |
ivoanjo
added a commit
that referenced
this issue
Jun 7, 2022
To avoid impacting customers, whenever we detect an issue which would've led to the profiling native extension compilation to fail, we instead skip the compilation. This ensures that customers (especially those *not* using profiling) don't get impacted at installation time. Instead, at execution time, if they do try to enable profiling, they'll get a message such as ``` WARN -- ddtrace: [ddtrace] Profiling was requested but is not supported, profiling disabled: Your ddtrace installation is missing support for the Continuous Profiler because there was a problem in setting up the libddprof dependency. For help solving this issue, please contact Datadog support at https://docs.datadoghq.com/help/. ``` But while investigating one such issue reported by a customer (<#2068>), I found that it's helpful to have a way to break during installation, because that will make Ruby print extra debug information, including the location of the `mkmf.log` log file which can be useful when debugging issues. Thus, I've added a new `DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION` environment variable that support can request customers set to `true` to force a failure at installation time. Here's how it looks when I simulate a broken `libddprof` setup by renaming a file: ``` # Installs successfully, but profiling will not work: $ gem install pkg/ddtrace-1.1.0.gem Building native extensions. This could take a while... Successfully installed ddtrace-1.1.0 Parsing documentation for ddtrace-1.1.0 Done installing documentation for ddtrace after 3 seconds 1 gem installed # Force failure to show up at gem installation time: $ DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION=true gem install pkg/ddtrace-1.1.0.gem Building native extensions. This could take a while... ERROR: Error installing pkg/ddtrace-1.1.0.gem: ERROR: Failed to build gem native extension. current directory: ~/.rvm/gems/ruby-2.7.6/gems/ddtrace-1.1.0/ext/ddtrace_profiling_native_extension ~/.rvm/rubies/ruby-2.7.6/bin/ruby -I ~/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0 -r ./siteconf20220607-83731-8dqh3w.rb extconf.rb +------------------------------------------------------------------------------+ | ** Preparing to build the ddtrace profiling native extension... ** | | | | If you run into any failures during this step, you can set the | | `DD_PROFILING_NO_EXTENSION` environment variable to `true` e.g. | | `$ DD_PROFILING_NO_EXTENSION=true bundle install` to skip this step. | | | | If you disable this extension, the Datadog Continuous Profiler will | | not be available, but all other ddtrace features will work fine! | | | | If you needed to use this, please tell us why on | | <https://github.com/DataDog/dd-trace-rb/issues/new> so we can fix it :) | | | | Thanks for using ddtrace! You rock! | +------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | Failing installation immediately because the | | `DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION` environment variable is set | | to `true`. | | When contacting support, please include the <mkmf.log> file that is shown | | below. | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=~/.rvm/rubies/ruby-2.7.6/bin/$(RUBY_BASE_NAME) --with-ddprof_ffi_with_rpath_x-config --without-ddprof_ffi_with_rpath_x-config --with-pkg-config --without-pkg-config To see why this extension failed to compile, please check the mkmf.log which can be found here: ~/.rvm/gems/ruby-2.7.6/extensions/x86_64-darwin-20/2.7.0/ddtrace-1.1.0/mkmf.log extconf failed, exit code 1 Gem files will remain installed in ~/.rvm/gems/ruby-2.7.6/gems/ddtrace-1.1.0 for inspection. Results logged to ~/.rvm/gems/ruby-2.7.6/extensions/x86_64-darwin-20/2.7.0/ddtrace-1.1.0/gem_make.out ``` You'll notice that a bunch more useful information shows up, including the path to `mkmf.log`, which can provide extra information.
ivoanjo
added a commit
that referenced
this issue
Jun 7, 2022
To avoid impacting customers, whenever we detect an issue which would've led to the profiling native extension compilation to fail, we instead skip the compilation. This ensures that customers (especially those *not* using profiling) don't get impacted at installation time. Instead, at execution time, if they do try to enable profiling, they'll get a message such as ``` WARN -- ddtrace: [ddtrace] Profiling was requested but is not supported, profiling disabled: Your ddtrace installation is missing support for the Continuous Profiler because there was a problem in setting up the libddprof dependency. For help solving this issue, please contact Datadog support at https://docs.datadoghq.com/help/. ``` But while investigating one such issue reported by a customer (<#2068>), I found that it's helpful to have a way to break during installation, because that will make Ruby print extra debug information, including the location of the `mkmf.log` log file which can be useful when debugging issues. Thus, I've added a new `DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION` environment variable that support can request customers set to `true` to force a failure at installation time. Here's how it looks when I simulate a broken `libddprof` setup by renaming a file: ``` # Installs successfully, but profiling will not work: $ gem install pkg/ddtrace-1.1.0.gem Building native extensions. This could take a while... Successfully installed ddtrace-1.1.0 Parsing documentation for ddtrace-1.1.0 Done installing documentation for ddtrace after 3 seconds 1 gem installed # Force failure to show up at gem installation time: $ DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION=true gem install pkg/ddtrace-1.1.0.gem Building native extensions. This could take a while... ERROR: Error installing pkg/ddtrace-1.1.0.gem: ERROR: Failed to build gem native extension. current directory: ~/.rvm/gems/ruby-2.7.6/gems/ddtrace-1.1.0/ext/ddtrace_profiling_native_extension ~/.rvm/rubies/ruby-2.7.6/bin/ruby -I ~/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0 -r ./siteconf20220607-83731-8dqh3w.rb extconf.rb +------------------------------------------------------------------------------+ | ** Preparing to build the ddtrace profiling native extension... ** | | | | If you run into any failures during this step, you can set the | | `DD_PROFILING_NO_EXTENSION` environment variable to `true` e.g. | | `$ DD_PROFILING_NO_EXTENSION=true bundle install` to skip this step. | | | | If you disable this extension, the Datadog Continuous Profiler will | | not be available, but all other ddtrace features will work fine! | | | | If you needed to use this, please tell us why on | | <https://github.com/DataDog/dd-trace-rb/issues/new> so we can fix it :) | | | | Thanks for using ddtrace! You rock! | +------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | Failing installation immediately because the | | `DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION` environment variable is set | | to `true`. | | When contacting support, please include the <mkmf.log> file that is shown | | below. | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=~/.rvm/rubies/ruby-2.7.6/bin/$(RUBY_BASE_NAME) --with-ddprof_ffi_with_rpath_x-config --without-ddprof_ffi_with_rpath_x-config --with-pkg-config --without-pkg-config To see why this extension failed to compile, please check the mkmf.log which can be found here: ~/.rvm/gems/ruby-2.7.6/extensions/x86_64-darwin-20/2.7.0/ddtrace-1.1.0/mkmf.log extconf failed, exit code 1 Gem files will remain installed in ~/.rvm/gems/ruby-2.7.6/gems/ddtrace-1.1.0 for inspection. Results logged to ~/.rvm/gems/ruby-2.7.6/extensions/x86_64-darwin-20/2.7.0/ddtrace-1.1.0/gem_make.out ``` You'll notice that a bunch more useful information shows up, including the path to `mkmf.log`, which can provide extra information.
|
I've been looking into this issue, but discovered that there's a bug/limitation in rubygems where the log file we would need to debug this issue is overwritten because dd-trace-rb ships two native extensions that both need compiling. @rgrey Would you be open to trying a development build of ddtrace that should be able to log more information about why the failure is happening? (If not, please do let me know, I can look into speeding up a release with this extended logging) You can get it from https://s3.amazonaws.com/gems.datadoghq.com/prerelease-v2/gems/ddtrace-1.1.0.ivoanjo.improve.native.extension.debugging.232295.gem and here's how to install it:
I've added the |
|
|
Thanks for the log! Just to add some context, When a Ruby native extension wants to compile and link to an external native library, Could you share the results of running the following commands:
I suspect these two final clues will allow me to get to the bottom of this issue and produce a fix for your situation. Thanks for the patience so far! |
It appears to build fine if we install pkg-config 😁 |
|
Great! It's really surprising that the Ruby Thanks a lot for the help debugging this! 🙇 |
ivoanjo
added a commit
that referenced
this issue
Jul 7, 2022
In #2068 a customer reported that they were not getting profiling with the message: > Profiling was requested but is not supported, profiling disabled: > Your ddtrace installation is missing support for the Continuous > Profiler because there was a problem in setting up the libddprof > dependency. After investigating this issue (and adding extra tools to debug it, see #2069) it turns out that the customer was missing the `pkg-config` tool. This tool is currently invoked indirectly via Ruby's `mkmf` helper, and is used to configure linking to libddprof/libdatadog. I must admit I was surprised that there's not better error logging in `mkmf` when the `pkg-config` tool is actually missing (vs it being installed but returning an error when being called). Thus, to hopefully avoid other customers running into this issue, I've added a bit of code to detect it, and hopefully present a better error message in that situation. I've also learned that `pkg-config` is "old news" on some Linux distributions, and instead they ship something called `pkgconf` which is a reimplementation of `pkg-config`. Beyond the RSpec tests, this can be triggered for testing by: 1. Changing the `pkg_config` call on extconf.rb ```diff -unless pkg_config('ddprof_ffi_with_rpath') +unless pkg_config('ddprof_ffi_with_rpath_broken') ``` which triggers the generic message (because pkg-config is available but returns an error since the configuration file is not found): ``` +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` 2. Crippling `pkg_config`: ```bash $ docker-compose run --no-deps --rm tracer-2.1 /bin/bash # doing this inside docker is fine because it doesn't persist $ rm /usr/bin/pkg-config $ bundle exec rake clean compile # ... +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | the `pkg-config` system tool is missing. | | This issue can usually be fixed by installing: | | 1. the `pkg-config` package on Homebrew and Debian/Ubuntu-based Linux; | | 2. the `pkgconf` package on Arch and Alpine-based Linux; | | 3. the `pkgconf-pkg-config` package on Fedora/Red Hat-based Linux. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` Closes #2068
ivoanjo
added a commit
that referenced
this issue
Jul 7, 2022
In #2068 a customer reported that they were not getting profiling with the message: > Profiling was requested but is not supported, profiling disabled: > Your ddtrace installation is missing support for the Continuous > Profiler because there was a problem in setting up the libddprof > dependency. After investigating this issue (and adding extra tools to debug it, see #2069) it turns out that the customer was missing the `pkg-config` tool. This tool is currently invoked indirectly via Ruby's `mkmf` helper, and is used to configure linking to libddprof/libdatadog. I must admit I was surprised that there's not better error logging in `mkmf` when the `pkg-config` tool is actually missing (vs it being installed but returning an error when being called). Thus, to hopefully avoid other customers running into this issue, I've added a bit of code to detect it, and hopefully present a better error message in that situation. I've also learned that `pkg-config` is "old news" on some Linux distributions, and instead they ship something called `pkgconf` which is a reimplementation of `pkg-config`. Beyond the RSpec tests, this can be triggered for testing by: 1. Changing the `pkg_config` call on extconf.rb ```diff -unless pkg_config('ddprof_ffi_with_rpath') +unless pkg_config('ddprof_ffi_with_rpath_broken') ``` which triggers the generic message (because pkg-config is available but returns an error since the configuration file is not found): ``` +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` 2. Crippling `pkg_config`: ```bash $ docker-compose run --no-deps --rm tracer-2.1 /bin/bash # doing this inside docker is fine because it doesn't persist $ rm /usr/bin/pkg-config $ bundle exec rake clean compile # ... +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | the `pkg-config` system tool is missing. | | This issue can usually be fixed by installing: | | 1. the `pkg-config` package on Homebrew and Debian/Ubuntu-based Linux; | | 2. the `pkgconf` package on Arch and Alpine-based Linux; | | 3. the `pkgconf-pkg-config` package on Fedora/Red Hat-based Linux. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` Closes #2068
ivoanjo
added a commit
that referenced
this issue
Jul 7, 2022
In #2068 a customer reported that they were not getting profiling with the message: > Profiling was requested but is not supported, profiling disabled: > Your ddtrace installation is missing support for the Continuous > Profiler because there was a problem in setting up the libddprof > dependency. After investigating this issue (and adding extra tools to debug it, see #2069) it turns out that the customer was missing the `pkg-config` tool. This tool is currently invoked indirectly via Ruby's `mkmf` helper, and is used to configure linking to libddprof/libdatadog. I must admit I was surprised that there's not better error logging in `mkmf` when the `pkg-config` tool is actually missing (vs it being installed but returning an error when being called). Thus, to hopefully avoid other customers running into this issue, I've added a bit of code to detect it, and hopefully present a better error message in that situation. I've also learned that `pkg-config` is "old news" on some Linux distributions, and instead they ship something called `pkgconf` which is a reimplementation of `pkg-config`. Beyond the RSpec tests, this can be triggered for testing by: 1. Changing the `pkg_config` call on extconf.rb ```diff -unless pkg_config('ddprof_ffi_with_rpath') +unless pkg_config('ddprof_ffi_with_rpath_broken') ``` which triggers the generic message (because pkg-config is available but returns an error since the configuration file is not found): ``` +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` 2. Crippling `pkg_config`: ```bash $ docker-compose run --no-deps --rm tracer-2.1 /bin/bash # doing this inside docker is fine because it doesn't persist $ rm /usr/bin/pkg-config $ bundle exec rake clean compile # ... +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | the `pkg-config` system tool is missing. | | This issue can usually be fixed by installing: | | 1. the `pkg-config` package on Homebrew and Debian/Ubuntu-based Linux; | | 2. the `pkgconf` package on Arch and Alpine-based Linux; | | 3. the `pkgconf-pkg-config` package on Fedora/Red Hat-based Linux. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` Closes #2068
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This was reported by @rgrey in #2045 (comment) , and I've extracted it to a separate issue so it can be investigated separately.
The issue is that profiling does not work with the following error message:
I'll add the next steps for this investigation soon.
The text was updated successfully, but these errors were encountered: