Add env var for debugging profiling native extension compilation issues + Log PKG_CONFIG_PATH to mkmf.log
#2069
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To avoid impacting customers, whenever we detect an issue which would've led to the profiling native extension compilation to fail, we instead skip the compilation. This ensures that customers (especially those *not* using profiling) don't get impacted at installation time. Instead, at execution time, if they do try to enable profiling, they'll get a message such as ``` WARN -- ddtrace: [ddtrace] Profiling was requested but is not supported, profiling disabled: Your ddtrace installation is missing support for the Continuous Profiler because there was a problem in setting up the libddprof dependency. For help solving this issue, please contact Datadog support at https://docs.datadoghq.com/help/. ``` But while investigating one such issue reported by a customer (<#2068>), I found that it's helpful to have a way to break during installation, because that will make Ruby print extra debug information, including the location of the `mkmf.log` log file which can be useful when debugging issues. Thus, I've added a new `DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION` environment variable that support can request customers set to `true` to force a failure at installation time. Here's how it looks when I simulate a broken `libddprof` setup by renaming a file: ``` # Installs successfully, but profiling will not work: $ gem install pkg/ddtrace-1.1.0.gem Building native extensions. This could take a while... Successfully installed ddtrace-1.1.0 Parsing documentation for ddtrace-1.1.0 Done installing documentation for ddtrace after 3 seconds 1 gem installed # Force failure to show up at gem installation time: $ DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION=true gem install pkg/ddtrace-1.1.0.gem Building native extensions. This could take a while... ERROR: Error installing pkg/ddtrace-1.1.0.gem: ERROR: Failed to build gem native extension. current directory: ~/.rvm/gems/ruby-2.7.6/gems/ddtrace-1.1.0/ext/ddtrace_profiling_native_extension ~/.rvm/rubies/ruby-2.7.6/bin/ruby -I ~/.rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0 -r ./siteconf20220607-83731-8dqh3w.rb extconf.rb +------------------------------------------------------------------------------+ | ** Preparing to build the ddtrace profiling native extension... ** | | | | If you run into any failures during this step, you can set the | | `DD_PROFILING_NO_EXTENSION` environment variable to `true` e.g. | | `$ DD_PROFILING_NO_EXTENSION=true bundle install` to skip this step. | | | | If you disable this extension, the Datadog Continuous Profiler will | | not be available, but all other ddtrace features will work fine! | | | | If you needed to use this, please tell us why on | | <https://github.com/DataDog/dd-trace-rb/issues/new> so we can fix it :) | | | | Thanks for using ddtrace! You rock! | +------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | Failing installation immediately because the | | `DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION` environment variable is set | | to `true`. | | When contacting support, please include the <mkmf.log> file that is shown | | below. | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=~/.rvm/rubies/ruby-2.7.6/bin/$(RUBY_BASE_NAME) --with-ddprof_ffi_with_rpath_x-config --without-ddprof_ffi_with_rpath_x-config --with-pkg-config --without-pkg-config To see why this extension failed to compile, please check the mkmf.log which can be found here: ~/.rvm/gems/ruby-2.7.6/extensions/x86_64-darwin-20/2.7.0/ddtrace-1.1.0/mkmf.log extconf failed, exit code 1 Gem files will remain installed in ~/.rvm/gems/ruby-2.7.6/gems/ddtrace-1.1.0 for inspection. Results logged to ~/.rvm/gems/ruby-2.7.6/extensions/x86_64-darwin-20/2.7.0/ddtrace-1.1.0/gem_make.out ``` You'll notice that a bunch more useful information shows up, including the path to `mkmf.log`, which can provide extra information.
This will help us investigate #2068 .
Codecov Report
@@ Coverage Diff @@
## master #2069 +/- ##
==========================================
- Coverage 97.49% 97.48% -0.01%
==========================================
Files 1031 1031
Lines 53213 53214 +1
==========================================
- Hits 51878 51877 -1
- Misses 1335 1337 +2
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
lloeki
approved these changes
Jun 8, 2022
| @@ -122,6 +140,7 @@ def add_compiler_flag(flag) | |||
|
|
|||
| # If we got here, libddprof is available and loaded | |||
| ENV['PKG_CONFIG_PATH'] = "#{ENV['PKG_CONFIG_PATH']}:#{Libddprof.pkgconfig_folder}" | |||
| Logging.message(" [ddtrace] PKG_CONFIG_PATH set to '#{ENV['PKG_CONFIG_PATH']}'\n") | |||
There was a problem hiding this comment.
I usually favour calling #inspect when logging values: when they're strings they'll be automatically wrapped in quotes, and if they happen to be an unexpected value it becomes very obvious (e.g distinguishing nil vs empty string)
Just a note, as here it's obviously a string.
There was a problem hiding this comment.
Yeap, good point and good rule of thumb, fixed in 455a808
| @@ -122,6 +140,7 @@ def add_compiler_flag(flag) | |||
|
|
|||
| # If we got here, libddprof is available and loaded | |||
| ENV['PKG_CONFIG_PATH'] = "#{ENV['PKG_CONFIG_PATH']}:#{Libddprof.pkgconfig_folder}" | |||
There was a problem hiding this comment.
Can ENV['PKG_CONFIG_PATH'] be nil, or ''? This would result in a leading :.
There was a problem hiding this comment.
Yes, it can. That's actually what happens almost always, since nothing in Ruby sets PKG_CONFIG_PATH by default. I can confirm pkg-config is OK with having the leading :.
ivoanjo
added a commit
that referenced
this pull request
Jul 7, 2022
In #2068 a customer reported that they were not getting profiling with the message: > Profiling was requested but is not supported, profiling disabled: > Your ddtrace installation is missing support for the Continuous > Profiler because there was a problem in setting up the libddprof > dependency. After investigating this issue (and adding extra tools to debug it, see #2069) it turns out that the customer was missing the `pkg-config` tool. This tool is currently invoked indirectly via Ruby's `mkmf` helper, and is used to configure linking to libddprof/libdatadog. I must admit I was surprised that there's not better error logging in `mkmf` when the `pkg-config` tool is actually missing (vs it being installed but returning an error when being called). Thus, to hopefully avoid other customers running into this issue, I've added a bit of code to detect it, and hopefully present a better error message in that situation. I've also learned that `pkg-config` is "old news" on some Linux distributions, and instead they ship something called `pkgconf` which is a reimplementation of `pkg-config`. Beyond the RSpec tests, this can be triggered for testing by: 1. Changing the `pkg_config` call on extconf.rb ```diff -unless pkg_config('ddprof_ffi_with_rpath') +unless pkg_config('ddprof_ffi_with_rpath_broken') ``` which triggers the generic message (because pkg-config is available but returns an error since the configuration file is not found): ``` +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` 2. Crippling `pkg_config`: ```bash $ docker-compose run --no-deps --rm tracer-2.1 /bin/bash # doing this inside docker is fine because it doesn't persist $ rm /usr/bin/pkg-config $ bundle exec rake clean compile # ... +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | the `pkg-config` system tool is missing. | | This issue can usually be fixed by installing: | | 1. the `pkg-config` package on Homebrew and Debian/Ubuntu-based Linux; | | 2. the `pkgconf` package on Arch and Alpine-based Linux; | | 3. the `pkgconf-pkg-config` package on Fedora/Red Hat-based Linux. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` Closes #2068
ivoanjo
added a commit
that referenced
this pull request
Jul 7, 2022
In #2068 a customer reported that they were not getting profiling with the message: > Profiling was requested but is not supported, profiling disabled: > Your ddtrace installation is missing support for the Continuous > Profiler because there was a problem in setting up the libddprof > dependency. After investigating this issue (and adding extra tools to debug it, see #2069) it turns out that the customer was missing the `pkg-config` tool. This tool is currently invoked indirectly via Ruby's `mkmf` helper, and is used to configure linking to libddprof/libdatadog. I must admit I was surprised that there's not better error logging in `mkmf` when the `pkg-config` tool is actually missing (vs it being installed but returning an error when being called). Thus, to hopefully avoid other customers running into this issue, I've added a bit of code to detect it, and hopefully present a better error message in that situation. I've also learned that `pkg-config` is "old news" on some Linux distributions, and instead they ship something called `pkgconf` which is a reimplementation of `pkg-config`. Beyond the RSpec tests, this can be triggered for testing by: 1. Changing the `pkg_config` call on extconf.rb ```diff -unless pkg_config('ddprof_ffi_with_rpath') +unless pkg_config('ddprof_ffi_with_rpath_broken') ``` which triggers the generic message (because pkg-config is available but returns an error since the configuration file is not found): ``` +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` 2. Crippling `pkg_config`: ```bash $ docker-compose run --no-deps --rm tracer-2.1 /bin/bash # doing this inside docker is fine because it doesn't persist $ rm /usr/bin/pkg-config $ bundle exec rake clean compile # ... +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | the `pkg-config` system tool is missing. | | This issue can usually be fixed by installing: | | 1. the `pkg-config` package on Homebrew and Debian/Ubuntu-based Linux; | | 2. the `pkgconf` package on Arch and Alpine-based Linux; | | 3. the `pkgconf-pkg-config` package on Fedora/Red Hat-based Linux. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` Closes #2068
ivoanjo
added a commit
that referenced
this pull request
Jul 7, 2022
In #2068 a customer reported that they were not getting profiling with the message: > Profiling was requested but is not supported, profiling disabled: > Your ddtrace installation is missing support for the Continuous > Profiler because there was a problem in setting up the libddprof > dependency. After investigating this issue (and adding extra tools to debug it, see #2069) it turns out that the customer was missing the `pkg-config` tool. This tool is currently invoked indirectly via Ruby's `mkmf` helper, and is used to configure linking to libddprof/libdatadog. I must admit I was surprised that there's not better error logging in `mkmf` when the `pkg-config` tool is actually missing (vs it being installed but returning an error when being called). Thus, to hopefully avoid other customers running into this issue, I've added a bit of code to detect it, and hopefully present a better error message in that situation. I've also learned that `pkg-config` is "old news" on some Linux distributions, and instead they ship something called `pkgconf` which is a reimplementation of `pkg-config`. Beyond the RSpec tests, this can be triggered for testing by: 1. Changing the `pkg_config` call on extconf.rb ```diff -unless pkg_config('ddprof_ffi_with_rpath') +unless pkg_config('ddprof_ffi_with_rpath_broken') ``` which triggers the generic message (because pkg-config is available but returns an error since the configuration file is not found): ``` +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | there was a problem in setting up the `libddprof` dependency. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` 2. Crippling `pkg_config`: ```bash $ docker-compose run --no-deps --rm tracer-2.1 /bin/bash # doing this inside docker is fine because it doesn't persist $ rm /usr/bin/pkg-config $ bundle exec rake clean compile # ... +------------------------------------------------------------------------------+ | Could not compile the Datadog Continuous Profiler because | | the `pkg-config` system tool is missing. | | This issue can usually be fixed by installing: | | 1. the `pkg-config` package on Homebrew and Debian/Ubuntu-based Linux; | | 2. the `pkgconf` package on Arch and Alpine-based Linux; | | 3. the `pkgconf-pkg-config` package on Fedora/Red Hat-based Linux. | | | | The Datadog Continuous Profiler will not be available, | | but all other ddtrace features will work fine! | | | | For help solving this issue, please contact Datadog support at | | <https://docs.datadoghq.com/help/>. | +------------------------------------------------------------------------------+ ``` Closes #2068
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To avoid impacting customers, whenever we detect an issue which would've led to the profiling native extension compilation to fail, we instead skip the compilation.
This ensures that customers (especially those not using profiling) don't get impacted at installation time.
Instead, at execution time, if they do try to enable profiling, they'll get a message such as
But while investigating one such issue reported by a customer (#2068), I found that it's helpful to have a way to break during installation, because that will make Ruby print extra debug information, including the location of the
mkmf.loglog file which can be useful when debugging issues.Thus, I've added a new
DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSIONenvironment variable that support can request customers set totrueto force a failure at installation time.Here's how it looks when I simulate a broken
libddprofsetup by renaming a file:You'll notice that a bunch more useful information shows up, including the path to
mkmf.log, which can provide extra information.As part of this PR, I've also added logging of the
PKG_CONFIG_PATHto the mkmf.log, so that we can use it to investigate #2068.