[PROF-7361] Implement "no signals" workaround and enable CPU Profiling 2.0 for all customers

[ivoanjo]

What does this PR do?:

This PR adds a new mode to the profiler, which we're calling the "no signals" workaround.

It also makes the new CPU Profiling 2.0 the default for Ruby 2.3, 2.4 and 2.5, and automatically enables the "no signals" workaround on these Ruby versions.

Motivation:

This mode is a workaround for a few incompatibility issues with the new CPU Profiling 2.0 profiler. In #2702, when we made CPU Profiling 2.0 the default mode, we listed two cases where we were falling back to the legacy profiler:

• Some gems use native libraries that don't handle the EINTR error code that can be caused by a signal handler interrupting a system call
• Ruby 2.5 or below are missing an API that allows the profiler to detect if the current thread is holding on to the Global VM Lock in the signal handler, which we suspect may lead to very rare crashes

The objective of the "no signals workaround" is to allow the new CPU Profiling 2.0 to be used by customers in the above situations, both of which are related to signal handling.

Thus, we're able to migrate every customer to the new codebase, allowing them to access the new features provided by the profiler, and allowing us to completely retire the legacy profiler.

(I'll still keep the legacy profiler alive for a couple more releases, but the plan is to delete it soon).

Additional Notes:

The "no signals workaround" mode works by making the CPU Profiling 2.0 profiler behave in a similar way to the legacy profiler when deciding to take samples.

That is, the CPU Profiling 2.0 profiler by default runs in a background thread that does not hold the global VM lock and thus is running in parallel with the Ruby application.

When it decides it wants to take a sample, it sends a unix SIGPROF signal to the currently-active Ruby thread, temporarily interrupting what that thread is doing to ask the VM to take a sample.

This allows us to get higher quality data, because it's the profiler that decides when it wants to take a sample and interrupts the code to do so.

The "no signals workaround" instead modifies the above to have the profiler grab the global VM lock when it wants to take a sample, instead of interrupting the currently-active Ruby thread.

This means that instead of sampling immediately, the profiler needs to wait for a Ruby VM scheduling switch point, and for the Ruby VM scheduler to actually schedule our thread to execute.

This means that the profiler may need to wait dozens or hundreds of milliseconds to sample. But... it works!

And the above is effectively how the legacy profiler worked -- since it was written in pure Ruby code, it always needed to wait for its turn to run.

Implementing the new behavior is not a lot of work because we already had the scaffolding to do it as part of being able to sample in periods of idleness (see #2468). So most of the work on this was validating and adding the needed configuration.

How to test the change?:

Validate that in the situations described above where the legacy profiler would be used no longer happen -- instead, the new profiler gets used, and the "no signals workaround" gets enabled.

[ivoanjo]

The specs that are changing from "false" to "true" are doing so because they used to belong to .enable_new_profiler? and previously they were reasons NOT to enable the new profiler.

Now the specs belong to .no_signals_workaround_enabled? and ARE reasons to enable the workaround. But much of the logic remained, which is why the diff is slightly weird :)

[GustavoCaso]

Why negate state->no_signals_workaround_enabled?

[ivoanjo]

Since having that option disabled is the "uncommon path", I decided to keep the common path at the top, but no strong feelings either way :)

[GustavoCaso]

I always find confusing the ! part.

[ivoanjo]

Changed in 5e4b131

[GustavoCaso]

Should we tell which are the valid values?

[ivoanjo]

Changed in 69fce3c

[GustavoCaso]

It looks like when initializing the component lib/datadog/profiling/component.rb we log enough information about what would happen if we use no_signals_workaround

Do we still need this log line?

[ivoanjo]

Good catch. I initially thought this may be useful for debugging, but yeah thinking about it again let's just remove it 09fef75 .

[GustavoCaso]

I'll set up another quick chat to go over some of the details of what this method does. I believe I more or less understand what it is doing, but I'm happy to go more into the details 😄

[ivoanjo]

Yup, happy to do it!

[GustavoCaso]

Minor cosmetic comments. The overall code LGTM 😄

[codecov-commenter]

Codecov Report

Merging #2873 (09fef75) into master (c38d420) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2873      +/-   ##
==========================================
+ Coverage   98.09%   98.10%   +0.01%     
==========================================
  Files        1270     1270              
  Lines       70023    70111      +88     
  Branches     3163     3174      +11     
==========================================
+ Hits        68686    68781      +95     
+ Misses       1337     1330       -7     

Impacted Files	Coverage Δ
...g/profiling/collectors/cpu_and_wall_time_worker.rb	100.00% <ø> (ø)
lib/datadog/core/configuration/settings.rb	100.00% <100.00%> (+0.70%)	⬆️
lib/datadog/profiling/component.rb	97.40% <100.00%> (+0.38%)	⬆️
spec/datadog/core/configuration/settings_spec.rb	100.00% <100.00%> (ø)
...filing/collectors/cpu_and_wall_time_worker_spec.rb	95.91% <100.00%> (+0.22%)	⬆️
spec/datadog/profiling/component_spec.rb	99.62% <100.00%> (+0.03%)	⬆️

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more