Lazily activate integrations

[delner]

Today, in a configuration block...

Datadog.configure do |c|
  c.use :faraday
end

c.use configures and patches the integration immediately. This has the unwanted side effect of causing trace components being initialized with default settings before the configuration block is completed, because patching integrations often calls Datadog.logger which is a trace component.

In this pull request, c.use adds the integration to a list of pending integrations, which are each patched after the configuration block completes and the trace components are initialized. This prevents trace components from being re-initialized multiple times during setup.

[delner]

Rails minitests don't like these changes for some reason, still haven't figured it out. Once CI is green this will be ready for review.

[delner]

Figured out why the Rails tests were failing.

When Rails is configured with Datadog.configure { |c| c.use :rails }, its patch behavior is to hook into before_initialize and after_initialize callbacks in the Rails application lifecycle. These callbacks are executed later to complete patching and configuration for Rails. It does this because there are some configuration settings like the application name and Rack application stack that are not available until later in the Rails startup process.

Prior to this PR, the after_initialize hook was updating Datadog configuration by calling Datadog.configuration.use :action_pack and other similar commands for the other Rails components. This was okay because patching ran immediately on c.use, so those components were patched properly.

In this PR after we switched to lazy patching, c.use no longer applies patches, it 'queues' integrations to be patched. Since patches are only applied when Datadog.configure completes, these patches were remaining 'queued.' The Datadog.configuration.use :action_pack command in after_initialize was thus not applying the patch to ActionPack or the other components. To fix this, we have to Datadog.configure.

This is not the only reason we have to Datadog.configure: since the default service name can only be sourced from Rails after its loaded, it must update c.service in after_initialize as well. In order for the tracer and other components to appropriately use this updated service name, they must be rebuilt, and Datadog.configure is the function that triggers that behavior.

The unfortunate side effect is that for Rails applications, the tracer components are effectively initialized twice; once after Datadog.configure in initializers/datadog.rb and a second time after the Rails application finishes loading in after_initialize. This should have no functional impact on Rails applications, and the cost of rebuilding is not very expensive, but it is nonetheless undesirable and (thus far seemingly) unavoidable.

Despite this, I think we do have a reasonable solution in place, and while it isn't perfect, it stands to grant benefits that justify any imperfections we've seen and considered thus far.

[marcotc]

👍

[saarons]

@delner Something might be strange in my configuration but this change has the unfortunate side-effect of creating 2 instances of Datadog::Tracer.

ObjectSpace.each_object(Datadog::Tracer).count #=> 2

One of these is accessible by calling Datadog.tracer in the Rails application, but the other is seemingly not as easy to access. The issue is that Rails is instrumented with the tracer I can't access directly, so calling things like Datadog.tracer.active_correlation just give me empty spans when called inside my app code.

[saarons]

@delner Ignore me, I see you're already on it in #1064