feat(ddtrace): disable telemetry collection by default

[mabdinur]

What does this PR do?

Disables ddtrace instrumentation telemetry data collection by setting the following environment variable: DD_INSTRUMENTATION_TELEMETRY_ENABLED=True. This change is required to use ddtrace>=1.5 in serverless environments to fix a performance regression in cold starts.

Where this feature is enabled in the ddtrace library: https://github.com/DataDog/dd-trace-py/blob/v1.5.3/ddtrace/internal/writer.py#L514

Motivation

Instrumentation Telemetry is designed to collect, transform and visualize usage, diagnostics and onboarding data for the various APM client libraries (tracer/profiler/debugger/appsec/etc.). The data collected and sent to the datadog backend contains information about an application (dependencies, traced libraries, python versions and implementations, etc.), host/environment data (os, hostname, kernel, container id, etc.) and ddtrace configurations. Collecting this data in lambda functions can increase cold starts by up to 2 seconds and the overhead of the tracer (initial tests show a 5-10% increase but this needs further verification).

Note - Enabling instrumentation telemetry is required for application security products.

Testing

Use the changes in this branch and ddtrace>=1.5 to verify the performance overhead of tracing lambda functions. Test hot and cold starts with the following cases:

• case 1 (baseline): DD_INSTRUMENTATION_TELEMETRY_ENABLED=False.
  • expected result: Performance is similar to ddtrace v1.4
• case 2: DD_INSTRUMENTATION_TELEMETRY_ENABLED is not set.
  • expected result: Same as case 1
• case 3: DD_INSTRUMENTATION_TELEMETRY_ENABLED=True.
  • expected result: Some additional overhead. Add results to this PR description.

If the results from case 3 are acceptable then this PR should be closed. If the additional overhead is not acceptable then we should move forward with this change.

Types of Changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)

[astuyve]

sorry friends, I didn't realize I missed this in my notifications. Do we still need this?

[astuyve]

Closing for now, will reopen when we determine cost/benefit for this.