chore(ci_visibility): introduce experimental internal coverage collector

[romainkomorndatadog]

Introduces the ModuleCodeCollector which collects coverage and executable lines for imported modules.

The collector has two modes, one that stores executed lines on the singleton instance, and one that uses context variables and a context manager.

This also introduces changes to the pytest integration as well as the CIVisibility service's use of coverage to feature-flag using the new module collector.

The features are gated behind _DD_USE_INTERNAL_COVERAGE and _DD_COVER_SESSION (which introduces a new coverage run like behavior).

There are no unit tests though the overall use of the feature flags has been tested quite extensively in the process of collecting performance data.

There are no release notes since this is an entirely undocumented feature for the moment.

Checklist

• Change(s) are motivated and described in the PR description
• Testing strategy is described if automated tests are not included in the PR
• Risks are described (performance impact, potential for breakage, maintainability)
• Change is maintainable (easy to change, telemetry, documentation)
• Library release note guidelines are followed or label changelog/no-changelog is set
• Documentation is included (in-code, generated user docs, public corp docs)
• Backport labels are set (if applicable)
• If this PR changes the public interface, I've notified @DataDog/apm-tees.
• If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.

Reviewer Checklist

• Title is accurate
• All changes are related to the pull request's stated goal
• Description motivates each change
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Change is maintainable (easy to change, telemetry, documentation)
• Release note makes sense to a user of the library
• Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[romainkomorndatadog]

For posterity's sake, I insist on mentioning that the overwhelming majority of this work comes from @P403n1x87 , and I'm just piggybacking my changes on top.

Code review should mainly focus on the safety of the feature flags (eg: that we don't break things unless people are using undocumented flags from unsupported features).

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-05-22 09:20:02

Comparing candidate commit d5fcbec in PR branch romainkomorn/add_clear with baseline commit 2008dd7 in branch main.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 199 metrics, 9 unstable metrics.

scenario:span-start

• 🟥 max_rss_usage [+6.917MB; +6.968MB] or [+22.922%; +23.090%]

scenario:span-start-traceid128

• 🟥 max_rss_usage [+6.477MB; +6.526MB] or [+20.678%; +20.835%]

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: romainkomorn/add_clear
Commit report: 8a68963
Test service: dd-trace-py

✅ 0 Failed, 117032 Passed, 58751 Skipped, 3h 54m 40.27s Total duration (5h 15m 39.44s time saved)
⌛ 6 Performance Regressions

⌛ Performance Regressions vs Default Branch (6)

This report shows up to 5 performance regressions.

• test_patch_unpatch_unpatch_import - test_aiopg_patch.py 3.7s (+3.06s, +478%) - Details
• test_patch_import_unpatch_patch - test_aiopg_patch.py 3.28s (+2.64s, +413%) - Details
• test_patch_unpatch_import - test_aiopg_patch.py 9.9s (+9.27s, +1465%) - Details
• test_import_unpatch_patch - test_aiopg_patch.py 7.5s (+6.86s, +1071%) - Details
• test_import_patch_unpatch_unpatch - test_aiopg_patch.py 3.29s (+2.64s, +411%) - Details

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 23.71134% with 296 lines in your changes are missing coverage. Please review.

Project coverage is 10.46%. Comparing base (d8c21b0) to head (43e7b02).

Files	Patch %	Lines
ddtrace/internal/coverage/code.py	29.13%	107 Missing ⚠️
ddtrace/internal/coverage/report.py	16.52%	96 Missing ⚠️
ddtrace/internal/ci_visibility/coverage.py	19.23%	21 Missing ⚠️
ddtrace/internal/coverage/instrumentation.py	26.08%	17 Missing ⚠️
ddtrace/contrib/pytest/plugin.py	20.00%	16 Missing ⚠️
ddtrace/contrib/pytest/_plugin_v1.py	0.00%	14 Missing ⚠️
ddtrace/internal/module.py	46.15%	14 Missing ⚠️
ddtrace/internal/coverage/util.py	15.38%	11 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #8727       +/-   ##
===========================================
- Coverage   76.17%   10.46%   -65.72%     
===========================================
  Files        1288     1262       -26     
  Lines      122505   121045     -1460     
===========================================
- Hits        93321    12665    -80656     
- Misses      29184   108380    +79196     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.