ci: add pip cache to CI jobs

[brettlangdon]

Adds a pip cache to a large number of our CI jobs.

For riot: this does this by using a unique pip cache key per-job based on the sha256sum of all of the .riot/requirements/*.txt files for all the venvs matching the suite name or pattern. This way if any deps change we'll invalidate the cache to ensure it doesn't grow in size forever.

For hatch: this uses the job name as the key.

The goal is to cache the downloaded wheels/distributions to save on network bandwidth and effort for downloading every time. This will also help cache built wheels if the downloaded distribution is a source distribution.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

.gitlab/scripts/get-riot-hashes.sh                                      @DataDog/python-guild @DataDog/apm-core-python
.gitlab/scripts/get-riot-pip-cache-key.sh                               @DataDog/python-guild @DataDog/apm-core-python
.gitlab/tests.yml                                                       @DataDog/python-guild @DataDog/apm-core-python
scripts/gen_gitlab_config.py                                            @DataDog/apm-core-python

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and main.

Summary

The average import time in this PR is: 246 ± 6 ms.

The average import time in main is: 249 ± 6 ms.

The import time difference between this PR and main is: -3.2 ± 0.3 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 2.071 ms (0.84%)

ddtrace.bootstrap.sitecustomize 1.374 ms (0.56%)

ddtrace.bootstrap.preload 1.374 ms (0.56%)

ddtrace.internal.products 1.374 ms (0.56%)

ddtrace.internal.remoteconfig.client 0.690 ms (0.28%)

ddtrace 0.697 ms (0.28%)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-03-25 16:13:20

Comparing candidate commit 835b284 in PR branch brettlangdon/riot-pip-cache with baseline commit 03a5ea0 in branch main.

Found 2 performance improvements and 0 performance regressions! Performance is the same for 496 metrics, 2 unstable metrics.

scenario:iast_aspects-format_map_aspect

• 🟩 execution_time [-1.197µs; -1.158µs] or [-23.908%; -23.121%]

scenario:iast_aspects-replace_aspect

• 🟩 execution_time [-1.422µs; -1.373µs] or [-21.414%; -20.679%]

[Copilot]

Pull Request Overview

This PR adds pip caching to various CI jobs to improve performance and reduce repeated downloads of packages. Key changes include:

• Adding pip cache commands and environment variables to the GitLab CI configuration for both "riot" and "hatch" runners.
• Introducing a new mechanism to generate a unique pip cache key for "riot" jobs using a SHA256 sum from requirements files.
• Updating the docs and precheck job configurations to include pip caching details.

Reviewed Changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 1 comment.

File	Description
scripts/gen_gitlab_config.py	Introduces pip cache configuration with dynamic key generation and environment variable setup for CI jobs.
.gitlab/tests.yml	Updates the hash generation for riot tests by replacing the inline command with a dedicated script.

Files not reviewed (2)

• .gitlab/scripts/get-riot-hashes.sh: Language not supported
• .gitlab/scripts/get-riot-pip-cache-key.sh: Language not supported