ci: parallel system tests

[gnufede]

CI: Slashes system tests running time in half (~40 min to ~20).

Main drawback for this approach:
Doubles actual usage time. We can group the scenarios to reduce the overhead, since most of them take ~1 min to run, and now an extra ~1 min to fetch the docker images.

Checklist

• Change(s) are motivated and described in the PR description
• Testing strategy is described if automated tests are not included in the PR
• Risks are described (performance impact, potential for breakage, maintainability)
• Change is maintainable (easy to change, telemetry, documentation)
• Library release note guidelines are followed or label changelog/no-changelog is set
• Documentation is included (in-code, generated user docs, public corp docs)
• Backport labels are set (if applicable)
• If this PR changes the public interface, I've notified @DataDog/apm-tees.
• If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.

Reviewer Checklist

• Title is accurate
• All changes are related to the pull request's stated goal
• Description motivates each change
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Change is maintainable (easy to change, telemetry, documentation)
• Release note makes sense to a user of the library
• Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: gnufede/ci-parallel-system-tests
Commit report: 7a9eac1
Test service: dd-trace-py

✅ 0 Failed, 157566 Passed, 14834 Skipped, 10h 1m 2.14s Total duration (1h 39m 40.59s time saved)

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-03-27 20:22:44

Comparing candidate commit 54570d7 in PR branch gnufede/ci-parallel-system-tests with baseline commit be880c1 in branch main.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 199 metrics, 9 unstable metrics.

scenario:httppropagationextract-tracecontext_headers

• 🟥 max_rss_usage [+571.409KB; +755.695KB] or [+2.695%; +3.564%]

scenario:sethttpmeta-all-enabled

• 🟥 max_rss_usage [+488.327KB; +715.488KB] or [+2.265%; +3.319%]

[emmettbutler]

Amazing

[github-actions]

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.5 2.5
# Navigate to the new working tree
cd .worktrees/backport-2.5
# Create a new branch
git switch --create backport-8791-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ef4d80459a594b8c70610b03b74593c6c997133c
# Push it to GitHub
git push --set-upstream origin backport-8791-to-2.5
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport-8791-to-2.5.

[github-actions]

The backport to 2.6 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.6 2.6
# Navigate to the new working tree
cd .worktrees/backport-2.6
# Create a new branch
git switch --create backport-8791-to-2.6
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ef4d80459a594b8c70610b03b74593c6c997133c
# Push it to GitHub
git push --set-upstream origin backport-8791-to-2.6
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.6

Then, create a pull request where the base branch is 2.6 and the compare/head branch is backport-8791-to-2.6.

[github-actions]

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.7 2.7
# Navigate to the new working tree
cd .worktrees/backport-2.7
# Create a new branch
git switch --create backport-8791-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ef4d80459a594b8c70610b03b74593c6c997133c
# Push it to GitHub
git push --set-upstream origin backport-8791-to-2.7
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport-8791-to-2.7.

[emmettbutler]

@gnufede this broke on the main branch, so i reverted it