add pytest-split to allow splitting test running in groups

[rochdev]

Motivation

System tests are notoriously slow to the point that they are quickly becoming unusable as they take too long to run. While improving their speed would definitely be the better option, having the ability to split the tests on multiple machines is a quick fix that will allow running them faster at least in CI.

Changes

Add pytest-split to allow splitting test running in groups.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes (if something not related to your task is failing, you can ignore it)
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner. We're working on refining the codeowners file quickly.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Relevant label (run-parametric-scenario, run-profiling-scenario...) are presents
• If PR title starts with [<language>], double-check that only <language> is impacted by the change
• No system-tests internal is modified. Otherwise, I have the approval from R&P team
• CI is green, or failing jobs are not related to this change (and you are 100% sure about this statement)
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added (or removed)?
  • Get a review from R&P team

[cbeauchesne]

Hi @rochdev !

What scenarios do you want to split ?

If it's parametric, there is already pytest-xdist on the repo, if it's not activated on your repo, I can help you on that.

If it's other scenarios, unfortunately it won't work, as the test part of the run is actually the fasted, The long part is spawning the infra, and running the set-up, which can't be splitted. The solution here is parallel execution in the CI, where I can also help.

[cbeauchesne]

Added a comment yesterday 👆

[rochdev]

If it's parametric, there is already pytest-xdist on the repo, if it's not activated on your repo, I can help you on that.

First step is parametric, but then other tests as well if at all possible. For xdist, unfortunately it doesn't help at all, and parallelizing on 16 cores didn't make any difference unfortunately (well, a tiny difference of maybe 20% but that could just be faster cores).

[rochdev]

Working example: https://github.com/DataDog/dd-trace-js/actions/runs/9664627172/job/26659723787?pr=4436