ci: run tests with -shuffle=on

[felixge]

Motivated by #2263

Opening this to see if we can pass CI with this option after the PR above is merged. I suspect there will be more failures ...

What does this PR do?

Motivation

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.

For Datadog employees:

• If this PR touches code that handles credentials of any kind, such as Datadog API keys, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-01-02 12:49:10

Comparing candidate commit 0f9f89a in PR branch felix.geisendoerfer/PROF-8356-shuffle-tests with baseline commit cb699a6 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 40 metrics, 1 unstable metrics.

[evanj]

I think my PRs will fix all the ddtrace/tracer failures here (#2262, #2261, #2260, and #2257). I didn't look at any other packages.

[github-actions]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[evanj]

@felixge and @RomainMuller it looks like this PR passed the last time it was attempted. Do you want to make shuffling tests the default? I'm mostly adding this comment since it was marked stale, so we should either update and merge it, or abandon it.

FWIW: I personally think this is useful to make it less likely to have tests accidentally depending on global state modified by other tests, but also don't think it is super important.

[github-actions]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was closed because it has been open for 30 days with no activity.