feat: BuildPulse flaky test detection

[janpio]

This PR adds BuildPulse integration to our test GitHub Actions workflow:

Automatically detect flaky tests. Focus on the top offenders to quickly improve your builds. Save developer time and frustration. Regain trust in your test suite.

That sounds like something we can use right now.

Currently it adds the reporter only to our no-docker MacOS runs, where I think we have the most flaky tests right now.

To make the BuildPulse integration smoother, I also extracted the Client tests from the current no-docker job into a no-docker-client at the cost of adding 4 jobs per workflow run. I think that is worth it to get a result quicker. (Along the way I resorted our jest.config.jss to be the same order of properties)

PS: BuildPulse is currently working on a Windows reporter, as soon as that is available we can also add it for the Windows matrix runs. If Buildpulse turns out to work correctly, we can also add the reporting to our other Linux jobs of course.

Thanks @jasonrudolph for the concierge service with this.

[jasonrudolph]

👋 @janpio: A couple suggestions to make sure that forks don't run into any issues attempting to upload test results. I hope this helps. If you have any questions, just let me know. 🙇

[janpio]

The last thing left @jasonrudolph is to get the check for the OS working (before it was always skipped for some reason) until you have the Windows reporter ready (and then we will need to adapt the script to download the correct binary of course). Then I think we would be good for a first merge to get actual test run data.

[jasonrudolph]

@janpio: That sounds right to me. We need to teach these two lines to only run for macOS:

prisma/.github/workflows/test.yml

Line 727 in 2e140ec

if: startsWith(github.repository, 'prisma/') && !cancelled()

prisma/.github/workflows/test.yml

Line 813 in 2e140ec

if: startsWith(github.repository, 'prisma/') && !cancelled()

I'll see if I can figure out the right syntax, and I'll let you know what I find out.

[jasonrudolph]

Following up on #11967 (comment), I think these suggestions will do the trick.

It turns out that runner.os has the values documented at https://docs.github.com/en/actions/learn-github-actions/contexts#runner-context, so we need to check for macOS as the value.

If we were using matrix.os, then we'd check for macos-latest as the value.

[ghost]

CodeSee Review Map:

Review in an interactive map

View more CodeSee Maps

Legend

[janpio]

Thanks @jasonrudolph, that worked as intended - in the last commit we have a failure on Windows and one on MacOS - and only the MacOS one triggered the BuildPulse step and successfully reported.

We'll try to review and merge this now and see if we can get BuildPulse to recognize some of these as flaky (vs. currently "always broken" I guess 😛). Then we can also try to add Windows (when you have it working) and Linux via the other builds.

[jasonrudolph]

Thanks @jasonrudolph, that worked as intended

@janpio: That's great! Thanks for taking the time to get things set up. ⚡

[millsp]

Nice, thank you! Let's try this new tool out 🚀