Auto-balancing Playwright test shards

[aslushnikov]

Hey folks,

I used to work on Playwright, and I noticed that Supabase uses Playwright too, including sharding in CI.

These days I’m building Flakiness.io: a GitHub-native test analytics dashboard that tracks flaky tests and detects regressions in pull requests.

We’re prototyping Playwright shard balancing, where Flakiness.io uses historical test timings to produce better-balanced shards. Since Supabase has a real-world Playwright setup, I’d be very happy if you’re open to testing it on your workload.

OSS projects can use Flakiness.io for free. I’d be happy to help set it up and see if you find it useful.

P.S. We’re also happy Supabase customers. Thank you for the product!

[aslushnikov]

FYI: we’ve released shard balancing as part of the flakiness/playwright reporter.

To try it out:

1. Create a project on Flakiness.io to store your test history
2. Install flakiness/playwright
3. Replace npx playwright test with npx flakiness-playwright-shard

-npx playwright test --shard 1/2 --project foo
+npx flakiness-playwright-shard --shard 1/2 --project foo

Details: https://docs.flakiness.io/shard-balancing/

Let me know if I can help with the integration - we’d love to see how it works for you.