feat(test runner): improve sharding algorithm to better spread similar tests among shards #30962

muhqu · 2024-05-22T15:02:25Z

Adds alternative algorithms to assign test groups to shards to better distribute tests.

Problem

Currently the way sharding works is something like this…

         [  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12]
Shard 1:  ^---------^                                      : [  1, 2, 3 ]
Shard 2:              ^---------^                          : [  4, 5, 6 ]
Shard 3:                          ^---------^              : [  7, 8, 9 ]
Shard 4:                                      ^---------^  : [ 10,11,12 ]

Tests are ordered in the way they are discovered, which is mostly alphabetically. This has the effect that test cases are sorted nearby similar tests… for example your have first 6 tests which are testing logged-in state and then 6 tests which test logged-out state. The first 6 tests require more setup time as they are testing logged-in behaviour… With the current sharding algorithm shard 1 & 2 get those slow logged-in tests and shard 3 & 4 get the more quicker tests…

Solution

This PR adds a new shardingMode configuration which allows to specify the sharding algorithm to be used…

`shardingMode: 'partition'`

That's the current behaviour, which is the default. Let me know if you have a better name to describe the current algorithm...

`shardingMode: 'round-robin'`

Distribute the test groups more evenly. It…

sorts test groups by number of tests in descending order
then loops through the test groups and assigns them to the shard with the lowest number of tests.

Here is a simple example where every test group represents a single test (e.g. --fully-parallel) ...

         [  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12]
Shard 1:    ^               ^               ^              : [  1, 5, 9 ]
Shard 2:        ^               ^               ^          : [  2, 6,10 ]
Shard 3:            ^               ^               ^      : [  3, 7,11 ]
Shard 4:                ^               ^               ^  : [  4, 8,12 ]

…or a more complex scenario where test groups have different number of tests…

Original Order: [ [1], [2, 3], [4, 5, 6], [7], [8], [9, 10], [11], [12] ]
Sorted Order:   [ [4, 5, 6], [2, 3], [9, 10], [1], [7], [8], [11], [12] ]
Shard 1:           ^-----^                                                : [ [ 4,   5,   6] ]
Shard 2:                      ^--^                       ^                : [ [ 2,  3],  [8] ]
Shard 3:                              ^---^                    ^          : [ [ 9, 10], [11] ]
Shard 4:                                       ^    ^                ^    : [ [1], [7], [12] ]

`shardingMode: 'duration-round-robin'`

It's very similar to round-robin, but it uses the duration of a tests previous run as cost factor. The duration will be read from .last-run.json when available. When a test can not be found in .last-run.json it will use the average duration of available tests. When no last run info is available, the behaviour would be identical to round-robin.

Other changes

Add testDurations?: { [testId: string]: number } to .last-run.json
Add builtin lastrun reporter, which allows merge-reports to generate a .last-run.json to be generated

Appendix

Below are some runtime stats from a project I've been working on, which shows the potential benefit of this change.

The tests runs had to complete 161 tests. Single test duration ranges from a few seconds to over 2 minutes.

The partition run gives the baseline performance and illustrates the problem quite good. We have a single shard that takes almost 16 min while another one completes in under 5 min.

The round-robin algorithm gives a bit better performance, but it still has a shard that requires twice the time of another shard.

The duration-round-robin run was using the duration info from a previous run and achieves the best result by far. All shards complete in 10-11 minutes. 🏆 🎉

muhqu · 2024-05-22T15:11:53Z

Maybe it's better to make this an option to allow restoring the old behaviour. ¯_(ツ)_/¯

~~And… there should be unit-tests, no?~~ found them…

pavelfeldman · 2024-05-22T23:46:17Z

Do you think you can achieve the same better behavior with your sharding seed? Or are you looking for additional bias against subsequent tests being put into the same group?

muhqu · 2024-05-23T05:45:19Z

Do you think you can achieve the same better behavior with your sharding seed?

Not sure yet. But I will test this new sharding logic in our test setup to gather some results.

Or are you looking for additional bias against subsequent tests being put into the same group?

The seeded shuffle is basically just a quick and easy way to influence the test group to shard assignment… it's random and so it's results may vary.

However, this change is aimed to improve the sharding logic to generally yield better results, which yet needs to be proved. 😅

Currently this sharding algorithm uses the number of tests per test group as a cost metric. It would be great if we could use the test duration of a previous run (when available) to even better distribute the tests among the shards. But the algorithm would be quite similar.

pavelfeldman · 2024-05-24T15:36:32Z

I think your seed change allows users to experiment with the seeds and arrive at some better state than they are today. Any other changes w/o the timing feedback are going to yield similar results, not need to experiment with biases.

It would be great if we could use the test duration of a previous run (when available) to even better distribute the tests among the shards.

This requires a feedback loop with the test time stats which we don't have today. We recently started storing the last run stats in .last-run.json, I think it is Ok to store the test times there and to use it on the subsequent runs for better sharding, if it is available. Would you be interested in working on it?

muhqu · 2024-05-24T15:46:49Z

Yes, I would like to work on that.

I was not yet aware of the .last-run.json. Is that something that is also written by the merge reports command? Because we need the stats combined from all shard runs.

I was thinkings about adding a separate reporter for that purpose, but if those last run stats are already there…, then there might not be the need to create a separate reporter.

pavelfeldman · 2024-05-24T17:02:11Z

I was thinkings about adding a separate reporter for that purpose, but if those last run stats are already there…, then there might not be the need to create a separate reporter.

Shaping this code as reporter sounds good, but Playwright core would need to consume the output of that reporter, so it needs to be baked in. Merging those should not be a hard problem, reporter or not. Unfortunately merging mangles test ids today, so we'd need to figure that out. Maybe not using the ids altogether and falling back to the file names and test titles. Also has some tricky edge cases as tests that are fast on Chromium and are slow on Firefox...

muhqu · 2024-05-27T09:31:15Z

I've added a lastrun reporter that can be used with merge-reports to generate .last-run.json.

Surprisingly when merging the reports the test ids just had a 1 character suffix that I was able to strip off… but it doesn't feel like the right way to do this.

What's the reason to modify test ids when merging blobs? Couldn't this be done in a way that only modifies a test id when there is a collision?

muhqu · 2024-05-28T11:10:17Z

packages/playwright/src/runner/runner.ts

+  const testDurations = testRun.rootSuite?.allTests().reduce((map, t) => {
+    if (t.results.length)
+      map[t.id] = t.results.reduce((a, b) => a + b.duration, 0);
+    return map;
+  }, {} as { [testId: string]: number });


I'm actually not sure it is the right way to sum all the durations… maybe it makes more sense to calc the average? Or only include durations from successful test runs… 🤔

I think this is ok for now, we'll iterate based on feedback.

muhqu · 2024-05-28T16:32:23Z

@pavelfeldman last-run.json is a little bit tricky to work with atm… it constantly gets overwritten even if you just list tests and VS Code seems to do that from time to time to refresh the tests panel... I think it should only get written when it's actually running tests.

…icrosoft#30817)" This reverts commit 825e0e4. API review notes: sounds like this change did not solve the problem for the contributor, there is a new approach under development in microsoft#30962

lukeapage · 2024-06-22T10:39:26Z

This is awesome - we were planning on implementing something similar outside playwright before we migrate cypress tests that currently shard across 50 build servers.

Is there anything todo here I could help with to get this merged quicker ?

muhqu · 2024-06-22T12:26:44Z

Hey, I did not noticed this PR got conflicts due to the revert of my other change… I’ll resolve those conflicts in the next days. There was still on open question on whether we should introduce a command line flag to specify a last run state file or not… but I think that are improvements that can also happen after this has been merged

Though, this PR could have some more documentation changes… anyone want to help with that?

dgozman

There was still on open question on whether we should introduce a command line flag to specify a last run state file or not

We discussed this and decided that it should be fine to add --last-run-file= command line flag that will affect both the reading and writing location of the last-run.json.

Though, this PR could have some more documentation changes… anyone want to help with that?

Of course! We can help with adding more documentation after landing this PR.

dgozman · 2024-06-24T22:35:08Z

packages/playwright/src/runner/runner.ts

@@ -152,6 +152,7 @@ export class Runner {
 export type LastRunInfo = {
  status: FullResult['status'];
  failedTests: string[];
+  testDurations?: { [testId: string]: number };


We are a bit worried that this will force testId to never change to preserve backwards compatibility. Therefore, we would like to use testCase.titlePath() instead of id for now. Something like this:

export type LastRunInfo = { status: FullResult['status'], projects: { name: string, files: { path: string, tests: { title: string[], duration: number, status: TestStatus, }[], }[], }[], };

Here file.path is toPosixPath(path.relative(config.rootDir, filePath)), similar to this line.

Yes, more verbose is also better for future use-cases that will consume the last-run info. Having the titles and paths in there is more user friendly… 👍

However, removing the already existing failedTests: string[] from LastRunInfo would be a backward incompatible change… maybe we should keep it for now?

However, removing the already existing failedTests: string[] from LastRunInfo would be a backward incompatible change… maybe we should keep it for now?

There are no backwards compatibility concerns as of today, since we do not expect last-run.json to be used between different versions. It will be the case once we support "commit last-run.json to the repo for shard optimization" scenario.

That said, you can leave it as is, and we'll iterate on it.

I just tried to write the LastRunInfo in the proposed format, but its not as straight forward as I thought it would be.

That said, you can leave it as is, and we'll iterate on it.

You mean, I don't need to proceed here to get this merged? 🙏

[…] we do not expect last-run.json to be used between different versions

In our CI setup, I'm using pipeline steps to restore the last-run.json from the last successful main build. So, in our case last-run info from previous playwright versions will be used, and it would be great if there is some level of backward compatibility... at least it should not fail when the format is incorrect.

dgozman · 2024-06-24T22:36:50Z

packages/playwright/src/runner/runner.ts

+  const testDurations = testRun.rootSuite?.allTests().reduce((map, t) => {
+    if (t.results.length)
+      map[t.id] = t.results.reduce((a, b) => a + b.duration, 0);
+    return map;
+  }, {} as { [testId: string]: number });


I think this is ok for now, we'll iterate based on feedback.

muhqu · 2024-06-25T17:55:46Z

@dgozman I've just added --last-run-file= cli flag and CI seems to be happy… any further work needed on this PR to get merged?

github-actions · 2024-07-02T09:02:37Z

Test results for "tests 1"

6 passed
✔️✔️✔️

Merge workflow run.

github-actions · 2024-07-02T09:19:21Z

Test results for "tests 1"

5 flaky

⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-page] › page/page-event-request.spec.ts:138:3 › should report navigation requests and responses handled by service worker with routing

19538 passed, 329 skipped
✔️✔️✔️

Merge workflow run.

github-actions · 2024-07-02T09:42:32Z

Test results for "tests 1"

1 failed
❌ [installation tests] › playwright-electron-should-work.spec.ts:31:5 › electron should work with special characters in path

6 flaky

⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [playwright-test] › ui-mode-test-screencast.spec.ts:21:5 › should show screenshots
⚠️ [webkit-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture

25063 passed, 489 skipped
✔️✔️✔️

Merge workflow run.

github-actions · 2024-07-02T10:06:55Z

Test results for "tests 1"

7 flaky

⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [firefox-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [webkit-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [playwright-test] › ui-mode-test-screencast.spec.ts:21:5 › should show screenshots

28509 passed, 655 skipped
✔️✔️✔️

Merge workflow run.

…ghting is not an option

github-actions · 2024-07-02T10:33:33Z

Test results for "tests 1"

1 failed
❌ [playwright-test] › reporter-html.spec.ts:390:5 › created › should use different path if attachments base url option is provided

5 flaky

⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [webkit-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture

24434 passed, 473 skipped
✔️✔️✔️

Merge workflow run.

github-actions · 2024-07-02T10:56:55Z

Test results for "tests 1"

7 flaky

⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [firefox-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [chromium-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture
⚠️ [playwright-test] › ui-mode-test-screencast.spec.ts:21:5 › should show screenshots
⚠️ [webkit-library] › library/browsercontext-fetch-happy-eyeballs.spec.ts:49:3 › get should work on request fixture

28509 passed, 655 skipped
✔️✔️✔️

Merge workflow run.

muhqu · 2024-07-02T11:23:27Z

Just added some more documentation, mostly copying what’s in this PR description to the test-sharding page. And cleaned up imports which were previously automatically “organized” due to my vs code settings, which resulted in unnecessary changes with high conflict potential…

It would be great if we could get this over the finish line for the next playwright release. People already asked me how they could use playwright of my PR in their CI… 😅

@dgozman @pavelfeldman can I get a review?

dgozman

@muhqu Sorry for the slow review, I was unavailable for a week.

I think we can land this PR if you remove the reporter for now, and remove the round-robin strategy due to stability concerns. This will give us the necessary functionality, and we can iterate further.

I can follow up with updating the duration-round-robin to fallback to partition.

dgozman · 2024-07-11T14:14:29Z

docs/src/test-api/class-testconfig.md

+Defines the algorithm to be used for sharding. Defaults to `'partition'`.
+* `'partition'` - divide the set of test groups by number of shards. e.g. first
+  half goes to shard 1/2 and seconds half to shard 2/2.
+* `'round-robin'` - spread test groups to shards in a round-robin way. e.g. loop


I feel like this is not a really useful strategy, because it is not stable. Whenever I add a test or move it between files, all my shards will be completely new. This introduces a lot of variability on CI, which from our experience is something to avoid. I wonder whether we can make this stable somehow?

In my current project round-robin yields much better distribution than partition. Here is why, our tests are organised something like this (simplified)…

tests/framework/toolbar.test.ts tests/framework/stage.test.ts tests/logged-in/login-free.test.ts tests/logged-in/login-premium.test.ts tests/logged-in/premium-upsell.test.ts tests/logged-out/view-only.test.ts tests/logged-out/seo.test.ts

Our framework tests are pretty exhaustive and take the most time. The logged-in tests require more time than logged-out tests due to additional bootstrap and teardown. By using the partition algorithm the first shards require significantly more time as they have all the long-tests from either framework or logged-in suites. With round-robin we're already achieving a much better balanced overall runtime, even though the algorithm doesn't know anything about per test duration.

I don't feel like the round-robin algorithm is "not stable". It's deterministic, given the same set of tests it always yields the same result. Also I think in CI there should be no assumption that one test goes after another, unless they are explicitly configured to execute sequentially within the same test group.

I guess that's fair. If you would like to opt-in into round-robin and it works for you, that's great. I still think it is less stable than I would prefer, but it doesn't hurt to have it as an option anyway.

dgozman · 2024-07-11T14:15:29Z

docs/src/test-api/class-testconfig.md

+  test groups to shards in a round-robin way. e.g. loop over test groups and
+  always assign to the shard that has the lowest duration of tests. new tests
+  which were not present in the last run will use an average duration time. When
+  no `.last-run.json` could be found the behavior is identical to


Given the comment above about the round-robin strategy, I think we should use partition when no .last-run.json is present, and also use partition for any tests that were not found in the .last-run.json.

I disagree.

Also please note that duration-round-robin and 'round-robin' are using the same implementation. The only difference is that in case of 'duration-round-robin' the last run info is passed to the implementation. However, in case the last run info is passed but it is empty, the behaviour is identical to 'round-robin' where no last run info is passed.

I don't think it would be a good to choose a different algorithm based on whether last run info is available or not.

dgozman · 2024-07-11T14:20:04Z

packages/playwright/src/common/config.ts

@@ -271,7 +277,7 @@ export function toReporters(reporters: BuiltInReporter | ReporterDescription[] |
  return reporters;
 }

-export const builtInReporters = ['list', 'line', 'dot', 'json', 'junit', 'null', 'github', 'html', 'blob', 'markdown'] as const;
+export const builtInReporters = ['list', 'line', 'dot', 'json', 'junit', 'null', 'github', 'html', 'blob', 'markdown', 'lastrun'] as const;


I don't think we need lastrun here, because it should not be user-visible.

dgozman · 2024-07-11T14:20:59Z

packages/playwright/src/program.ts

@@ -347,6 +353,7 @@ const testOptions: [string, string][] = [
  ['--headed', `Run tests in headed browsers (default: headless)`],
  ['--ignore-snapshots', `Ignore screenshot and snapshot expectations`],
  ['--last-failed', `Only re-run the failures`],
+  ['--last-run-file', `Path to a json file where the last run information is read from and written to (default: .last-run.json)`],


Suggested change

['--last-run-file', `Path to a json file where the last run information is read from and written to (default: .last-run.json)`],

['--last-run-file', `Path to a json file where the last run information is read from and written to (default: test-results/.last-run.json)`],

dgozman · 2024-07-11T14:21:22Z

packages/playwright/src/program.ts

@@ -281,6 +285,8 @@ function overridesFromOptions(options: { [key: string]: any }): ConfigCLIOverrid
    retries: options.retries ? parseInt(options.retries, 10) : undefined,
    reporter: resolveReporterOption(options.reporter),
    shard: shardPair ? { current: shardPair[0], total: shardPair[1] } : undefined,
+    shardingMode: options.shardingMode ? options.shardingMode : undefined,


We should validate that shardingMode has one of the supported values.

dgozman · 2024-07-11T14:24:53Z

packages/playwright/src/runner/runner.ts

  const failedTests = testRun.rootSuite?.allTests().filter(t => !t.ok()).map(t => t.id);
-  const lastRunReport = JSON.stringify({ status, failedTests }, undefined, 2);
+  const testDurations = testRun.rootSuite?.allTests().reduce((map, t) => {


Suggested change

const testDurations = testRun.rootSuite?.allTests().reduce((map, t) => {

const testDurations = (testRun.rootSuite?.allTests() || []).reduce((map, t) => {

dgozman · 2024-07-11T14:27:20Z

packages/playwright/src/runner/runner.ts

+      map[t.id] = t.results.reduce((a, b) => a + b.duration, 0);
+    return map;
+  }, {} as { [testId: string]: number });
+  const lastRunReport = JSON.stringify({ status, failedTests, testDurations }, undefined, 2);


Since we are still writing the file here, why do we need a separate lastrun reporter? Let's remove the reporter for now, to make the scope of worker more narrow. We can introduce the reporter later if needed.

The lastrun reporter is required to generate last run info when merging blob reports from shards.

Maybe it would be better to implicitly write the lastrun info when merging blob reports?

I think so. From the user's perspective, lastrun just working out of the box would be easier. Whether it is a reporter internally or not is an implementation detail.

How about always using the last run reporter? Then we would only need to have a single implementation….

That definitely works. We'll have to add it here when isTestServer === false.

dgozman · 2024-07-11T14:30:23Z

packages/playwright/src/runner/testGroups.ts

+    return group.tests.reduce((sum, test) => sum + Math.max(1, lastRunInfo.testDurations?.[test.id] || averageDuration), 0);
+  };
+
+  // We sort the test groups by the number of tests in descending order.


Suggested change

// We sort the test groups by the number of tests in descending order.

// We sort the test groups by group duration in descending order.

dgozman · 2024-07-11T14:41:38Z

tests/playwright-test/shard-roundrobin.spec.ts

+  ]);
+});
+
+test('should not produce skipped tests for zero-sized shards', async ({ runInlineTest }) => {


I think you can safely remove tests starting from this one, because it is already covered in shard.spec.ts, and sharding strategy does not affect any of them.

muhqu mentioned this pull request May 22, 2024

feat(test runner): shuffle order of tests with sharding seed #30817

Merged

This comment has been minimized.

Sign in to view

muhqu added 8 commits May 27, 2024 07:05

sharding algorithm to better spread similar tests among shards

54bbba8

improve shard algorithm by sorting test groups by number of tests

ee64b15

adjust shard.spec.ts to new algorithm

6543baf

fix reporter-json.spec.ts due to sharding

59c625e

fix reporter-junit.spec.ts due to sharding

d3b4fad

empty commit to trigger ci

4bcfc78

add test durations to last run info

061f559

allow .last-run.json to be generated via merge-reports

14ca30a

muhqu force-pushed the sharding-algorithm branch from 82809eb to 14ca30a Compare May 27, 2024 08:50

This comment has been minimized.

Sign in to view

muhqu added 2 commits May 27, 2024 15:47

Add shardingMode configuration

ce09f88

revert json/junit test changes

d42c499

This comment has been minimized.

Sign in to view

empty commit to trigger ci

b5b8174

This comment has been minimized.

Sign in to view

muhqu commented May 28, 2024

View reviewed changes

pavelfeldman mentioned this pull request Jun 11, 2024

Revert "feat(test runner): shuffle order of tests with sharding seed … #31260

Merged

This comment has been minimized.

Sign in to view

muhqu added 2 commits June 24, 2024 15:08

Merge branch 'main' into sharding-algorithm

843d629

fix test.d.ts

2536da7

muhqu force-pushed the sharding-algorithm branch from 904a8dd to 2536da7 Compare June 24, 2024 13:11

This comment has been minimized.

Sign in to view

dgozman reviewed Jun 24, 2024

View reviewed changes

add --last-run-file CLI parameter

087a57f

This comment has been minimized.

Sign in to view

muhqu requested a review from dgozman June 28, 2024 09:04

muhqu added 4 commits July 2, 2024 09:37

Adjust since to v1.46

7aa2d95

Merge branch 'main' into sharding-algorithm

29f67f3

Add documentation on sharding modes

7b5f7c6

linting

65a2b55

Add into text to more complex round-robin scenario

df6d05d

no ts highlight for sharding illustration code blocks

3fed470

muhqu added 2 commits July 2, 2024 12:09

use yaml highlight for sharding illustration code blocks as NO highli…

050e837

…ghting is not an option

revert / reorder import changes

f483ac6

dgozman reviewed Jul 11, 2024

View reviewed changes

	['--last-run-file', `Path to a json file where the last run information is read from and written to (default: .last-run.json)`],
	['--last-run-file', `Path to a json file where the last run information is read from and written to (default: test-results/.last-run.json)`],

	const testDurations = testRun.rootSuite?.allTests().reduce((map, t) => {
	const testDurations = (testRun.rootSuite?.allTests() \|\| []).reduce((map, t) => {

	// We sort the test groups by the number of tests in descending order.
	// We sort the test groups by group duration in descending order.

feat(test runner): improve sharding algorithm to better spread similar tests among shards #30962

Are you sure you want to change the base?

feat(test runner): improve sharding algorithm to better spread similar tests among shards #30962

Conversation

muhqu commented May 22, 2024 • edited Loading

Problem

Solution

shardingMode: 'partition'

shardingMode: 'round-robin'

shardingMode: 'duration-round-robin'

Other changes

Appendix

muhqu commented May 22, 2024 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pavelfeldman commented May 22, 2024

muhqu commented May 23, 2024 • edited Loading

pavelfeldman commented May 24, 2024

muhqu commented May 24, 2024

pavelfeldman commented May 24, 2024

This comment has been minimized.

muhqu commented May 27, 2024

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muhqu commented May 28, 2024

lukeapage commented Jun 22, 2024

muhqu commented Jun 22, 2024

This comment has been minimized.

This comment has been minimized.

dgozman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

muhqu commented Jun 25, 2024

github-actions bot commented Jul 2, 2024

Test results for "tests 1"

github-actions bot commented Jul 2, 2024

Test results for "tests 1"

github-actions bot commented Jul 2, 2024

Test results for "tests 1"

github-actions bot commented Jul 2, 2024

Test results for "tests 1"

github-actions bot commented Jul 2, 2024

Test results for "tests 1"

github-actions bot commented Jul 2, 2024

Test results for "tests 1"

muhqu commented Jul 2, 2024

dgozman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muhqu commented May 22, 2024 •

edited

Loading

`shardingMode: 'partition'`

`shardingMode: 'round-robin'`

`shardingMode: 'duration-round-robin'`

muhqu commented May 22, 2024 •

edited

Loading

muhqu commented May 23, 2024 •

edited

Loading