-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(test runner): shuffle order of tests with sharding seed #30817
Changes from all commits
d4e236e
06d6062
aa9eb3a
d2beee3
25b9c0d
b56a351
86422bb
23d755d
0be8d54
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
/** | ||
* Copyright (c) Microsoft Corporation. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
/** | ||
* Shuffles the given array of items using the given seed. | ||
* | ||
* @param items The array of items to shuffle. | ||
* @param seed The seed to use for shuffling. | ||
*/ | ||
export function shuffleWithSeed(items: any[], seed: string): void { | ||
const random = rng(cyrb32(seed)); | ||
for (let i = items.length - 1; i > 0; i--) { | ||
const j = Math.floor(random() * (i + 1)); | ||
[items[i], items[j]] = [items[j], items[i]]; | ||
} | ||
} | ||
|
||
/** | ||
* Returns a random number generator seeded with the given seed. | ||
* | ||
* @param seed The seed for the random number generator. | ||
* @returns The random number generator. | ||
*/ | ||
function rng(seed: number) { | ||
const m = 2 ** 35 - 31; | ||
const a = 185852; | ||
let s = seed % m; | ||
return function() { | ||
return (s = s * a % m) / m; | ||
}; | ||
} | ||
|
||
/** | ||
* Return a 32-bit hash from a string. | ||
* | ||
* @param str The string to hash. | ||
* @returns The 32-bit hash. | ||
*/ | ||
function cyrb32(str: string) { | ||
let h = 0x2323; | ||
for (let i = 0; i < str.length; i++) { | ||
h = h ^ str.charCodeAt(i); | ||
h = Math.imul(h, 2654435761); | ||
} | ||
return h >>> 0; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1407,6 +1407,29 @@ interface TestConfig<TestArgs = {}, WorkerArgs = {}> { | |
total: number; | ||
}; | ||
|
||
/** | ||
* Shuffle the order of test groups with a seed. By default tests are run in the order they are discovered, which is | ||
* mostly alphabetical. This could lead to an uneven distribution of slow and fast tests. Shuffling the order of tests | ||
* in a deterministic way can help to distribute the load more evenly. | ||
* | ||
* The sharding seed is a string that is used to initialize a random number generator. | ||
* | ||
* Learn more about [parallelism and sharding](https://playwright.dev/docs/test-parallel) with Playwright Test. | ||
* | ||
* **Usage** | ||
* | ||
* ```js | ||
* // playwright.config.ts | ||
* import { defineConfig } from '@playwright/test'; | ||
* | ||
* export default defineConfig({ | ||
* shardingSeed: 'string value' | ||
* }); | ||
* ``` | ||
* | ||
*/ | ||
shardingSeed?: string; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These will disappear. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like it did not. Is that a problem? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We have user-defined config TestConfig and resolved FullConfig. Should be in TestConfig and not in FullConfig. So you seem to be fine. |
||
|
||
/** | ||
* **NOTE** Use | ||
* [testConfig.snapshotPathTemplate](https://playwright.dev/docs/api/class-testconfig#test-config-snapshot-path-template) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
revert: no need to pass it to the workers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shardingSeed
is required onConfigCLIOverrides
to support--sharding-seed
cli option... It would be limiting to only support this via config file. Or am I missing something?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
…OR if not via cli option, it would be great to support it via environment variable e.g.
export PW_SHARDING_SEED=...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, you are right, we need it here since we are setting it in the command line.