Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: add parallel test running and order randomizer #575

Merged
merged 1 commit into from Dec 29, 2020

Conversation

dirkdev98
Copy link
Member

The test runner now uses worker_threads to dispatch test files. Every worker runs the global setup before executing tests and runs the global teardown when no test files are available.

The unit of distribution used are files, so tests in a single file will always run in serial.

When --serial is passed, the standard test runner is used, and the same happens when you execute a test file directly and call mainTestFn.

To find flaky tests you can use --randomize-rounds=n, where n is any integer. This will randomize the files passed to workers. Note that the workers are restarted after every full test run, and thus the global setup and teardown are execute multiple times.

To enable these arguments, the cli parser also got upgraded. We try to match Node.js arguments against their long flags as specified in node --help.

Closes #509

The test runner now uses worker_threads to dispatch test files. Every worker runs the global setup before executing tests and runs the global teardown when no test files are available.

The unit of distribution used are files, so tests in a single file will always run in serial.

When `--serial` is passed, the standard test runner is used, and the same happens when you execute a test file directly and call `mainTestFn`.

To find flaky tests you can use `--randomize-rounds=n`, where n is any integer. This will randomize the files passed to workers. Note that the workers are restarted after every full test run, and thus the global setup and teardown are execute multiple times.

To enable these arguments, the cli parser also got upgraded. We try to match Node.js arguments against their long flags as specified in `node --help`.

Closes #509
@dirkdev98 dirkdev98 enabled auto-merge (squash) December 29, 2020 22:54
@github-actions
Copy link
Contributor

==== Benchmark result ====

Node.js version: 15
Total benchmarks: 13
          Passed: 13
          Failed: 0
-----------
compas validator simple                 5000000  iterations     253  ns/op
yup validator simple                     100000  iterations   12373  ns/op
fastest-validator validator simple     10000000  iterations     117  ns/op
compas validator nested                 1000000  iterations    1455  ns/op
yup validator nested                      50000  iterations   38083  ns/op
fastest-validator validator nested      2000000  iterations     554  ns/op
logger - strings                         500000  iterations    2301  ns/op
logger - objects                         500000  iterations    2798  ns/op
logger - deep objects                    500000  iterations    4746  ns/op
uuid                                    2000000  iterations     571  ns/op
uuid.isValid                           10000000  iterations     159  ns/op
isNil                                 500000000  iterations       0  ns/op
isPlainObject                          50000000  iterations      21  ns/op

@github-actions
Copy link
Contributor

==== Benchmark result ====

Node.js version: 14
Total benchmarks: 13
          Passed: 13
          Failed: 0
-----------
compas validator simple                 5000000  iterations     251  ns/op
yup validator simple                     100000  iterations   11906  ns/op
fastest-validator validator simple     10000000  iterations     119  ns/op
compas validator nested                 1000000  iterations    1362  ns/op
yup validator nested                      50000  iterations   36832  ns/op
fastest-validator validator nested      2000000  iterations     543  ns/op
logger - strings                         500000  iterations    2345  ns/op
logger - objects                         500000  iterations    2826  ns/op
logger - deep objects                    500000  iterations    4961  ns/op
uuid                                    2000000  iterations     576  ns/op
uuid.isValid                           10000000  iterations     168  ns/op
isNil                                 500000000  iterations       0  ns/op
isPlainObject                         100000000  iterations      19  ns/op

@github-actions
Copy link
Contributor

==== C8 test coverage ====
Statements: 68.98% ( 22041/31953 )
Branches : 68.18% ( 2486 /3646 )
Functions : 68.77% ( 751 /1092 )
Lines : 68.98% ( 22041 /31953 )

@dirkdev98 dirkdev98 merged commit b006807 into main Dec 29, 2020
@dirkdev98 dirkdev98 deleted the feat/cli-parallel-test branch December 29, 2020 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cli: support randomizer in test runner
1 participant