Add `--parallel`, `--bail` and `--grep` params to the test task #2359

fvictorio · 2022-02-09T13:43:11Z

Add support for the --parallel, --bail and --grep flags in the test task.

The first commit is just the mocha upgrades, the second one adds the --parallel flag. Notice that this flag also adds hardhat/register to Mocha's required files. This makes parallel tests mostly work out-of-the-box, but there might be a risk here that I'm not seeing.

Pending stuff:

Check if hardhat-gas-reporter and solidity-coverage work on parallel mode. Document how to make them work if not, or just document the limitation if making them work is too much effort.
Run tests in parallel mode in the e2e tests
Add --bail and --grep flags
Document this mode, its limitations, and tips.
Add changeset.

changeset-bot · 2022-02-09T13:43:13Z

🦋 Changeset detected

Latest commit: 7a6d522

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
hardhat	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

fvictorio · 2022-02-11T12:21:47Z

solidity-coverage works fine because it has its own coverage task.

hardhat-gas-reporter doesn't work with parallel tests. I created an issue about it, but we should also document that and also be explicit about this in the release notes.

docs/guides/waffle-testing.md

packages/hardhat-core/src/builtin-tasks/test.ts

zoeyTM · 2022-02-15T21:34:36Z

packages/hardhat-core/src/builtin-tasks/test.ts

+      const mochaConfig: MochaOptions = {
+        ...config.mocha,
+        bail,
+        grep,


Should/do we allow these to be configured in config.mocha as well? If so, this would overwrite them (i.e. if they were set in config.mocha but not passed as params to the subtask).

Oh, I hadn't noticed that, thanks for pointing it out. I actually think that would be the correct behavior though? Normally CLI flags override what's in the config (for example, that's what Prettier does by default). But it's good to know that this is the case. I will add a comment explaining that this is intentional.

Wait I see what you mean. For the grep flag this works correctly, but bail has a default value, and so it will always override whatever is in the config. I'll fix this, thank you.

Great catch! I was also tricked by this

alcuadrado · 2022-02-18T13:23:33Z

docs/guides/waffle-testing.md

@@ -125,6 +125,23 @@ expect(await greeter.greet()).to.equal("Hola, mundo!");

 We can modify the state of a contract in the same way we read from it. Calling `setGreeting` will set a new greeting message. After the `Promise` is resolved, we perform another assertion to verify that the greeting change took effect.

+### Running tests in parallel


I'd move this to its own guide, to make it more visible.

Also, this is not a waffle thing.

packages/hardhat-core/src/builtin-tasks/test.ts

alcuadrado · 2022-02-18T14:50:21Z

As discussed offline, this PR looks great, but we should test that the params actually enable those features. @fvictorio suggested using a fixture project for that.

Co-authored-by: F. Eugene Aumson <feuGeneA@users.noreply.github.com>

Co-authored-by: Patricio Palladino <email@patriciopalladino.com>

fvictorio · 2022-02-21T21:50:34Z

@alcuadrado this is ready for review.

fvictorio · 2022-02-22T14:03:53Z

Blocked by this. The obvious solution is to update our stack traces to not have circular references, but there might be other approaches.

fvictorio · 2022-02-22T14:06:28Z

We could do something like this.

SolidityError instances cannot be JSON.stringified because they contain circular references. This causes an issue with parallel tests, because mocha tries to serialize errors thrown by tests. This commit changes the SourceReference interface to prevent these circular references.

alcuadrado

LFG!

fvictorio added 2 commits February 9, 2022 09:15

Upgrade mocha to latest

b97808f

Add --parallel flag

de16fcc

Fix autocomplete test

f498fc7

This was referenced Feb 11, 2022

Throw error when --parallel flag is set cgewecke/hardhat-gas-reporter#99

Merged

feature: support hardhat test --parallel flag sc-forks/solidity-coverage#691

Open

fvictorio added 5 commits February 15, 2022 09:37

Run tests in parallel in e2e tests

ee68191

Add --bail and --grep params to test task

368c8bf

Add docs on running tests in parallel

c972d67

Merge branch 'master' into parallel-tests

983f554

Create hip-planes-invent.md

345ebab

fvictorio marked this pull request as ready for review February 15, 2022 13:27

fvictorio changed the title ~~Add --parallel flag to test task~~ Add --parallel, --bail and --grep params to the test task Feb 15, 2022

Fix autocomplete tests

ac001a4

feuGeneA reviewed Feb 15, 2022

View reviewed changes

docs/guides/waffle-testing.md Outdated Show resolved Hide resolved

docs/guides/waffle-testing.md Outdated Show resolved Hide resolved

docs/guides/waffle-testing.md Outdated Show resolved Hide resolved

zoeyTM reviewed Feb 15, 2022

View reviewed changes

alcuadrado reviewed Feb 18, 2022

View reviewed changes

packages/hardhat-core/src/builtin-tasks/test.ts Outdated Show resolved Hide resolved

alcuadrado reviewed Feb 18, 2022

View reviewed changes

packages/hardhat-core/src/builtin-tasks/test.ts Outdated Show resolved Hide resolved

fvictorio and others added 3 commits February 18, 2022 11:57

Update docs/guides/waffle-testing.md

30c1b7d

Co-authored-by: F. Eugene Aumson <feuGeneA@users.noreply.github.com>

Update docs/guides/waffle-testing.md

057dcdf

Co-authored-by: F. Eugene Aumson <feuGeneA@users.noreply.github.com>

Improve explanation about potential timeouts

fe479e0

fvictorio force-pushed the parallel-tests branch from 49dc2d2 to fe479e0 Compare February 18, 2022 15:01

fvictorio and others added 5 commits February 18, 2022 12:02

Update packages/hardhat-core/src/builtin-tasks/test.ts

b399960

Co-authored-by: Patricio Palladino <email@patriciopalladino.com>

Update packages/hardhat-core/src/builtin-tasks/test.ts

7feb7f3

Co-authored-by: Patricio Palladino <email@patriciopalladino.com>

Fix linter

19df511

Move parallel tests docs to their own guide

bf6417e

Force type imports to be at the beginning

60def3a

fvictorio added 6 commits February 18, 2022 13:32

Merge branch 'master' into parallel-tests

b74ee2e

Add tests for test task

84b070c

Fix issue with false flags overriding mocha config

3917ebe

Fix linter

3047a83

Fix autocomplete test

8ba2070

Change how parallel mode is tested

36411fd

fvictorio force-pushed the parallel-tests branch from a41bb6a to 36411fd Compare February 21, 2022 16:53

Clarify how the HRE is shared in parallel tests doc

2aa70d4

fvictorio added 2 commits February 22, 2022 19:29

Merge branch 'master' into parallel-tests

7a6d522

alcuadrado approved these changes Feb 23, 2022

View reviewed changes

fvictorio merged commit 43dd1d5 into master Feb 23, 2022

fvictorio deleted the parallel-tests branch February 23, 2022 16:10

cameel mentioned this pull request Mar 1, 2022

Expanding the functionality of the test task #1116

Closed

feuGeneA mentioned this pull request Mar 21, 2022

Allow passing --slow param to mocha in hardhat test #2502

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `--parallel`, `--bail` and `--grep` params to the test task #2359

Add `--parallel`, `--bail` and `--grep` params to the test task #2359

fvictorio commented Feb 9, 2022 •

edited

Loading

changeset-bot bot commented Feb 9, 2022 •

edited

Loading

fvictorio commented Feb 11, 2022

zoeyTM Feb 15, 2022

fvictorio Feb 16, 2022

fvictorio Feb 16, 2022

alcuadrado Feb 18, 2022

fvictorio Feb 18, 2022

alcuadrado Feb 18, 2022

fvictorio Feb 18, 2022

alcuadrado commented Feb 18, 2022

fvictorio commented Feb 21, 2022

fvictorio commented Feb 22, 2022

fvictorio commented Feb 22, 2022

alcuadrado left a comment

		@@ -125,6 +125,23 @@ expect(await greeter.greet()).to.equal("Hola, mundo!");

		We can modify the state of a contract in the same way we read from it. Calling `setGreeting` will set a new greeting message. After the `Promise` is resolved, we perform another assertion to verify that the greeting change took effect.

		### Running tests in parallel

Add --parallel, --bail and --grep params to the test task #2359

Add --parallel, --bail and --grep params to the test task #2359

Conversation

fvictorio commented Feb 9, 2022 • edited Loading

changeset-bot bot commented Feb 9, 2022 • edited Loading

🦋 Changeset detected

fvictorio commented Feb 11, 2022

zoeyTM Feb 15, 2022

Choose a reason for hiding this comment

fvictorio Feb 16, 2022

Choose a reason for hiding this comment

fvictorio Feb 16, 2022

Choose a reason for hiding this comment

alcuadrado Feb 18, 2022

Choose a reason for hiding this comment

fvictorio Feb 18, 2022

Choose a reason for hiding this comment

alcuadrado Feb 18, 2022

Choose a reason for hiding this comment

fvictorio Feb 18, 2022

Choose a reason for hiding this comment

alcuadrado commented Feb 18, 2022

fvictorio commented Feb 21, 2022

fvictorio commented Feb 22, 2022

fvictorio commented Feb 22, 2022

alcuadrado left a comment

Choose a reason for hiding this comment

Add `--parallel`, `--bail` and `--grep` params to the test task #2359

Add `--parallel`, `--bail` and `--grep` params to the test task #2359

fvictorio commented Feb 9, 2022 •

edited

Loading

changeset-bot bot commented Feb 9, 2022 •

edited

Loading