Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable parallel execution of tests #637

Merged
merged 1 commit into from
Mar 8, 2024

Conversation

carbolymer
Copy link
Contributor

@carbolymer carbolymer commented Mar 7, 2024

Changelog

- description: |
    Enable parallel execution of tests
# uncomment types applicable to the change:
  type:
  # - feature        # introduces a new feature
  # - breaking       # the API has changed in a breaking way
  # - compatible     # the API has changed but is non-breaking
  # - optimisation   # measurable performance improvements
  # - improvement    # QoL changes e.g. refactoring
  # - bugfix         # fixes a defect
   - test           # fixes/modifies tests
  # - maintenance    # not directly related to the code
  # - release        # related to a new release preparation
  # - documentation  # change in code docs, haddocks...

Context

See an explanation in similar PR in node: IntersectMBO/cardano-node#5692

Before change:

All 204 tests passed (16.75s)
Test suite cardano-cli-golden: PASS
...
All 22 tests passed (3.94s)
Test suite cardano-cli-test: PASS

After change

All 204 tests passed (12.38s)
Test suite cardano-cli-golden: PASS
...
All 22 tests passed (1.97s)
Test suite cardano-cli-test: PASS

Checklist

  • Commit sequence broadly makes sense and commits have useful messages
  • New tests are added if needed and existing tests are updated. See Running tests for more details
  • Self-reviewed the diff

@carbolymer carbolymer marked this pull request as ready for review March 7, 2024 14:58
@carbolymer carbolymer requested review from smelc and palas March 7, 2024 15:00
@carbolymer carbolymer added this pull request to the merge queue Mar 7, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 7, 2024
@carbolymer carbolymer force-pushed the mgalazyn/chore/enable-parallel-tests branch from b43d72b to 26770f4 Compare March 7, 2024 17:41
@carbolymer carbolymer requested a review from smelc March 7, 2024 17:41
@carbolymer carbolymer force-pushed the mgalazyn/chore/enable-parallel-tests branch from 503ab1c to 104550b Compare March 8, 2024 10:04
Copy link
Contributor

@smelc smelc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New semaphore mechanism LGTM 🍾

for golden files overwritten in multiple tests.
@carbolymer carbolymer force-pushed the mgalazyn/chore/enable-parallel-tests branch from 104550b to d0440d4 Compare March 8, 2024 11:04
@carbolymer carbolymer added this pull request to the merge queue Mar 8, 2024
Merged via the queue into main with commit d2449e0 Mar 8, 2024
17 checks passed
@carbolymer carbolymer deleted the mgalazyn/chore/enable-parallel-tests branch March 8, 2024 11:23
@@ -103,16 +111,17 @@ golden_create_testnet_data mShelleyTemplate =
generated'' = map (\c -> if c == '\\' then '/' else c) generated'
void $ H.note generated''

H.diffVsGoldenFile generated'' "test/cardano-cli-golden/files/golden/conway/create-testnet-data.out"
bracketSem createTestnetDataOutSem $
H.diffVsGoldenFile generated'' "test/cardano-cli-golden/files/golden/conway/create-testnet-data.out"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this test is run once, how does concurrent execution cause it to run multiple times?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's executed once. The problem is that this golden file is shared between multiple test cases, which are executed concurrently. So when we're testing golden files regeneration, two test cases see the file missing so they try to recreate file at the same time. This somehow is an issue on windows, which results in an exception here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants