Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api): add new test runner api #2249

Merged
merged 15 commits into from
Jun 16, 2020

Conversation

nicojs
Copy link
Member

@nicojs nicojs commented Jun 9, 2020

This adds a new test runner API to Stryker. Differences with the old API:

  1. Explicit difference between a dryRun (initial test run) and a mutantRun
    1. A mutantRun result has an activeMutant field indicating which mutant should be activated by the runner
    2. The result types are also explicit. A MutantRunResult now reports back the status as Killed or Survived instead of letting Stryker make that decision.
  2. Code coverage interfaces are replaced with the simpler MutantCoverage interface.
  3. The MutantRunOptions now has a testFilter, which replaces the old TestFramework API.

I also took the liberty of adding some improvements.

  • Add an id field to a TestResult and using that id for the filtering instead of the cumbersome TestSelection interface we introduced to handle string identifiers (names)
  • TestResult, DryRunResult and MutantRunResult to be implemented using discrimitated unions. This removes a lot of the optional fields. For example, errorMessage is now a required field on an ErrorDryRunResult.
  • Defined errorMessage and failureMessage to be a string, instead of a string[]. If runners want to report more messages, they will have to concatenate themselves.
  • A MutantRunResult no longer has to report back all tests that have run. Instead, it only reports the id of the test that killed the mutant (if the status is indeed killed).
  • A MutantRunResult no longer has to report back all tests that have run. Instead, it only reports the id of the test that killed the mutant (if the status is indeed killed).
  • Added a toMutantRunResult helper function that takes in a dryRunResult and can convert it to a mutantRunResult.

I've added it as TestRunner2, as well as a separate plugin type. This way we can migrate test runner's one by one without breaking the build. Later we can then remove the old test runner API and rename this new one to TestRunner proper.

This adds a new test runner api to Stryker. Differences with the old API:

1. Explicit difference between a `dryRun` (initial test run) and a `mutantRun`
2. Code coverage interfaces are replaced with the simpler `MutantCoverage` interface.
3. The `MutantRunOptions` now has a `testFilter`, which replaces the old `TestFramework` api.
@nicojs nicojs requested a review from simondel June 9, 2020 05:49
@nicojs nicojs merged commit bbbc308 into epic/mutation-switching Jun 16, 2020
@nicojs nicojs deleted the feature/new-test-runner-api branch June 16, 2020 10:29
nicojs added a commit that referenced this pull request Jun 17, 2020
Implement new test runner api in the jasmine-runner.

See #2249
nicojs added a commit that referenced this pull request Jun 19, 2020
* Implement new v2 test runner API (see #2249 )
* Make sure to not report coverage when it isn't needed
* Add a framework adapter file that adds mutant coverage to the karma result, to be picked up by the stryker reporter
* Add test filtering capability when running with jasmine or mocha
* Add integration tests on instrumented code.
* Some more housekeeping.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant