try Stryker (v2) testing, for evaluation, etc. - outdated #16
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
November 13, 2019 18:18
yarn add --dev @stryker-mutator/core ref: https://stryker-mutator.io/stryker/quickstart
using npx: $ npx stryker init to initialize testing with Stryker (version 2.3.0) (with jest runner, on JavaScript code) with HTML reporter included and with Yarn package manager selected as documented in: https://stryker-mutator.io/stryker/quickstart NOTE that the generated `stryker.conf.js` has a known eslint error at this point, to be resolved in an upcoming commit (in the very near future). Here is a capture of the `stryker init` work from the command line, for future reference: $ npx stryker init ? Are you using one of these frameworks? Then select a preset configuration. None/other ? Which test runner do you want to use? If your test runner isn't listed here, you can choose "command" (it uses your `npm test` command, but will come with a big performance penalty) jest No stryker test framework plugin found that is compatible with jest, downgrading coverageAnalysis to "all" ? What kind of code do you want to mutate? javascript ? [optional] What kind transformations should be applied to your code? ? Which reporter(s) do you want to use? html, clear-text, progress ? Which package manager do you want to use? yarn Writing & formatting stryker.conf.js... Installing NPM dependencies... yarn add @stryker-mutator/jest-runner @stryker-mutator/javascript-mutator @stryker-mutator/html-reporter --dev yarn add v1.19.1 [1/5] Validating package.json... [2/5] Resolving packages... [3/5] Fetching packages... info fsevents@1.2.9: The platform "linux" is incompatible with this module. info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation. [4/5] Linking dependencies... warning " > angular-estree-parser@1.1.5" has incorrect peer dependency "@angular/compiler@^6.0.0 || ^7.0.0". [5/5] Building fresh packages... success Saved lockfile. success Saved 6 new dependencies. info Direct dependencies ├─ @stryker-mutator/html-reporter@2.3.0 ├─ @stryker-mutator/javascript-mutator@2.3.0 └─ @stryker-mutator/jest-runner@2.3.0 info All dependencies ├─ @stryker-mutator/html-reporter@2.3.0 ├─ @stryker-mutator/javascript-mutator@2.3.0 ├─ @stryker-mutator/jest-runner@2.3.0 ├─ file-url@3.0.0 ├─ lodash.clonedeep@4.5.0 └─ mutation-testing-elements@1.2.0 Done in 5.20s. Done configuring stryker. Please review `stryker.conf.js`, you might need to configure transpilers or your test runner correctly. Let's kill some mutants with this command: `stryker run` Note: Stryker will use `.stryker-temp` as location for temporary files. Be sure to add it to your ignored files in source control.
for now, at least This seems to be needed to avoid excessive surviving mutation test runs, which seems to happen when using `testRunner: "jest"` in stryker.conf.js.
|
Some bad news: after about 3 hours the following shows up at the end of Mutation testing 72% (ETC 1h, 8m, 58s) 857/1186 tested (152 survived, 59 timed out)
Mutation testing 72% (ETC 1h, 9m, 2s) 857/1186 tested (152 survived, 59 timed out)
Mutation testing 72% (ETC 1h, 9m, 6s) 857/1186 tested (152 survived, 59 timed out)
Mutation testing 72% (ETC 1h, 9m, 9s) 857/1186 tested (152 survived, 59 timed out)
Mutation testing 72% (ETC 1h, 9m, 13s) 857/1186 tested (152 survived, 59 timed out)
Mutation testing 72% (ETC 1h, 9m, 17s) 857/1186 tested (152 survived, 59 timed out)
Mutation testing 72% (ETC 1h, 9m, 21s) 857/1186 tested (152 survived, 59 timed out)
Killed
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.From a quick search this looks like something ran out of memory. I wonder if using something like |
added 5 commits
November 13, 2019 23:05
limited down to src/common at this point
by adding the following line to the beginning: "use strict"; (with an extra blank line included)
brodybits
force-pushed
the
try-stryker-testing
branch
from
a7ce7ee
to
6d597ce
Compare
|
Trying now with the following command on a virtual server with 40 (virtual) CPUs and 160 GB memory available: nohup node --max-old-space-size=100000 `which yarn` stryker | tee stryker-src-common-progress-output.txt &(with slightly different progress output file name this time) The P.S. I realized that something like this may be better: nohup node --max-old-space-size=100000 ./node_modules/.bin/stryker run | tee node-stryker-src-common-progress-output.txt &I started a Stryker testing run with this in parallel to the other nohup command above in this comment. I will report with some more explanation once I get some results if either or both of these commands gives me a successful Stryker testing run on |
|
The HTML mutation report generated by is now posted at: https://chrisbrody.com/prettier-src-common-mutation-report-2019-11-13/ Report generated by So using the P.S. I think the one major thing missing test coverage in the language-js processing path is the internal |
|
One more observation I have is that almost half an hour after the report by this command is finished: nohup node --max-old-space-size=100000 `which yarn` stryker | tee stryker-src-common-progress-output.txt &
I think the best workaround solution if I still want to use the virtual machine would be to simply reboot it. Same workaround as if I have to stop an ongoing Stryker test run on 40 (virtual) CPUs for any reason. |
|
The report generated by |
This was referenced Nov 14, 2019
brodybits
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to start using Stryker Mutator for mutation testing of Prettier, as a continuation of prettier#6681.
I am adding Stryker as described in https://stryker-mutator.io/stryker/quickstart but with a few tweaks & updates:
testRunner: "command"instryker.conf.js(and remove@stryker-mutator/jest-runnerfromdevDependencies) - seems to be needed to to avoid excessive surviving mutation test runs, which seems to happen when usingtestRunner: "jest"instryker.conf.jsstryker.conf.jssrc/commonfor now"use strict";line tostryker.conf.jsto make eslint happyAt this point I was able to start Stryker testing on
src/common/*.json a virtual server in the cloud, with 40 (virtual) CPUs. After about 1.5 hours, Stryker claims ETC in just under 2.5 more hours.I started with nohup & tee on the command line for increased resilience against any possible network connection issues:
Here is the command to watch the progress log if needed after disconnecting:
As I said in prettier#6681 (prettier#6681 (comment)), I encountered some issues with running Stryker on combinations such as
src/common/*.js,src/language-js/*.js, andsrc/utils/*.js. I would really love to see some improvements in support for this kind of combination.I was already able to generate a now outdated report on
src/cli(https://chrisbrody.com/prettier-cli-mutation-report/index.html) in prettier#6681. I think it would be much more useful to go through a report on src/common and src/language-js first.So far I found a few things that could probably be improved on Stryker:
testRunner: "jest"setting instryker.conf.js