test: add mutation testing setup

[ollelauribostrom]

Description

I had some fun doing mutation testing in the Gatsby code base this weekend and came up with a mutation testing script that I think could benefit this project in general. It could be a useful tool for investigating/improving test cases, making sure they are efficient. This PR adds a mutation testing setup based on Stryker.

TL;DR

• Add mutation testing setup based on Stryker.
• Add mutation testing script to run mutation testing on individual packages/files.
• Update prettierignore/gitignore to ignore some Stryker stuff.
• Mutation reports for most Gatsby packages: ollelauribostrom/gatsby-mutation-score

What is mutation testing?

Mutation testing is basically testing the quality of your tests to see if they are effective in catching bugs. Don’t get me wrong, measuring test coverage is great — it can help you find untested parts of your code. Test coverage does however not tell you much about the actual quality of your tests or give any sort of indication when they are good enough. Even if you reach 100% test coverage, that basically only tells you that all lines were executed during the test run — not that they actually work as expected. Mutation testing helps you to verify that your assertions are good and points out if you have missed anything important

From Stryker: Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

See an example here

Use cases in the Gatsby project

• Run on the current state of the project to find tests that could use some refactoring.
• Run when adding new tests/features to verify that we are actually catching bugs and that no imporant test cases are missing.
• (maybe in the future) run in CI to detect if a PR could use some more test cases.

Mutating Gatsby packages

Mutation testing is a somewhat tedious process (takes a lot of time to run). The script, therefore, takes some useful arguments to make this easier to work with.

Mutate one/several packages

yarn test:mutate <package-name-1> <package-name-2>

# Example
yarn test:mutate gatsby-link

Mutate a single file

yarn test:mutate <path-to-file> --file

# Example
yarn test:mutate packages/gatsby-link/src/index.js --file

Mutate all packages

yarn test:mutate --all

Note 1: Reports are outputted to /mutation
Note 2: The mutation testing process will only run unit tests in the package folder (no e2e/integration).

Gatsby mutation reports

The setup includes a HTML reporter which outputs useful HTML-files showing detailed information about each mutated file.

The reports for most Gatsby packages can be found in this repo:
ollelauribostrom/gatsby-mutation-score.

(Only packages containing unit tests are included in the repo above and unfortunately the gatsby package is not included since I didn't have time to wait for stryker to check all 7500 mutants that were generated) 😀

[m-allanson]

I haven't had a chance to look at this properly yet, but it sounds really interesting! I published the mutation test results from your repo, if anyone wants to check them out: https://gatsby-mutation-score-n6fbqokk2.now.sh/

[wardpeet]

@ollelauribostrom are there ways to tell stryker about false positives? It's not playing well with some react stuff as it doesn't look at PropTypes

[ollelauribostrom]

@wardpeet Thanks for having a look! I don't think there is a way to instruct stryker about false positives right now but we could try passing the following config here to see if that makes a difference,

    jest: {
      config: Object.assign({}, defaultJestConfig, {
        notify: false,
        project: `react`,
      }),
    },

If I understand you correctly you are not getting any error messages, just that Stryker failes to kill some mutants because it is not looking at the prop types?

[m-allanson]

Thank you for your work on this @ollelauribostrom, particularly all the follow up tests that you added!

I think there's some really interesting possibilities from using mutation testing. However I'm going to close this PR for now - as you noted the long runtime makes it unsuitable for usage in CI. Historically we've seen that our testing tools don't get much usage unless we can force them to run automatically.

Despite closing it, I love seeing experimental PRs like this so thank you again 💪💜