Continuous Integration compatibility

[CheshireSwift]

Currently Stryker always returns exit code 0, which makes it hard to integrate with CI servers and similar tooling.

It would be good to have some way of reporting "failure". Ideally this would be via exit code and "failure" would probably be a configurable threshold of mutation score, though it could be a fixed value like "mutation score on covered code was less than 100%".

[nicojs]

Thanks for reporting this issue.

I agree that this is a missing feature at the moment. What would you like the config to look like?

We could combine this feature with a way to configure a high and low threshold. Those thresholds could be used by reporters. The clear-text reporter right now reports a mutation score > 80 in green, > 50 in yellow and everything else in red. It would be nice to have that configurable.

For example:

thresholds: {
  high: 80,
  low: 50,
  break: 40
}

Everything above 80% will be green, above 50% yellow, below 50 red, below 40 would set the exitCode to 1. By default i think it should be break: null so it wouldn't be breaking anyone's build unexpectedly.

Also, i think the break should use the "mutation score based on entire code base" metric instead of "mutation score on covered code". Otherwise improving your general mutation coverage by adding unit tests could break your build, because you improve your code coverage, but while doing that lowering your mutation score based on covered code.

@simondel, @CheshireSwift do you agree with this approach? Do you agree with the naming? Will the name thresholds be clear enough for users?

[CheshireSwift]

All sounds good, though I'd personally like to be able to choose which mutation score breaks the build (I agree the full code base one is probably the sensible default).

[jeznag]

Here's the code I'm using. It creates stryker config based on which files were changed in the last commit (so only files in the src directory that were changed get mutated), runs Stryker and returns a non zero exit code if the mutant kill score was not within a specified threshold. I've used it in my CI system and it works well.

const git = require('nodegit-kit');
const currentBranch = require('git-branch');

getDiff().then(updateStrykerConfigBasedOnDiff).then(runStryker);

/* eslint-disable no-console */
function getDiff() {
  return git
    .open('.')
    .then((repo) => {
      return git
        .log(repo, { sort: 'reverse', branch: currentBranch.sync() })
        .then((history) => {
          var commit1 = history[history.length - 1].commit;
          var commit2 = history[history.length - 2].commit;
          // git diff <from> <to>
          return git.diff(repo, commit1, commit2);
        })
        .catch((e) => {
          console.log(e);
        });
    })
    .catch((e) => {
      console.log(e);
    });
}

function updateStrykerConfigBasedOnDiff(diff) {
  return new Promise((resolve) => {
    const diffedFiles = diff
      .filter((changedFile) => changedFile.path.startsWith('src'))
      .map((changedFile) => `"${changedFile.path}"`)
      .join(',');

    if (!diffedFiles.length) {
      console.log('Nothing was changed in the src folder. Run stryker again against base config just in case.');
      return resolve();
    } else {
      console.log(`The following files were changed in this commit: ${diffedFiles}. Let's mutate them and see if our tests fail :D`);
    }
    const fs = require('fs');
    const strykerConfigFile = './stryker.conf.js';
    const currentConfig = fs.readFileSync(strykerConfigFile, 'utf8');
    const updatedConfig = currentConfig.replace(
      /mutate: \[.*\]/,
      `mutate: [${diffedFiles}]`
    );
    fs.writeFileSync(strykerConfigFile, updatedConfig);
    resolve();
  });
}

function runStryker() {
  const MINIMUM_MUTANT_KILL_SCORE = 80;
  const MAXIMUM_MUTANT_KILL_SCORE = 90;
  const spawn = require('child_process').spawn;
  const strykerProcess = spawn('npm', ['run', 'stryker']);

  strykerProcess.stdout.on('data', (data) => {
    if (data.toString().includes('Mutation score based on all code')) {
      const percentageScore = parseInt(
        data.toString().split(':')[1].replace('%', ''),
        10
      );
      if (percentageScore < MINIMUM_MUTANT_KILL_SCORE) {
        console.log(
          `Mutant kill score is too low: it was only ${percentageScore} but expected it to be at least ${MINIMUM_MUTANT_KILL_SCORE}`
        );
        process.exit(1);
      } else if (percentageScore > MAXIMUM_MUTANT_KILL_SCORE) {
        console.log(
          `Mutant kill score is too high. This may indicate that files were deleted. It was ${percentageScore} but expected it to be less than ${MAXIMUM_MUTANT_KILL_SCORE}`
        );
        process.exit(1);
      }
      process.exit(0);
    } else {
      console.log(data.toString());
    }
  });

  strykerProcess.stderr.on('data', (data) => {
    console.log('ERROR', data.toString());
  });

  strykerProcess.on('exit', (code) => {
    console.log(`Child exited with code ${code}`);
  });
}

[riezebosch]

@jeznag Nice! I had the following scenario in mind:

1. Write surviving mutants to disk
2. Include this file in the repo as baseline (= accepted mutants)
3. Determine differences for subsequent runs between surviving mutants and baseline mutants
4. Return exit code 0 if there are no differences
5. Update baseline file if you want to accept new mutants.

Because I just don't want to rely on percentages and/or thresholds.

[riezebosch]

Update: I created this baseline reporter. There are only two minor problems:

1. Stryker runs are not consistent (Stryker runs are not consistent #291)
2. It does not influence the exit code (yet)

On the plus side, it is neat to be told when you introduced new mutants since the previous test run.