Do a built-in exec command with the behaviour of

"test-mutated-package.sh"

Fixes #16

README.md

@@ -7,8 +7,7 @@ go-mutesting is a framework for performing mutation testing on Go source code. I
 The following command mutates the go-mutesting project with all available mutators.
 
 ```bash
-cd $GOPATH/src/github.com/zimmski/go-mutesting
-go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/test-mutated-package.sh" --exec-timeout 1 github.com/zimmski/go-mutesting/...
+go-mutesting github.com/zimmski/go-mutesting/...
 ```
 
 The execution of this command prints for every mutation if it was successfully tested or not. If not, the source code patch is printed out, so the mutation can be investigated. The following shows an example for a patch of a mutation.
@@ -71,27 +70,32 @@ go-mutesting includes a binary which is go-getable.
 go get -t -v github.com/zimmski/go-mutesting/...
 ```
 
-The binary's help can be invoked by executing the binary without arguments or with the `--help` option.
+The binary's help can be invoked by executing the binary without arguments or with the `--help` argument.
 
 ```bash
 go-mutesting --help
 ```
 
-> **Note**: This README describes only a few of the available options and arguments. It is therefore advisable to examine the help.
+> **Note**: This README describes only a few of the available arguments. It is therefore advisable to examine the output of the `--help` argument.
 
-The targets of the mutation testing can be defined as arguments to the binary. Every target can be either a Go source file, a directory or a package. Directories and packages can also include the `...` wildcard pattern which will search recursively for Go source files. Test source files with the ending `_test` are excluded, since this would interfere with testing the mutation most of the time.
+The targets of the mutation testing can be defined as arguments to the binary. Every target can be either a Go source file, a directory or a package. Directories and packages can also include the `...` wildcard pattern which will search recursively for Go source files. Test source files with the suffix `_test` are excluded, since this would interfere with the testing process most of the time.
 
-The following example will gather all Go files which are defined by the targets and generate mutations with all available mutators of the binary.
+The following example gathers all Go files which are defined by the targets and generate mutations with all available mutators of the binary.
 
 ```bash
 go-mutesting parse.go example/ github.com/zimmski/go-mutesting/mutator/...
 ```
 
-Since every mutation has to be tested it is necessary to define a [command](#write-mutation-exec-commands) with the `--exec` option. The [scripts](/scripts) directory holds basic exec commands for Go projects. The [test-mutated-package.sh](/scripts/test-mutated-package.sh) script for example implements the replacement of the original file with the mutation, the execution of all tests of the package of the mutated file, and the reporting if the mutation was killed. It can be for example used to test the [github.com/zimmski/go-mutesting/example](/example) package.
+Every mutation has to be tested using an [exec command](#write-mutation-exec-commands). By default the built-in exec command is used, which tests a mutation using the following steps:
+
+- Replace the original file with the mutation.
+- Execute all tests of the package of the mutated file.
+- Report if the mutation was killed.
+
+Alternatively the `--exec` argument can be used to invoke an external exec command. The [/scripts/exec](/scripts/exec) directory holds basic exec commands for Go projects. The [test-mutated-package.sh](/scripts/exec/test-mutated-package.sh) script implements all steps of the built-in exec command. It can be for example used to test the [github.com/zimmski/go-mutesting/example](/example) package.
 
 ```bash
-cd $GOPATH/src/github.com/zimmski/go-mutesting/example
-go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/test-mutated-package.sh" github.com/zimmski/go-mutesting/example
+go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/exec/test-mutated-package.sh" github.com/zimmski/go-mutesting/example
 ```
 
 The execution will print the following output.
@@ -137,7 +141,7 @@ The summary also shows the **mutation score** which is a metric on how many muta
 
 ### <a name="black-list-false-positives"></a>Blacklist false positives
 
-Mutation testing can generate many false positives since mutation algorithms do not fully understand the given source code. `early exits` are one common example. They can be implemented as optimizations and will almost always trigger a false-positive since the unoptimized code path will be used which will lead to the same result. go-mutesting is meant to be used as an addition to automatic test suites. It is therefore necessary to mark such mutations as false-positives. This is done with the `--blacklist` option. The option defines a file which contains in every line a MD5 checksum of a mutation. These checksums can then be used to ignore mutations.
+Mutation testing can generate many false positives since mutation algorithms do not fully understand the given source code. `early exits` are one common example. They can be implemented as optimizations and will almost always trigger a false-positive since the unoptimized code path will be used which will lead to the same result. go-mutesting is meant to be used as an addition to automatic test suites. It is therefore necessary to mark such mutations as false-positives. This is done with the `--blacklist` argument. The argument defines a file which contains in every line a MD5 checksum of a mutation. These checksums can then be used to ignore mutations.
 
 > **Note**: The blacklist feature is currently badly implemented as a change in the original source code will change all checksums.
 
@@ -150,8 +154,7 @@ The example output of the [How do I use go-mutesting?](#how-do-i-use-go-mutestin
 The blacklist file, which is named `example.blacklist` in this example, can then be used to invoke go-mutesting.
 
 ```bash
-cd $GOPATH/src/github.com/zimmski/go-mutesting/example
-go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/test-mutated-package.sh" --blacklist example.blacklist github.com/zimmski/go-mutesting/example
+go-mutesting --blacklist example.blacklist github.com/zimmski/go-mutesting/example
 ```
 
 The execution will print the following output.
@@ -214,7 +217,7 @@ A command must exit with an appropriate exit code.
 | 2         | The mutation was skipped, since there are other problems e.g. compilation errors.                             |
 | >2        | The mutation produced an unknown exit code which might be a flaw in the exec command.                         |
 
-Examples for exec commands can be found in the [scripts](/scripts) directory.
+Examples for exec commands can be found in the [scripts](/scripts/exec) directory.
 
 ## <a name="list-of-mutators"></a>Which mutators are implemented?
 

cmd/go-mutesting/main.go

@@ -67,7 +67,8 @@ type options struct {
 	} `group:"Filter options"`
 
 	Exec struct {
-		Exec    string `long:"exec" description:"Execute this command for every mutation"`
+		Exec    string `long:"exec" description:"Execute this command for every mutation (by default the built-in exec command is used)"`
+		NoExec  bool   `long:"no-exec" description:"Skip the built-in exec command and just generate the mutations"`
 		Timeout uint   `long:"exec-timeout" description:"Sets a timeout for the command execution (in seconds)" default:"10"`
 	} `group:"Exec options"`
 
@@ -137,7 +138,7 @@ func exitError(format string, args ...interface{}) int {
 	return returnError
 }
 
-type Stats struct {
+type mutationStats struct {
 	passed  int
 	failed  int
 	skipped int
@@ -231,7 +232,7 @@ MUTATOR:
 		execs = strings.Split(opts.Exec.Exec, " ")
 	}
 
-	stats := &Stats{}
+	stats := &mutationStats{}
 
 	for _, file := range files {
 		debug(opts, "Mutate %q", file)
@@ -281,17 +282,17 @@ MUTATOR:
 		debug(opts, "Remove %q", tmpDir)
 	}
 
-	if len(execs) > 0 {
+	if !opts.Exec.NoExec {
 		fmt.Printf("The mutation score is %f (%d passed, %d failed, %d skipped, total is %d)\n", float64(stats.passed)/float64(stats.passed+stats.failed), stats.passed, stats.failed, stats.skipped, stats.passed+stats.failed+stats.skipped)
 	} else {
-		fmt.Println("Cannot do a mutation testing summary since no exec command was given.")
+		fmt.Println("Cannot do a mutation testing summary since no exec command was executed.")
 	}
 
 	return returnOk
 }
 
-func mutate(opts *options, mutators []mutator.Mutator, mutationBlackList map[string]struct{}, mutationID int, file string, fset *token.FileSet, src ast.Node, node ast.Node, tmpFile string, execs []string, stats *Stats) int {
-	p, err := build.ImportDir(filepath.Dir(file), build.FindOnly)
+func mutate(opts *options, mutators []mutator.Mutator, mutationBlackList map[string]struct{}, mutationID int, file string, fset *token.FileSet, src ast.Node, node ast.Node, tmpFile string, execs []string, stats *mutationStats) int {
+	pkg, err := build.ImportDir(filepath.Dir(file), build.FindOnly)
 	if err != nil {
 		panic(err)
 	}
@@ -318,43 +319,8 @@ func mutate(opts *options, mutators []mutator.Mutator, mutationBlackList map[str
 			} else {
 				debug(opts, "Save mutation into %q with checksum %s", mutationFile, checksum)
 
-				if len(execs) > 0 {
-					debug(opts, "Execute %q for mutation", opts.Exec.Exec)
-
-					execCommand := exec.Command(execs[0], execs[1:]...)
-
-					execCommand.Stderr = os.Stderr
-					execCommand.Stdout = os.Stdout
-
-					execCommand.Env = append(os.Environ(), []string{
-						"MUTATE_CHANGED=" + mutationFile,
-						fmt.Sprintf("MUTATE_DEBUG=%t", opts.General.Debug),
-						"MUTATE_ORIGINAL=" + file,
-						"MUTATE_PACKAGE=" + p.ImportPath,
-						fmt.Sprintf("MUTATE_TIMEOUT=%d", opts.Exec.Timeout),
-						fmt.Sprintf("MUTATE_VERBOSE=%t", opts.General.Verbose),
-					}...)
-					if opts.Test.Recursive {
-						execCommand.Env = append(execCommand.Env, "TEST_RECURSIVE=true")
-					}
-
-					err = execCommand.Start()
-					if err != nil {
-						panic(err)
-					}
-
-					// TODO timeout here
-
-					err = execCommand.Wait()
-
-					var execExitCode int
-					if err == nil {
-						execExitCode = 0
-					} else if e, ok := err.(*exec.ExitError); ok {
-						execExitCode = e.Sys().(syscall.WaitStatus).ExitStatus()
-					} else {
-						panic(err)
-					}
+				if !opts.Exec.NoExec {
+					execExitCode := mutateExec(opts, mutators, pkg, file, src, mutationFile, execs)
 
 					debug(opts, "Exited with %d", execExitCode)
 
@@ -392,6 +358,121 @@ func mutate(opts *options, mutators []mutator.Mutator, mutationBlackList map[str
 	return mutationID
 }
 
+func mutateExec(opts *options, mutators []mutator.Mutator, pkg *build.Package, file string, src ast.Node, mutationFile string, execs []string) (execExitCode int) {
+	if len(execs) == 0 {
+		debug(opts, "Execute built-in exec command for mutation")
+
+		diff, err := exec.Command("diff", "-u", file, mutationFile).CombinedOutput()
+		if err == nil {
+			execExitCode = 0
+		} else if e, ok := err.(*exec.ExitError); ok {
+			execExitCode = e.Sys().(syscall.WaitStatus).ExitStatus()
+		} else {
+			panic(err)
+		}
+		if execExitCode != 0 && execExitCode != 1 {
+			fmt.Printf("%s\n", diff)
+
+			panic("Could not execute diff on mutation file")
+		}
+
+		defer func() {
+			_ = os.Rename(file+".tmp", file)
+		}()
+
+		err = os.Rename(file, file+".tmp")
+		if err != nil {
+			panic(err)
+		}
+		err = osutil.CopyFile(mutationFile, file)
+		if err != nil {
+			panic(err)
+		}
+
+		pkgName := pkg.Name
+		if opts.Test.Recursive {
+			pkgName += "/..."
+		}
+
+		test, err := exec.Command("go", "test", "-timeout", fmt.Sprintf("%ds", opts.Exec.Timeout), pkgName).CombinedOutput()
+		if err == nil {
+			execExitCode = 0
+		} else if e, ok := err.(*exec.ExitError); ok {
+			execExitCode = e.Sys().(syscall.WaitStatus).ExitStatus()
+		} else {
+			panic(err)
+		}
+
+		if opts.General.Debug {
+			fmt.Printf("%s\n", test)
+		}
+
+		switch execExitCode {
+		case 0: // Tests passed -> FAIL
+			fmt.Printf("%s\n", diff)
+
+			execExitCode = 1
+		case 1: // Tests failed -> PASS
+			if opts.General.Debug {
+				fmt.Printf("%s\n", diff)
+			}
+
+			execExitCode = 0
+		case 2: // Did not compile -> SKIP
+			if opts.General.Verbose {
+				fmt.Println("Mutation did not compile")
+			}
+
+			if opts.General.Debug {
+				fmt.Printf("%s\n", diff)
+			}
+		default: // Unknown exit code -> SKIP
+			fmt.Println("Unknown exit code")
+			fmt.Printf("%s\n", diff)
+		}
+
+		return execExitCode
+	}
+
+	debug(opts, "Execute %q for mutation", opts.Exec.Exec)
+
+	execCommand := exec.Command(execs[0], execs[1:]...)
+
+	execCommand.Stderr = os.Stderr
+	execCommand.Stdout = os.Stdout
+
+	execCommand.Env = append(os.Environ(), []string{
+		"MUTATE_CHANGED=" + mutationFile,
+		fmt.Sprintf("MUTATE_DEBUG=%t", opts.General.Debug),
+		"MUTATE_ORIGINAL=" + file,
+		"MUTATE_PACKAGE=" + pkg.ImportPath,
+		fmt.Sprintf("MUTATE_TIMEOUT=%d", opts.Exec.Timeout),
+		fmt.Sprintf("MUTATE_VERBOSE=%t", opts.General.Verbose),
+	}...)
+	if opts.Test.Recursive {
+		execCommand.Env = append(execCommand.Env, "TEST_RECURSIVE=true")
+	}
+
+	err := execCommand.Start()
+	if err != nil {
+		panic(err)
+	}
+
+	// TODO timeout here
+
+	err = execCommand.Wait()
+
+	if err == nil {
+		execExitCode = 0
+	} else if e, ok := err.(*exec.ExitError); ok {
+		execExitCode = e.Sys().(syscall.WaitStatus).ExitStatus()
+	} else {
+		panic(err)
+	}
+
+	return execExitCode
+}
+
 func main() {
 	os.Exit(mainCmd(os.Args[1:]))
 }

cmd/go-mutesting/main_test.go

@@ -13,7 +13,7 @@ func TestMain(t *testing.T) {
 	testMain(
 		t,
 		"../../example",
-		[]string{"--exec", "../scripts/test-mutated-package.sh", "--exec-timeout", "1", "./..."},
+		[]string{"--debug", "--exec-timeout", "1", "./..."},
 		returnOk,
 		"The mutation score is 0.538462 (7 passed, 6 failed, 1 skipped, total is 14)",
 	)
@@ -23,7 +23,7 @@ func TestMainMatch(t *testing.T) {
 	testMain(
 		t,
 		"../../example",
-		[]string{"--exec", "../scripts/test-mutated-package.sh", "--exec-timeout", "1", "--match", "baz", "./..."},
+		[]string{"--debug", "--exec", "../scripts/exec/test-mutated-package.sh", "--exec-timeout", "1", "--match", "baz", "./..."},
 		returnOk,
 		"The mutation score is 0.000000 (0 passed, 1 failed, 0 skipped, total is 1)",
 	)

example/example.go

@@ -49,7 +49,6 @@ func bar() int {
 
 func baz() int {
 	i := 1
-
 	i = i + i
 
 	return i