feat: Specify build Configuration (#2928)

Purpose
Improve the project discovery and analysis phase and improve the design of associated classes. Fixes relevant issues along the way

Changes
Project discovery and Solution mode (main changes)
Alignement The general processing is now identical for both modes. The difference is in how individual projects are discovered:
in solution mode, Stryker works with every project of the solution
in project discovery mode, Stryker recursively discovers project, starting with the provide test project(s) and add any project dependencies
Note that other strategies are easy to implement, such as full recursive discovery: testing all projects are referenced (incl. transitively) by a set of test projects.
Project Analysis
Configuration: the user can now specify the desired (project/solution configuration (e.g Release) instead of Stryker peeking the default one.
Multi-target: Stryker respects exact inter project dependencies, including target framework version and target platform
Filtering: SourceProject option works for solution mode too
Platform: Stryker uses the target platform settings (if specified) when running tests.
Resiliency: Stryker retries project analysis when Buildalyzer failed to detect dependency. Stryker considers an analysis as failed if it did not report any source files or any dependencies, disregarding the BuildAlyzer status. MsBuild may report a build as failed if some secondary target fails, on the other hand, BuildAlyzer will report a success while it failed to capture any dependencies.
Logging: Stryker provides MsBuild log for both project and solution mode (with -dev—mode modifier) as well as analysis result details.
Content files: Stryker perform a post build scan of dependencies to identify content files from nugget packages such as ‘MimeTypes’. This is a workaround until BuildAlyzer is able to detect them
CLI
-configuration : new option that allows to specify which configuration to build (Debug, Release…). This should be a solution configuration in ‘solution’ mode.
Misc
compilation phase: compilation stops (and fails) immediately if Stryker is unable to identify any (new) mutation causing errors. Previously it recompiled the same code until the max attempts is reached
initial build: log first build attempt when it fails (Trace level), use dotnet msbuild instead of msbuild.exe on non windows platform
initial build: ensure quotes are applied for space containing path in every situation
Main changes in design
Merged ProjectFileReader class with InputFileSolver as responsibilities were blurry between them.
Added several tests for not yet covered cases
Github issues
fix #2930, fix #2748, fix #2693, fix #2587
related: #2886, #2393, #2077, #2938

docs/configuration.md

@@ -138,7 +138,9 @@ Default: `latest`
 Command line: `N/A`  
 Config file: `"language-version": 'CSharp7_3'`
 
-Stryker compiles with the latest stable csharp version by default. This should generally be fine as csharp language features are forward compatible. You should not have to change the option from latest unless you're using preview versions of dotnet/csharp. If you do have compilation errors regarding language features you can explicitly set the language version.
+Stryker compiles using the project language settings (latest by default); but note that since Stryker embeds the Roslyn compiler, `preview` and `latest` are relative to this compiler 
+and may be different from your set up. If you do have compilation errors regarding language features you can explicitly set the language version.
+Note that there is always a delay between C# SDK beta releases and Roslyn and Stryker supporting them, some new features may not be supported immediately.
 
 Valid language versions:
 - Default (Latest)
@@ -161,10 +163,17 @@ Valid language versions:
 
 *\* Csharp version 1 is not allowed because stryker injects helper code that uses csharp 2 language features.*
 
+### `configuration` <`string`>
+Default: default for SDK, usually `Debug`
+Command line: `--configuration Release`  
+Config file: `"configuration": "Release"
+
+Allows you to specify the build configuration to use when building the project. This can be useful when you want to test the release build of your project.
+
 ### `target-framework` <`string`>
 
-Default: randomly selected
-Command line: `target-framework`  
+Default: as defined by the configuration, randomly chosen if multiple frameworks are targeted
+Command line: `--target-framework`  
 Config file: `"target-framework": "netcoreapp3.1"`
 
 If the project targets multiple frameworks, this way you can specify the particular framework to build against. If you specify a non-existent target, Stryker will build the project against a random one (or the only one if so).
@@ -634,8 +643,8 @@ Default: `false`
 Command line: `--dev-mode`  
 Config file: `N/A`
 
-Stryker will not gracefully recover from compilation errors, but instead crash immediately. Used during development to quickly diagnose errors.  
-Also enables more debug logs not generally useful to normal users.
+You should activate `dev mode` when diagnosing an issue with Stryker. This will enable additional logging, specific checks, disable some optimizations...
+As a result, Stryker will be slower, but the log file should help diagnose the hardest issues.
 
 ## Misc
 

src/Stryker.CLI/Stryker.CLI.UnitTest/ConfigBuilderTests.cs

@@ -92,6 +92,7 @@ private static Mock<IStrykerInputs> GetMockInputs()
             inputs.Setup(x => x.VerbosityInput).Returns(new VerbosityInput());
             inputs.Setup(x => x.ConcurrencyInput).Returns(new ConcurrencyInput());
             inputs.Setup(x => x.SolutionInput).Returns(new SolutionInput());
+            inputs.Setup(x => x.ConfigurationInput).Returns(new ConfigurationInput());
             inputs.Setup(x => x.SourceProjectNameInput).Returns(new SourceProjectNameInput());
             inputs.Setup(x => x.TestProjectsInput).Returns(new TestProjectsInput());
             inputs.Setup(x => x.MsBuildPathInput).Returns(new MsBuildPathInput());

src/Stryker.CLI/Stryker.CLI/CommandLineConfig/CommandLineConfigReader.cs

@@ -174,6 +174,7 @@ private void PrepareCliOptions(IStrykerInputs inputs)
             AddCliInput(inputs.DisableBailInput, "disable-bail", null, optionType: CommandOptionType.NoValue);
             // Category: Build
             AddCliInput(inputs.SolutionInput, "solution", "s", argumentHint: "file-path", category: InputCategory.Build);
+            AddCliInput(inputs.ConfigurationInput, "configuration", null, argumentHint: "Release,Debug", category: InputCategory.Build);
             AddCliInput(inputs.SourceProjectNameInput, "project", "p", argumentHint: "project-name.csproj", category: InputCategory.Build);
             AddCliInput(inputs.TestProjectsInput, "test-project", "tp", CommandOptionType.MultipleValue, InputCategory.Build);
             AddCliInput(inputs.MsBuildPathInput, "msbuild-path", null, category: InputCategory.Build);

src/Stryker.CLI/Stryker.CLI/FileBasedInput.cs

@@ -40,6 +40,9 @@ public class FileBasedInput : IExtraData
         [JsonPropertyName("solution")]
         public string Solution { get; init; }
 
+        [JsonPropertyName("configuration")]
+        public string Configuration { get; init; }
+
         [JsonPropertyName("target-framework")]
         public string TargetFramework { get; init; }
 

src/Stryker.CLI/Stryker.CLI/FileConfigReader.cs

@@ -54,6 +54,7 @@ public static void DeserializeConfig(string configFilePath, IStrykerInputs input
             inputs.ReportersInput.SuppliedInput = config.Reporters;
 
             inputs.SolutionInput.SuppliedInput = config.Solution;
+            inputs.ConfigurationInput.SuppliedInput = config.Configuration;
             inputs.TargetFrameworkInput.SuppliedInput = config.TargetFramework;
 
             inputs.SourceProjectNameInput.SuppliedInput = config.Project;

src/Stryker.CLI/Stryker.CLI/FileConfigWriter.cs

@@ -51,6 +51,7 @@ private static FileBasedInputOuter CreateConfig(IStrykerInputs inputs)
                 MutationLevel = inputs.MutationLevelInput.SuppliedInput ?? inputs.MutationLevelInput.Default,
                 Reporters = inputs.ReportersInput.SuppliedInput?.ToArray() ?? inputs.ReportersInput.Default.ToArray(),
                 Solution = inputs.SolutionInput.SuppliedInput ?? inputs.SolutionInput.Default,
+                Configuration = inputs.ConfigurationInput.SuppliedInput ?? inputs.ConfigurationInput.Default,
                 TargetFramework = inputs.TargetFrameworkInput.SuppliedInput ?? inputs.TargetFrameworkInput.Default,
                 Project = inputs.SourceProjectNameInput.SuppliedInput ?? inputs.SourceProjectNameInput.Default,
                 Verbosity = inputs.VerbosityInput.SuppliedInput ?? inputs.VerbosityInput.Default,

src/Stryker.CLI/Stryker.CLI/Logging/ApplicationLogging.cs

@@ -19,16 +19,14 @@ public static void ConfigureLogger(LogEventLevel logLevel, bool logToFile, bool
             if (logToFile)
             {
                 // log on the lowest level to the log file
-                var logFilesPath = Path.Combine(outputPath, "logs");
-                LoggerFactory.AddFile(logFilesPath + "/log-{Date}.txt", traceToFile ? MSLogLevel.Trace : MSLogLevel.Debug);
+                LoggerFactory.AddFile(Path.Combine(outputPath, "logs", "log-{Date}.txt"), traceToFile ? MSLogLevel.Trace : MSLogLevel.Debug);
             }
 
             // When stryker log level is debug or trace, set LibGit2Sharp loglevel
-            if (logLevel < LogEventLevel.Information) // LibGit2Sharp does not handle LogEventLevel.None properly.
-            {
-                var libGit2SharpLogger = LoggerFactory.CreateLogger(nameof(LibGit2Sharp));
-                GlobalSettings.LogConfiguration = new LogConfiguration(LogLevelConverter.Convert(logLevel), (level, message) => libGit2SharpLogger.Log(LogLevelConverter.Convert(level), message));
-            }
+            if (logLevel >= LogEventLevel.Information) return; // LibGit2Sharp does not handle LogEventLevel.None properly.
+
+            var libGit2SharpLogger = LoggerFactory.CreateLogger(nameof(LibGit2Sharp));
+            GlobalSettings.LogConfiguration = new LogConfiguration(LogLevelConverter.Convert(logLevel), (level, message) => libGit2SharpLogger.Log(LogLevelConverter.Convert(level), message));
         }
 
         public static ILoggerFactory LoggerFactory

src/Stryker.Core/Stryker.Core.UnitTest/Compiling/CSharpCompilingProcessTests.cs

@@ -62,12 +62,11 @@ public int Subtract(int first, int second)
 
             var target = new CsharpCompilingProcess(input, rollbackProcessMock.Object);
 
-            using (var ms = new MemoryStream())
-            {
-                var result = target.Compile(new Collection<SyntaxTree>() { syntaxTree }, ms, null);
-                result.Success.ShouldBe(true);
-                ms.Length.ShouldBeGreaterThan(100, "No value was written to the MemoryStream by the compiler");
-            }
+            using var ms = new MemoryStream();
+            using var symbol = new MemoryStream();
+            var result = target.Compile(new Collection<SyntaxTree>() { syntaxTree }, ms, symbol);
+            result.Success.ShouldBe(true);
+            ms.Length.ShouldBeGreaterThan(100, "No value was written to the MemoryStream by the compiler");
         }
 
         [Fact]

src/Stryker.Core/Stryker.Core.UnitTest/Compiling/CSharpRollbackProcessTests.cs

@@ -1,4 +1,5 @@
 using System;
+using System.CodeDom.Compiler;
 using System.Collections.Generic;
 using System.Collections.ObjectModel;
 using System.IO;
@@ -839,28 +840,51 @@ public void Break()
             var fixedCompilation = target.Start(compiler, compiler.Emit(ms).Diagnostics, false, false);
             fixedCompilation.Compilation.Emit(ms).Success.ShouldBeTrue();
 
-            // validate that only one of the compile errors marked the mutation as rollbacked.
-            fixedCompilation.RollbackedIds.ShouldBe(new Collection<int> { 1 });
+            // validate that only one of the compile errors marked the mutation as rolled back.
+            fixedCompilation.RollbackedIds.ShouldBe([1]);
         }
 
         [Fact]
         public void RollbackProcess_ShouldOnlyRaiseExceptionOnFinalAttempt()
         {
-            var syntaxTree = CSharpSyntaxTree.ParseText(@"
+
+                        var syntaxTree = CSharpSyntaxTree.ParseText(@"
 using System;
 
 namespace ExampleProject
 {
-    public class MyException: Exception
+    public class Query
     {
-        public MyException(""a""-""b"", new Exception())
+        public int ActiveMutation = 1;
+
+        public void Break()
         {
+            if(ActiveMutation == 1)
+            {
+                string someQuery = ""test"";
+                new Uri(new Uri(string.Empty), ""/API?"" - someQuery);
+            }
+            else
+            {
+                string someQuery = ""test"";
+                new System.Uri(new System.Uri(string.Empty), ""/API?"" + someQuery);
+            }
+            var error = ""a""-""b"":
         }
     }
 }");
+            var ifStatement = syntaxTree
+                .GetRoot()
+                .DescendantNodes()
+                .First(x => x is IfStatementSyntax);
+            var annotatedSyntaxTree = syntaxTree.GetRoot()
+                .ReplaceNode(
+                    ifStatement,
+                    ifStatement.WithAdditionalAnnotations(GetMutationIdMarker(1), _ifEngineMarker)
+                ).SyntaxTree;
 
             var compiler = CSharpCompilation.Create("TestCompilation",
-                syntaxTrees: new Collection<SyntaxTree>() { syntaxTree },
+                syntaxTrees: new Collection<SyntaxTree>() { annotatedSyntaxTree },
                 options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary),
                 references: new List<PortableExecutableReference>() {
                     MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
@@ -871,10 +895,11 @@ public MyException(""a""-""b"", new Exception())
             var target = new CSharpRollbackProcess();
 
             using var ms = new MemoryStream();
-            var compileResult = compiler.Emit(ms);
+            // first compilation will roll back the mutation
+            var fixedCompilation = target.Start(compiler, compiler.Emit(ms).Diagnostics, false, false);
 
-            Should.NotThrow(() => target.Start(compiler, compileResult.Diagnostics, false, false));
-            Should.Throw<CompilationException>(() => target.Start(compiler, compileResult.Diagnostics, true, false));
+            // next attempt cannot roll back anything, so it assumes this is not fixable
+            Should.Throw<CompilationException>(() => target.Start(fixedCompilation.Compilation, fixedCompilation.Compilation.Emit(ms).Diagnostics, true, false));
         }
     }
 }

src/Stryker.Core/Stryker.Core.UnitTest/Fsharp/FsharpFileDetectionTests.cs

@@ -45,7 +45,7 @@ public FsharpFileDetectionTests()
     </ItemGroup>
 </Project>";
         }
-
+        /*
         public void Stryker_FsharpShouldRetrieveSourcefiles()
         {
             var fileSystem = new MockFileSystem(new Dictionary<string, MockFileData>
@@ -59,13 +59,13 @@ public void Stryker_FsharpShouldRetrieveSourcefiles()
                     { Path.Combine(_filesystemRoot, "ExampleProject", "node_modules", "Some package"), new MockFileData("bla") }, // node_modules should be excluded
                 });
             var projectFileReaderMock = new Mock<IProjectFileReader>(MockBehavior.Strict);
-            projectFileReaderMock.Setup(x => x.AnalyzeProject(_testProjectPath, null, "fsharp", null))
+            projectFileReaderMock.Setup(x => x.AnalyzeProject(_testProjectPath, null, "fsharp", null, null))
                 .Returns(TestHelper.SetupProjectAnalyzerResult(
                     projectReferences: new List<string>() { _sourceProjectPath },
                     targetFramework: "netcoreapp2.1",
                     projectFilePath: _testProjectPath,
                     references: new string[] { "" }).Object);
-            projectFileReaderMock.Setup(x => x.AnalyzeProject(_sourceProjectPath, null, "fsharp", null))
+            projectFileReaderMock.Setup(x => x.AnalyzeProject(_sourceProjectPath, null, "fsharp", null, null))
                 .Returns(TestHelper.SetupProjectAnalyzerResult(
                     projectReferences: new List<string>() { _sourceProjectPath },
                     targetFramework: "netcoreapp2.1",
@@ -77,5 +77,6 @@ public void Stryker_FsharpShouldRetrieveSourcefiles()
 
             result.ProjectContents.GetAllFiles().Count().ShouldBe(2);
         }
+        */
     }
 }