[Suggestion] add API for detecting benchmark run failures.

[ig-sinicyn]

There's no way to detect run failures such as incompatible target platform.

Use case:

		[Fact]
		public void InProcessCannotRun()
		{
			var otherPlatform = IntPtr.Size == 8
				? Platform.X86
				: Platform.X64;

			var otherPlatformConfig = new ManualConfig()
				.With(Job.Dry.With(InProcessToolchain.Instance).WithInvocationCount(UnrollFactor).WithUnrollFactor(UnrollFactor).With(otherPlatform))
				.With(new OutputLogger(Output))
				.With(DefaultColumnProviders.Instance);

			var runInfo = BenchmarkConverter.TypeToBenchmarks(typeof(BenchmarkAllCases), otherPlatformConfig);
			var summary = BenchmarkRunner.Run(runInfo);

			// summary is empty; no way to detect from code what vent wrong;
			// output is:
			/*
				// Benchmark BenchmarkAllCases.InvokeOnceVoid: Dry(Platform=X86, Toolchain=InProcessToolchain, InvocationCount=16, IterationCount=1, LaunchCount=1, RunStrategy=ColdStart, UnrollFactor=16, WarmupCount=1)
				// cannot be run in-process. Validation errors:
				//    * Job Dry, EnvironmentMode.Platform was run as X64 (X86 expected). Fix your test runner options.
			 */

			// Workaround (not reliable)
			Assert.Empty(summary.BenchmarksCases);

			return;
		}

Suggestions:

• call the IToolchain.IsSupported method from validator and add the result into summary.ValidationErrors
• add a property that indicates that benchmark completed without errors, summary.CompletedWithoutErrors or something like this.

[AndreyAkinshin]

call the IToolchain.IsSupported method from validator and add the result into summary.ValidationErrors

It's a good idea!

add a property that indicates that benchmark completed without errors, summary.CompletedWithoutErrors or something like this.

We already have a benchmark report in the summary. Currently, MarkdownExporter uses the following code to detect broken benchmarks:

var benchmarksWithTroubles = summary.Reports.Where(r => !r.GetResultRuns().Any()).Select(r => r.BenchmarkCase).ToList();

Currently, BenchmarkReport contains GenerateResult, BuildResult, and the list of result runs. I think we can introduce Status property which contains the final conclusion about benchmark status (including error text messages in case of any problems). In this case, we can implement an extension method for CompletedWithoutErrors based on the status values.

[ig-sinicyn]

@AndreyAkinshin
Just checked, the summary.Reports is empty too. To repro: copy the code into the InProcessTest.

		[Fact]
		public void InProcessCannotRun()
		{
			var otherPlatform = IntPtr.Size == 8
				? Platform.X86
				: Platform.X64;

			var otherPlatformConfig = new ManualConfig()
				.With(Job.Dry.With(InProcessToolchain.Instance).WithInvocationCount(UnrollFactor).WithUnrollFactor(UnrollFactor).With(otherPlatform))
				.With(new OutputLogger(Output))
				.With(DefaultColumnProviders.Instance);

			var runInfo = BenchmarkConverter.TypeToBenchmarks(typeof(BenchmarkAllCases), otherPlatformConfig);
			var summary = BenchmarkRunner.Run(runInfo);

			// proof:
			Assert.Empty(summary.BenchmarksCases);
			Assert.Empty(summary.Reports);
		}

[adamsitnik]

I've marked this issue as up-for-grabs. The contributor who is going to work on this should make sure that the errors are present in summary.ValidationErrors

[emanuel-v-r]

Hi,

Can I grab this one?