Skip to content

Add Orchard Core command-line evaluation benchmark - #14634

Open
OvesN wants to merge 7 commits into
dotnet:mainfrom
OvesN:dev/ovesn/orchard-evaluation-benchmark
Open

Add Orchard Core command-line evaluation benchmark#14634
OvesN wants to merge 7 commits into
dotnet:mainfrom
OvesN:dev/ovesn/orchard-evaluation-benchmark

Conversation

@OvesN

@OvesN OvesN commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Context

MSBuild has synthetic evaluation benchmarks, but no representative benchmark for targetless -getProperty and -getItem queries against a large SDK-style project. These queries exercise command-line processing, SDK and toolset setup, and project evaluation, and their cost depends on how much of the evaluation pipeline is required to produce the requested result.

Changes Made

Adds a dedicated MSBuild.OrchardCore.Benchmarks executable for benchmarks that depend on an external Orchard Core checkout, keeping those dependencies and configuration separate from the general synthetic benchmark suite.

The OrchardCoreEvaluationBenchmark measures repeated command-line queries against Orchard Core's outer cross-targeting project:

  • GetProperty queries TargetFrameworks.
  • GetItems queries PackageReference items.
  • Each benchmark method executes 100 queries per invocation and reports elapsed time and managed allocations per query.
  • Queries use the production MSBuildApp.Execute path, including command-line parsing and evaluation setup, without directly selecting a ProjectEvaluationStage; the code under test determines which evaluation phases run.
  • Setup validates that both queries succeed and produce non-empty expected output before measurements begin.

The dedicated runner follows the existing MSBuild.Benchmarks configuration and supports the same ETW, NGEN/ReadyToRun, and JIT-inlining options, while accepting the Orchard Core project path as its workload input.

Testing

  • Full repository build
  • Dedicated Orchard Core benchmark project build
  • Orchard Core benchmark dry run for both query methods

Notes

The benchmark invokes the MSBuild command-line implementation in-process, so it measures repeated query processing and project evaluation but excludes operating-system process startup. It intentionally does not set TargetFramework, matching queries against the outer cross-targeting Orchard Core project. The dedicated project targets the current .NET runtime and requires a restored Orchard Core checkout.

OvesN added 2 commits August 4, 2026 08:25
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
@OvesN OvesN changed the title Add Orchard Core partial evaluation benchmark Add Orchard Core command-line evaluation benchmark Aug 4, 2026
OvesN and others added 2 commits August 4, 2026 10:02
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
@OvesN
OvesN marked this pull request as ready for review August 4, 2026 19:08
Copilot AI review requested due to automatic review settings August 4, 2026 19:08
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in BenchmarkDotNet benchmark that exercises MSBuild’s -getProperty / -getItem command-line query path against a large, representative SDK-style project (Orchard Core), focusing on evaluation cost and allocations per query.

Changes:

  • Added OrchardCoreEvaluationBenchmark to repeatedly execute in-proc MSBuild CLI queries for TargetFrameworks and PackageReference.
  • Updated benchmark runner (Program.cs) to include/exclude the Orchard Core benchmark based on a project-path option/env-var.
  • Documented how to run the new benchmark and added a new custom CLI option in the benchmarks readme.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
src/MSBuild.Benchmarks/readme.md Documents how to run the Orchard Core evaluation benchmarks and the new --orchard-core-project option.
src/MSBuild.Benchmarks/Program.cs Adds argument parsing for --orchard-core-project and conditionally filters benchmark types based on env var presence.
src/MSBuild.Benchmarks/OrchardCoreEvaluationBenchmark.cs New benchmark measuring repeated -getProperty / -getItem queries via MSBuildApp.Execute, with setup/validation and SDK location logic.
src/MSBuild.Benchmarks/MSBuild.Benchmarks.csproj Excludes the new benchmark from .NET Framework builds and adds a reference to the MSBuild command-line project.

Comment thread src/MSBuild.Benchmarks/Program.cs Outdated
Comment on lines +119 to +121
if (parameterIndex == argsList.Count - 1 ||
string.IsNullOrWhiteSpace(argsList[parameterIndex + 1]) ||
argsList[parameterIndex + 1].StartsWith("--", StringComparison.Ordinal))
Comment on lines +74 to +78
ValidateQuery(
_getItemsArguments,
output =>
output.Contains(ItemType, StringComparison.OrdinalIgnoreCase) &&
!output.Contains($"\"{ItemType}\": []", StringComparison.OrdinalIgnoreCase));
OvesN added 2 commits August 4, 2026 21:14
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants