Add Orchard Core command-line evaluation benchmark - #14634
Open
OvesN wants to merge 7 commits into
Open
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in BenchmarkDotNet benchmark that exercises MSBuild’s -getProperty / -getItem command-line query path against a large, representative SDK-style project (Orchard Core), focusing on evaluation cost and allocations per query.
Changes:
- Added
OrchardCoreEvaluationBenchmarkto repeatedly execute in-proc MSBuild CLI queries forTargetFrameworksandPackageReference. - Updated benchmark runner (
Program.cs) to include/exclude the Orchard Core benchmark based on a project-path option/env-var. - Documented how to run the new benchmark and added a new custom CLI option in the benchmarks readme.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/MSBuild.Benchmarks/readme.md | Documents how to run the Orchard Core evaluation benchmarks and the new --orchard-core-project option. |
| src/MSBuild.Benchmarks/Program.cs | Adds argument parsing for --orchard-core-project and conditionally filters benchmark types based on env var presence. |
| src/MSBuild.Benchmarks/OrchardCoreEvaluationBenchmark.cs | New benchmark measuring repeated -getProperty / -getItem queries via MSBuildApp.Execute, with setup/validation and SDK location logic. |
| src/MSBuild.Benchmarks/MSBuild.Benchmarks.csproj | Excludes the new benchmark from .NET Framework builds and adds a reference to the MSBuild command-line project. |
Comment on lines
+119
to
+121
| if (parameterIndex == argsList.Count - 1 || | ||
| string.IsNullOrWhiteSpace(argsList[parameterIndex + 1]) || | ||
| argsList[parameterIndex + 1].StartsWith("--", StringComparison.Ordinal)) |
Comment on lines
+74
to
+78
| ValidateQuery( | ||
| _getItemsArguments, | ||
| output => | ||
| output.Contains(ItemType, StringComparison.OrdinalIgnoreCase) && | ||
| !output.Contains($"\"{ItemType}\": []", StringComparison.OrdinalIgnoreCase)); |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e4186d97-7831-4ee8-8280-8876770c8cd7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
MSBuild has synthetic evaluation benchmarks, but no representative benchmark for targetless
-getPropertyand-getItemqueries against a large SDK-style project. These queries exercise command-line processing, SDK and toolset setup, and project evaluation, and their cost depends on how much of the evaluation pipeline is required to produce the requested result.Changes Made
Adds a dedicated
MSBuild.OrchardCore.Benchmarksexecutable for benchmarks that depend on an external Orchard Core checkout, keeping those dependencies and configuration separate from the general synthetic benchmark suite.The
OrchardCoreEvaluationBenchmarkmeasures repeated command-line queries against Orchard Core's outer cross-targeting project:GetPropertyqueriesTargetFrameworks.GetItemsqueriesPackageReferenceitems.MSBuildApp.Executepath, including command-line parsing and evaluation setup, without directly selecting aProjectEvaluationStage; the code under test determines which evaluation phases run.The dedicated runner follows the existing
MSBuild.Benchmarksconfiguration and supports the same ETW, NGEN/ReadyToRun, and JIT-inlining options, while accepting the Orchard Core project path as its workload input.Testing
Notes
The benchmark invokes the MSBuild command-line implementation in-process, so it measures repeated query processing and project evaluation but excludes operating-system process startup. It intentionally does not set
TargetFramework, matching queries against the outer cross-targeting Orchard Core project. The dedicated project targets the current .NET runtime and requires a restored Orchard Core checkout.