Update bdn 0.16 binarydata and async by DrewScoggins · Pull Request #5217 · dotnet/performance

DrewScoggins · 2026-05-07T20:25:20Z

This change enables runtime-async. It is not on by default, but must be activated by an experiment. The runs will be added in a separate PR.

This change also includes updating BDN to latest as it was required for runtime-async.

Copilot

Pull request overview

This PR updates the performance harness and microbenchmark suite to work with newer BenchmarkDotNet (0.16 nightly), including BDN’s new async validator APIs and addressing a BinaryData naming collision, while also adding an experiment-gated runtime-async feature flag for net11+ runs.

Changes:

Update BenchmarkDotNet version and migrate custom validators to ValidateAsync returning IAsyncEnumerable<ValidationError>.
Rename microbenchmarks’ BinaryData payload type to avoid collision with System.BinaryData.
Adjust harness exporter implementation for BDN 0.16’s exporter API changes and add an experiment-gated runtime-async MSBuild feature flag.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/tests/harness/BenchmarkDotNet.Extensions.Tests/UniqueArgumentsValidatorTests.cs	Switch test to use async validator API.
src/harness/BenchmarkDotNet.Extensions/ValuesGenerator.cs	Add `notnull` constraint to dictionary key generation to align with modern TFMs.
src/harness/BenchmarkDotNet.Extensions/UniqueArgumentsValidator.cs	Migrate to `ValidateAsync` and return `IAsyncEnumerable`.
src/harness/BenchmarkDotNet.Extensions/TooManyTestCasesValidator.cs	Migrate to `ValidateAsync` and nullability updates.
src/harness/BenchmarkDotNet.Extensions/NoWasmValidator.cs	Migrate to `ValidateAsync`.
src/harness/BenchmarkDotNet.Extensions/MandatoryCategoryValidator.cs	Migrate to `ValidateAsync`.
src/harness/BenchmarkDotNet.Extensions/PerfLabExporter.cs	Rework exporter to implement `IExporter` and write JSON artifact asynchronously.
src/harness/BenchmarkDotNet.Extensions/DiffableDisassemblyExporter.cs	Add null-forgiving operators for reflection results.
src/harness/BenchmarkDotNet.Extensions/BenchmarkDotNet.Extensions.csproj	Retarget harness to net8.0 and adjust BDN source reference TFM.
src/Directory.Build.targets	Add experiment-gated `runtime-async` feature flag for net11+.
src/Directory.Build.props	Whitespace-only formatting adjustments.
src/benchmarks/micro/Serializers/DataGenerator.cs	Rename `BinaryData` to `BinaryDataPayload` and update generation/metadata.
src/benchmarks/micro/libraries/System.Text.Json/Serializer/WriteJson.cs	Update generic args to `BinaryDataPayload`.
src/benchmarks/micro/libraries/System.Text.Json/Serializer/ReadJson.cs	Update generic args to `BinaryDataPayload`.
src/benchmarks/micro/README.md	Update documented “available” TFMs to net8.0+.
src/benchmarks/micro/MicroBenchmarks.csproj	Remove net472-only package references block.
scripts/ci_setup.py	Add `runtimeasync` experiment env-var plumbing (`EnableRuntimeAsync=true`).
eng/Versions.props	Bump `BenchmarkDotNetVersion` to a newer nightly.
docs/benchmarkdotnet.md	Update runtime options list and remove outdated/private CLR-build guidance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 4 comments.

@@ -29,10 +29,11 @@ public IEnumerable<ValidationError> Validate(ValidationParameters validationPara
                new ValidationError(
                    isCritical: true,
                    message: $"{group.Key.Descriptor.Type.Name}.{group.Key.Descriptor.WorkloadMethod.Name} has {group.Count()} test cases. It MUST NOT have more than {Limit} test cases. We don't have inifinite amount of time to run all the benchmarks!!",


LoopedBard3

Looks good if the test runs succeed 👍.

Bumps BenchmarkDotNet to the latest master nightly, built from dotnet/BenchmarkDotNet@c7632225 ("Disassembly follow jump trampolines (#3136)") and pushed to the benchmark-dotnet-prerelease internal feed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

BDN 0.16.0 ('Async Refactor', PR #2958) made breaking API changes that the harness still consumed: - ExporterBase.ExportToLog(Summary, ILogger) was removed; the new abstract ExportAsync uses an internal CancelableStreamWriter that subclasses can't satisfy. PerfLabExporter now implements IExporter directly. - IValidator.Validate returning IEnumerable<ValidationError> was replaced with ValidateAsync returning IAsyncEnumerable<ValidationError>. The four validators (UniqueArguments, TooManyTestCases, NoWasm, MandatoryCategory) use .ToAsyncEnumerable() (transitively from BDN's System.Linq.AsyncEnumerable dep), not async + yield return - the latter produces an AsyncIteratorMethodBuilder state machine that deadlocks with BDN's BenchmarkSynchronizationContext. Switched BenchmarkDotNet.Extensions from netstandard2.0 to net8.0. The netstandard2.0 target was only used to enable an opt-in net472 path, but PERFLAB_TARGET_FRAMEWORKS=net472 is no longer exercised in CI and the new BDN APIs would otherwise require polyfill packages. Removed the net472 package conditional from MicroBenchmarks.csproj and the net472/.NET Framework references from docs/benchmarkdotnet.md and the micro README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Switching BenchmarkDotNet.Extensions from netstandard2.0 to net8.0 brought in stricter nullable annotations from the BCL and BDN package, which combined with TreatWarningsAsErrors=true (set in src/Directory.Build.props) turned previously-hidden nullability warnings into build errors. - ValuesGenerator.Dictionary<TKey, TValue>: add 'where TKey : notnull' constraint required by Dictionary<,>. - UniqueArgumentsValidator.BenchmarkArgumentsComparer.Equals: match the IEqualityComparer<T>.Equals(T?, T?) interface signature; handle nulls. - TooManyTestCasesValidator.SkipValidation: parameter must be MemberInfo? since MemberInfo.DeclaringType returns Type?. - DiffableDisassemblyExporter: add null-forgiving (!) on reflection lookups whose return values are intentionally trusted by this type (it's a copy of internal BDN code that operates on known types). - PerfLabExporter: BuildJson can return null; use string?. Verified by running the exact CI commands locally (dotnet restore + build with --framework net11.0) using the SDK from global.json. Build succeeds with 0 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

BDN 0.16's sync entrypoints (BenchmarkSwitcher.Run / BenchmarkRunner.Run) install BenchmarkDotNetSynchronizationContext (a single-threaded message pump) before benchmark discovery. Discovery executes [ParamsSource] and [ArgumentsSource] callbacks; some perf-repo callbacks do sync-over-async (notably SslStreamTests.GetTls13Support, which calls HandshakeAsync(...) .GetAwaiter().GetResult()). Sync-over-async deadlocks on the single-threaded SyncCtx because the awaited continuation is queued back to a pump that the caller is blocking. Switch Program.Main to async Task<int> and await BenchmarkSwitcher.RunAsync. The async entrypoint never installs BenchmarkDotNetSynchronizationContext on the caller, so discovery runs on the default context and sync-over-async in source callbacks no longer deadlocks. Real benchmark execution still gets the SyncCtx semantics it needs because BDN installs it inside the per-benchmark execute path, not on the entrypoint thread. This is the supported BDN-recommended fix for the discovery-deadlock symptom; no BDN code change is required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

PR #5195 enabled the runtime-async language feature unconditionally for net11.0+ via `<Features>`. To avoid affecting baseline measurement runs, gate it behind the existing experiment infrastructure so it only takes effect in the dedicated experiment lane. - src/Directory.Build.targets: only set `runtime-async=on` when the `EnableRuntimeAsync` MSBuild property is `true` (in addition to the existing TFM check). - scripts/ci_setup.py: when `--experiment-name=runtimeasync` is passed, emit `EnableRuntimeAsync=true` as an env var so MSBuild picks it up as a property (matches the pattern used by the `jitoptrepeat` experiment). Verified locally: BDN dry runs against the BinaryDataPayload tests succeed both with and without the env var (18/18 benchmarks each). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.

Copilot AI review requested due to automatic review settings May 7, 2026 20:25

Copilot started reviewing on behalf of DrewScoggins May 7, 2026 20:26 View session

DrewScoggins requested review from LoopedBard3 and caaavik-msft May 7, 2026 20:31

Copilot AI reviewed May 7, 2026

View reviewed changes

DrewScoggins force-pushed the update-bdn-0.16-binarydata-and-async branch from c2fb987 to 57d3cd2 Compare May 7, 2026 21:41

Copilot AI review requested due to automatic review settings May 7, 2026 22:02

DrewScoggins force-pushed the update-bdn-0.16-binarydata-and-async branch from 57d3cd2 to fbbc502 Compare May 7, 2026 22:02

Copilot started reviewing on behalf of DrewScoggins May 7, 2026 22:02 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

LoopedBard3 previously approved these changes May 7, 2026

View reviewed changes

timcassell reviewed May 7, 2026

View reviewed changes

Comment thread src/harness/BenchmarkDotNet.Extensions/PerfLabExporter.cs Outdated

DrewScoggins dismissed LoopedBard3’s stale review via 2ee9131 May 7, 2026 22:27

DrewScoggins force-pushed the update-bdn-0.16-binarydata-and-async branch from fbbc502 to 2ee9131 Compare May 7, 2026 22:27

build-analysis Bot mentioned this pull request May 8, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

agocke and others added 6 commits May 18, 2026 12:51

Enable runtime-async for net11.0+

c28a3ca

Copilot AI review requested due to automatic review settings May 18, 2026 19:55

DrewScoggins force-pushed the update-bdn-0.16-binarydata-and-async branch from 2ee9131 to e3e6736 Compare May 18, 2026 19:55

Copilot started reviewing on behalf of DrewScoggins May 18, 2026 19:55 View session

Copilot AI reviewed May 18, 2026

View reviewed changes

Comment thread src/harness/BenchmarkDotNet.Extensions/BenchmarkDotNet.Extensions.csproj

DrewScoggins merged commit c14e7c4 into dotnet:main May 19, 2026
48 of 76 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update bdn 0.16 binarydata and async#5217

Update bdn 0.16 binarydata and async#5217
DrewScoggins merged 6 commits into
dotnet:mainfrom
DrewScoggins:update-bdn-0.16-binarydata-and-async

DrewScoggins commented May 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LoopedBard3 left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

DrewScoggins commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LoopedBard3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DrewScoggins commented May 7, 2026 •

edited

Loading