[Perf] Linux/x64: 2 Regressions on 7/14/2026 4:54:43 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	1baa904ac12c971494469fc724fb3b9f7126716d
Compare	646ab80e65fcf3cb9f068157f644ead63761ee7c
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	5.87 μs	6.32 μs	1.08	0.06	False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	6.85 μs	7.34 μs	1.07	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Reader*'

Details

System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: BroadTree)

ETL Files

Histogram

JIT Disasms

System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: BroadTree)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: BroadTree) — caused by commit 6d47555 ("Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM (#130484)"). ⚠️ Note: One or more tests show bimodal history, but at least one clean test bisected to a related culprit.

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: 6d47555c87f335445a547d00da9cd7a2034d94ce

🤖 Proposed Next Actions: Known-cause regression — already linked — 0 done, 3 dry-run (rule R3_known_regression_with_prior — see Full Analysis for the table). · 🦆 Re-analysis: ⚠️ disputes (overconfident, bimodal-history, weak-bisection-evidence)

📊 Full Analysis (click to expand)

Summary: Two System.Text.Json.Tests.Perf_Reader benchmarks regressed ~+7% (ReadSingleSpanSequenceEmptyLoop +7.7%, ReadMultiSpanSequenceEmptyLoop +7.2%), both confirmed step changes with strong statistical signal (z=3.06 and z=3.46 over ~296 stable historical runs). The most likely cause is commit 6d47555 ("Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM"), whose merge timestamp matches the regression date exactly and which restructured Utf8JsonReader.SkipWhiteSpace — the hot path these IsDataCompact: False (indented, whitespace-heavy) benchmarks exercise. Regression confidence 4/5, finding confidence 4/5.

Issue Overview

• Issue: [Perf] Linux/x64: 2 Regressions on 7/14/2026 4:54:43 PM +00:00 #131159 - [Perf] Linux/x64: 2 Regressions on 7/14/2026 4:54:43 PM +00:00
• Date: 2026-07-14 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux (Ubuntu 22.04)/x64
• Commits: 1baa904 → 646ab80 (27 commits)
• Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Runtime stack: Desktop + CoreCLR (x64 Linux)
• Historical Data: Available (296–297 prior runs per test; low variance, CV 3.2–3.9%)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
ReadSingleSpanSequenceEmptyLoop (IsDataCompact: False, BroadTree)	5872.15 ns	6323.54 ns	+7.69%	5652.15	219.78	296	3.055	7.78%	N	Step change, stable after-cluster
ReadMultiSpanSequenceEmptyLoop (IsDataCompact: False, BroadTree)	6847.66 ns	7341.27 ns	+7.21%	6613.05	210.58	297	3.458	6.37%	N	Step change, stable after-cluster

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold. Values in ns.

Finding Confidence: 4/5 - Strong historical data (~296 runs, CV ~3–4%), tight after-cluster rules out transient noise, complete commit info. Not raised to 5 because the attribution has not been reproduced via RunWithBuildAtHash.

Candidate Stack Relevance

Detected stack: Desktop + CoreCLR. src/libraries/System.Text.Json/ is BCL (Shared), but it is the code directly executed by these benchmarks on x64 CoreCLR, so the exercised path is genuinely relevant here.

Commit	Touched Paths (summary)	Relevance	Notes
6d47555	src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs	Shared (BCL)	Restructured SkipWhiteSpace, the exact hot path for indented-JSON reads. Merge timestamp = regression date.
70e0b75	src/.../Serialization/Metadata/JsonTypeInfo.Cache.cs	Shared (BCL)	STJ, but serialization/metadata cache only — not touched by a raw Utf8JsonReader.Read() loop. Ruled out.
Remaining 25 commits	src/coreclr/jit, src/mono/wasm, eng/, DAC, networking, etc.	LessLikely / Shared	No path into the STJ reader tokenizer loop; none align with both correlated STJ reader tests.

Confirmed Regressions

System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop (IsDataCompact: False, TestCase: BroadTree) (+7.7%)

• Delta: 5872.15 ns → 6323.54 ns (+7.69%)
• Historical Context: μ=5652.15 ns (σ=219.78, n=296); z-score=3.055; adaptive threshold=7.78%
• Assessment: Step change; after-values cluster tightly at ~6290–6360 ns, fully outside the recent-before distribution.
• Finding Confidence: 4/5 - Abundant, low-variance history; complete commit range.
• Regression Confidence: 4/5 - Direct hot-path code hit, step change with |z|≥3, both correlated tests move together, 4 co-occurring sibling issues reinforce a real shared cause.
• Likely Cause: 6d47555 - "Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM (Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM #130484)"
  • Affected Areas: System.Text.Json Utf8JsonReader (whitespace skipping / tokenizer hot loop)
  • Stack Relevance: Shared (BCL) — directly on the executed x64 CoreCLR path
  • Rationale: The benchmark just constructs a Utf8JsonReader and drives while (json.Read()); over indented JSON (IsDataCompact: False), so SkipWhiteSpace runs between every token and dominates cost. This commit wrapped the vectorized scan in if (!OperatingSystem.IsBrowser()) and moved the scalar fallback loop to be compiled unconditionally after the #if NET block. The PR intends native codegen to be identical (branch folds to a constant, scalar loop dead-code-eliminated), but restructuring a small, hot, inline-candidate method like this can perturb JIT inlining/method layout under tiered compilation and produce exactly this kind of ~7% step. The commit's author/merge timestamp (2026-07-14T16:54:43Z) matches the regression date precisely, and it is the only change in the range touching the reader tokenizer.
  • Candidates Considered:
    • 6d47555 - Restructures the exact SkipWhiteSpace hot path; exact date match. Selected.
    • 70e0b75 - STJ, but serialization metadata cache; not on the reader-loop path. Rejected.
• Verification Status: Not verified. Recommend RunWithBuildAtHash at 6d47555 vs its parent to confirm, since the PR claims codegen-identical native output.

System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop (IsDataCompact: False, TestCase: BroadTree) (+7.2%)

• Delta: 6847.66 ns → 7341.27 ns (+7.21%)
• Historical Context: μ=6613.05 ns (σ=210.58, n=297); z-score=3.458; adaptive threshold=6.37%
• Assessment: Step change; after-values cluster tightly at ~7315–7663 ns, a clear step above the prior regime.
• Finding Confidence: 4/5 - Abundant, low-variance history; complete commit range.
• Regression Confidence: 4/5 - Same hot-path attribution as the single-span variant; |z|≥3, corroborated by the sibling test and co-occurring issues.
• Likely Cause: 6d47555 - "Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM (Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM #130484)"
  • Affected Areas: System.Text.Json Utf8JsonReader (whitespace skipping / tokenizer hot loop)
  • Stack Relevance: Shared (BCL) — directly on the executed x64 CoreCLR path
  • Rationale: Identical mechanism to the single-span variant; the multi-span reader drives the same Read()/SkipWhiteSpace loop over the same indented BroadTree payload. Both variants regressing together by the same magnitude on the same date is strong corroboration of a shared reader hot-path cause.
  • Candidates Considered:
    • 6d47555 - SkipWhiteSpace restructure; exact date match. Selected.
    • 70e0b75 - Serialization metadata cache; off-path. Rejected.
• Verification Status: Not verified. Same rerun recommendation as above.

Open Questions

• Historical Data: Ample (296–297 runs) and low-variance; no data limitation. Note: history shows an earlier step (~170 runs back) likely from PR Use SearchValues in Utf8JsonReader.SkipWhiteSpace #129701 (the original vectorization), separate from this compare-point step.
• Confidence Limitations: The PR explicitly claims native codegen is unchanged ("identical to main"). If that holds, the observed step must arise from a second-order JIT effect (inlining/layout under tiered compilation) rather than logic change — this is the main residual uncertainty and the reason confidence is 4/5 rather than 5/5.
• Verification Needed: Isolated rerun / bisection of 6d47555 vs parent; optionally hardware counters + JIT disasm of SkipWhiteSpace to confirm inlining/layout change.
• Bisection Status: Not performed (deferred; comment posting and bisection handled separately downstream).

Recommended Actions

1. Verification - Run RunWithBuildAtHash on 6d47555 and its parent for both Perf_Reader.ReadSingleSpanSequenceEmptyLoop and ReadMultiSpanSequenceEmptyLoop (IsDataCompact: False, TestCase: BroadTree) to confirm the ~7% step.
2. Source/codegen review - Inspect JIT disasm/inlining of Utf8JsonReader.SkipWhiteSpace before vs after the change under tiered compilation; the PR asserts codegen-identical native output, so confirm whether the added branch + unconditionally-compiled scalar loop altered inlining or method layout.
3. Follow-up - If confirmed, consider isolating the scalar fallback (e.g., separate method / [MethodImpl] guidance) so the browser-only path does not perturb the native hot path; loop in the PR author (Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM #130484) via the runtime STJ area.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
ReadSingleSpanSequenceEmptyLoop(IsDat...	🔍 No change found	-	-	5.9→6.3 μs (+8%)	— (not measured)	2636s
ReadMultiSpanSequenceEmptyLoop(IsData...	✅ Culprit found	6d47555c87f3	✅ RELATED	6.8→7.3 μs (+7%)	6.9→7.3 μs (+6%)	1414s

Total bisection time: 4050s
Success rate: 1/2 tests

System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: BroadTree)

• Status: inconclusive

Helix jobs

Job ID	Submitted by bisection	Last status
fa1d632c-87ed-463f-a4b7-9f964e7423b5	✅	submitted
5ce9fd61-19ee-46b5-8521-01d820a266e1	✅	submitted

Bisection details

…[earlier output omitted]

ALIGNMENT WARNING:
This bisection found no monotone step change and no commit that stably introduces the regression. The benchmark alternates between two performance modes (~5870 ns and ~6270 ns) across unrelated commits and the head commit itself is bimodal. This strongly suggests the reported +7.7% change is due to code alignment / binary layout (or ambient noise near the 2σ floor) rather than a functional regression in the tested code path.

Both Helix jobs ran successfully; the result is analytically inconclusive, not a tooling failure. I'll clean up temp files.

● Remove temp output files (shell)
│ rm -f /tmp/1784627614629-copilot-tool-output-1853a4.txt
│ /tmp/1784629753794-copilot-tool-output-38f7c0.txt; echo cleaned
└ 2 lines…

Bisection complete — result is inconclusive (noise floor / bimodal alignment).

Summary of what I did and found:

• Retrieved the 27-commit range and picked quadrant midpoints via GetDoubleBisectCommits.
• Ran two Helix jobs (multi-sample) over base, ec150b9, 081d9a4, 4d33abd, and head.
• Both runs consistently show a bimodal, non-monotone signal (~5870 ns fast vs ~6270 ns slow) with head flip-flopping. The ~400 ns mode gap sits at the historical 2σ noise floor (439), and there is no single monotone step to localize a culprit.

Per the prompt's noise-floor guidance, I emitted a BISECTION FAILED: inconclusive (noise floor) block plus an alignment warning, rather than committing to a wrong culprit. Helix ran fine (jobs fa1d632c… and 5ce9fd61…); the failure is analytical, pointing to code-alignment/layout sensitivity as the likely cause of the reported +7.7%.

System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: BroadTree)

• Culprit: 6d47555c87f3
• Commit Message: Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM (Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM #130484)
• Relevance: RELATED
• Explanation: The commit modifies Utf8JsonReader.SkipWhiteSpace in System.Text.Json's Reader, the exact hot path the Perf_Reader.ReadMultiSpanSequenceEmptyLoop benchmark exercises. It restructures the vectorized whitespace-skip loop by wrapping it in an if (!OperatingSystem.IsBrowser()) branch; although intended to be codegen-neutral on native, this perturbation of the hot method coincides exactly with the measured step and is a plausible mechanism for the +7.2% x64 regression.

Helix jobs

Job ID	Submitted by bisection	Last status
526a1b2f-908d-4093-8146-f6bd20b787cc	✅	submitted
27710374-673c-43a9-998a-efe8bc837a93	✅	submitted

Per-commit measurements

Commit	Position	Measurements
1baa904ac12c	baseline	6.944 μs
081d9a410022	mid, idx14, good	6.862 μs
1d512aed6caf	idx15, last-good	6.886 μs
48291d45aaef	idx17, bad	7.377 μs
dab6af23aabd	idx18, bad	7.354 μs
3549857e95bf	idx19, bad	7.447 μs
cd8f850923ca	idx20, bad	7.302 μs
4d33abd354bd	threequarter, idx21, bad	7.344, 7.332 μs
646ab80e65fc	head	7.345 μs
6d47555c87f3	idx16, culprit	7.324 μs

Bisection details

…[earlier output omitted]

BISECTION RESULT:
- Culprit Commit: 6d47555c87f335445a547d00da9cd7a2034d94ce
- Commit Message: Disable Utf8JsonReader whitespace-skip vectorization on browser/WASM (#130484)
- Commits Tested: 12
- Confidence: 4
- Commit Relevance: RELATED
- Relevance Explanation: The commit modifies Utf8JsonReader.SkipWhiteSpace in System.Text.Json's Reader, the exact hot path the Perf_Reader.ReadMultiSpanSequenceEmptyLoop benchmark exercises. It restructures the vectorized whitespace-skip loop by wrapping it in an if (!OperatingSystem.IsBrowser()) branch; although intended to be codegen-neutral on native, this perturbation of the hot method coincides exactly with the measured step and is a plausible mechanism for the +7.2% x64 regression.
- Baseline Value: 6885.7 ns
- Culprit Value: 7323.8 ns
- Summary: Bisected a 27-commit range in two Helix passes (quadrant sweep then a 7-commit Q3 sweep). Measurements form two clean plateaus — good ~6861-6944 ns through 1d512aed, bad ~7302-7447 ns from 6d47555c onward — isolating the single step at 6d47555c. Note: the historical 2σ noise floor (~421 ns) is comparable to the ~438 ns step, but the step reproduces across 8 consecutive post-culprit measurements, mitigating single-sample noise concerns.

HELIX JOBS:
- 526a1b2f-908d-4093-8146-f6bd20b787cc
- 27710374-673c-43a9-998a-efe8bc837a93

MEASUREMENTS:
- 1baa904ac12c (baseline): 6944.2 ns
- ec150b96af2d (quarter, idx7): FAILED (0 measurements)
- 081d9a410022 (mid, idx14, good): 6861.5 ns
- 4d33abd354bd (threeQuarter, idx21, bad): 7344.3 ns
- 646ab80e65fc (head): 7345.0 ns
- 1d512aed6caf (idx15, last-good): 6885.7 ns
- 6d47555c87f3 (idx16, culprit): 7323.8 ns
- 48291d45aaef (idx17, bad): 7377.4 ns
- dab6af23aabd (idx18, bad): 7353.7 ns
- 3549857e95bf (idx19, bad): 7447.1 ns
- cd8f850923ca (idx20, bad): 7302.4 ns
- 4d33abd354bd (idx21, bad): 7332.2 ns

🤖 Proposed Next Actions

Rule: R3_known_regression_with_prior — Known-cause regression — already linked.

Verified regression attributed to commit 6d47555; an autofile issue is already linked: dotnet/perf-autofiling-issues#76372.

#	Action	Status	Details
1	Append related regression	⏭️ Skipped (dry-run)	on dotnet/perf-autofiling-issues#76372
2	Remove label	⏭️ Skipped (dry-run)	untriaged
3	Close issue	⏭️ Skipped (dry-run)	reason completed

🦆 Pre-bisection Re-analysis (advisory)

✅ Agrees with the pre-bisection plan — advisory only, not applied

Rationale: Both findings are classified as actionable (noise=False) with high change confidence (4/5) pointing at the same likely commit, and both are correctly queued for bisection, so the noise call and candidate set are consistent and evidence-supported.

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

⚠️ Disputes the analysis — advisory only, not applied

Flags: overconfident, bimodal-history, weak-bisection-evidence

Rationale: Top-level 5/5 change and finding confidence overstate the per-finding 4/5 scores, and a flagged bimodal history plus one inconclusive bisection (only one of two tests converged) warrant lowering both to 4. The culprit is not disputed since the successful bisection converged on it with RELATED relevance.

Suggested confidence: Finding Confidence → 4/5; Change Confidence → 4/5 (advisory — not applied)

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

---

Generated by PerfTriageAgent on 2026-07-21 10:54 UTC

[DrewScoggins]

Related to #130484

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.