[Perf] Linux/x64: 2 Regressions on 6/25/2026 3:50:04 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	818cfd5edcda78ac3eb201af392fd301ab9ad46b
Compare	4e28c96c3bd32ce2811ef7128641e8926874020e
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
WriteDeepUtf16 - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	2.15 ms	2.36 ms	1.10	0.03	False
WriteDeepUtf16 - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	2.30 ms	2.45 ms	1.07	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Details

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

ETL Files

Histogram

JIT Disasms

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in WriteDeepUtf16(Formatted: False, SkipValidation: True) — caused by commit 69417eb ("Use SearchValues to scan for characters to escape with default Json options (#12"). 2 tests bisected to related culprit commits.

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: 69417eb2873045ca96458ad632170a306e225c36

🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 4 dry-run (rule R4_known_regression_new — see Full Analysis for the table). · 🦆 Re-analysis: ✅ concurs

📊 Full Analysis (click to expand)

Summary: Two System.Text.Json WriteDeepUtf16 benchmarks regressed +9.8% and +6.6% with high confidence, attributed to commit 69417eb ("Use SearchValues to scan for characters to escape with default Json options"). Finding confidence 4/5 (excellent historical data, 294+ runs, low CV), regression confidence 5/5 (z-scores of 7.7 and 6.2, clear step change, surgical commit directly modifying the escaping hot path).

Issue Overview

• Issue: [Perf] Linux/x64: 2 Regressions on 6/25/2026 3:50:04 PM +00:00 #130044 - [Perf] Linux/x64: 2 Regressions on 6/25/2026 3:50:04 PM +00:00
• Date: 2026-06-25 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux/x64
• Commits: 818cfd5edcda → 4e28c96c3bd3 (17 commits)
• Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Runtime Stack: Desktop + CoreCLR (implicit)
• Historical Data: Available (294-297 runs) — data quality is excellent

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
WriteDeepUtf16(Formatted: False, SkipValidation: True)	2.146 ms	2.356 ms	+9.8%	2.172 ms	0.024 ms	297	7.684	5%	N	Step change, stable history (CV=1.1%)
WriteDeepUtf16(Formatted: False, SkipValidation: False)	2.298 ms	2.450 ms	+6.6%	2.277 ms	0.028 ms	294	6.201	5%	N	Step change, stable history (CV=1.2%)

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Excellent historical data (294+ runs per test), low coefficient of variation (1.1-1.2%), clear step change pattern confirmed by sustained elevation in all 31 post-regression runs. Not upgraded to 5 because no verification via RunWithBuildAtHash was performed.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
69417eb28730	src/libraries/System.Text.Json/src/System/Text/Json/Writer/JsonWriterHelper.Escaping.cs, JsonWriterHelper.cs	Relevant	Directly modifies JSON writer escaping hot path used by WriteDeepUtf16
4e28c96c3bd3	src/libraries/System.Private.CoreLib/.../Vector128.cs, SearchValues/...	Shared	Wasm-focused Vector128/SearchValues changes; could indirectly affect SearchValues used by 69417eb
42f3669b3a7f	src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	LessLikely	ComInterop crossgen2 fix, unrelated to JSON
946057658995	src/coreclr/jit/fgopt.cpp	LessLikely	JIT profile data fix for Head/Tail merge; unlikely to affect this benchmark

Confirmed Regressions

WriteDeepUtf16(Formatted: False, SkipValidation: True) (+9.8%)

• Delta: 2.146 ms → 2.356 ms (+9.8%)
• Historical Context: μ=2.172 ms (σ=0.024 ms, n=297); z-score=7.684; adaptive threshold=5%
• Assessment: Step change — before series was stable (earlyMean=2.165 ms vs recentMean=2.165 ms, CV=1.1%). All 31 post-regression values (2.344–2.410 ms) are entirely above the historical range.
• Finding Confidence: 4/5 - Excellent data quality, 297 historical runs, low variance, clear step change
• Regression Confidence: 5/5 - Extreme z-score (7.684), surgical commit directly modifying the escaping code path, both WriteDeepUtf16 variants regressed simultaneously corroborating each other
• Likely Cause: 69417eb28730 - "Use SearchValues to scan for characters to escape with default Json options (Use SearchValues to scan for characters to escape with default Json options #129781)"
  • Affected Areas: System.Text.Json / Utf8JsonWriter / Escaping
  • Stack Relevance: Relevant
  • Rationale: This commit replaces the JavaScriptEncoder.Default.FindFirstCharacterToEncodeUtf8 call in JsonWriterHelper.NeedsEscaping() with a SearchValues<byte>.IndexOfAnyExcept() lookup for the default encoder case. The WriteDeepUtf16 benchmark exercises Utf8JsonWriter which calls NeedsEscaping on every string value written. The new SearchValues-based scan, while intended to be faster via vectorization, appears to have regressed this specific workload — possibly due to the overhead of the SearchValues lookup for the character set used, or due to different branch prediction / inlining behavior. The SkipValidation=True variant regressed more (+9.8% vs +6.6%), consistent with the escaping scan being a larger fraction of work when validation is skipped.
  • Candidates Considered:
    • 69417eb28730 - Directly modifies JSON writer escaping hot path (primary candidate)
    • 4e28c96c3bd3 - Wasm Vector128/SearchValues changes; could affect SearchValues internals but unlikely on x64
• Verification Status: Not verified — bisection or RunWithBuildAtHash recommended to confirm
• Benchmark Source: Perf.Deep.cs#L93-L123

WriteDeepUtf16(Formatted: False, SkipValidation: False) (+6.6%)

• Delta: 2.298 ms → 2.450 ms (+6.6%)
• Historical Context: μ=2.277 ms (σ=0.028 ms, n=294); z-score=6.201; adaptive threshold=5%
• Assessment: Step change — before series was stable (earlyMean=2.263 ms vs recentMean=2.267 ms, CV=1.2%). All 31 post-regression values (2.440–2.511 ms) are well above the historical range.
• Finding Confidence: 4/5 - Excellent data quality, 294 historical runs, low variance, clear step change
• Regression Confidence: 5/5 - Extreme z-score (6.201), same surgical commit as above, corroborated by SkipValidation=True variant
• Likely Cause: 69417eb28730 - "Use SearchValues to scan for characters to escape with default Json options (Use SearchValues to scan for characters to escape with default Json options #129781)"
  • Affected Areas: System.Text.Json / Utf8JsonWriter / Escaping
  • Stack Relevance: Relevant
  • Rationale: Same escaping path change as above. The slightly smaller regression (+6.6% vs +9.8%) is consistent with validation overhead diluting the relative impact of the escaping change.
  • Candidates Considered: Same as above
• Verification Status: Not verified

Open Questions

• Verification Needed: RunWithBuildAtHash at commits 69417eb (compare) and abaf96f (its parent, baseline) would definitively confirm this single commit as the cause
• Root Cause Depth: The SearchValues approach was intended to be faster — profiling with hardware counters (branch mispredictions, cache misses) could reveal why it regressed for this specific workload pattern

Recommended Actions

1. Verification: Verify regression with RunWithBuildAtHash on commit 69417eb28730 vs its parent abaf96f8a4c1 using the WriteDeepUtf16 tests
2. Profiling: Collect hardware counters (collectHardwareCounters=true) comparing old (JavaScriptEncoder.Default.FindFirstCharacterToEncodeUtf8) vs new (SearchValues.IndexOfAnyExcept) paths to understand the regression mechanism
3. Source review: Manual code review of 69417eb28730 focusing on the NeedsEscaping fast path — the SearchValues<byte> and SearchValues<char> instances may have different performance characteristics than the encoder's built-in scan for the specific character set and input patterns used by deep JSON writing
4. Notify PR author: PR Use SearchValues to scan for characters to escape with default Json options #129781 author (MihaZupan) should be notified of the regression for assessment

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
WriteDeepUtf16(Formatted: False, Skip...	✅ Success	69417eb28730	✅ RELATED	2.1→2.4 ms (+10%)	2.2→2.4 ms (+10%)	793s
WriteDeepUtf16(Formatted: False, Skip...	✅ Success	69417eb28730	✅ RELATED	2.3→2.4 ms (+7%)	2.3→2.5 ms (+9%)	4128s

Total bisection time: 4920s
Success rate: 2/2 tests

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

• Culprit: 69417eb28730
• Commit Message: Use SearchValues to scan for characters to escape with default Json options (Use SearchValues to scan for characters to escape with default Json options #129781)
• Relevance: RELATED
• Explanation: The commit modifies System.Text.Json.Writer.JsonWriterHelper.Escaping.cs to use SearchValues instead of JavaScriptEncoder.Default for the NeedsEscaping character scan. The WriteDeepUtf16 benchmark directly exercises this JSON writer escaping path, making this a direct functional change to the code under test.

Helix jobs

Job ID	Submitted by bisection	Last status
cb946ef1-490a-4721-ac79-a3506dd6c359	✅	submitted
73eccdb9-6c19-46de-a633-5ed40ae31f5a	✅	submitted

Per-commit measurements

Commit	Position	Measurements
42f3669b3a7f	commit 0	2.147 ms
2e1bb6526a1f	commit 1	2.170 ms
abaf96f8a4c1	commit 3	2.166 ms
69417eb28730	commit 4	2.384 ms
d08d3aca601f	commit 8	2.399 ms

Bisection details

…[earlier output omitted]

2.170 ms ✓ (baseline)

• abaf96f (commit 3): 2.166 ms ✓ (baseline)
• 69417eb (commit 4): 2.384 ms ✗ (regressed, from job 1)

The regression is between abaf96f and 69417eb. The delta (0.218 ms) is well above the 2σ noise floor (0.048). Let me examine the culprit commit.

● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "69417eb2…
└ {"index":null,"sha":"69417eb2873045ca96458ad632170a306e225c36","abbrevSha":"6...

This is a clear match. The commit modifies System.Text.Json writer escaping logic — switching from JavaScriptEncoder.Default to SearchValues-based scanning for the NeedsEscaping methods. The WriteDeepUtf16 benchmark directly exercises the JSON writer path including character escaping.

Now let me compile the final results:

BISECTION RESULT:
- Culprit Commit: 69417eb2873045ca96458ad632170a306e225c36
- Commit Message: Use SearchValues to scan for characters to escape with default Json options (#129781)
- Commits Tested: 5
- Confidence: 5
- Commit Relevance: RELATED
- Relevance Explanation: The commit modifies `System.Text.Json.Writer.JsonWriterHelper.Escaping.cs` to use `SearchValues` instead of `JavaScriptEncoder.Default` for the `NeedsEscaping` character scan. The `WriteDeepUtf16` benchmark directly exercises this JSON writer escaping path, making this a direct functional change to the code under test.
- Baseline Value: 2166.27 µs
- Culprit Value: 2383.99 µs
- Summary: Bisected 17 commits in 2 Helix jobs. The regression was isolated to commit 69417eb which changes the JSON writer escaping scan from JavaScriptEncoder.Default to SearchValues-based lookup, directly impacting the WriteDeepUtf16 benchmark.

HELIX JOBS:
- cb946ef1-490a-4721-ac79-a3506dd6c359
- 73eccdb9-6c19-46de-a633-5ed40ae31f5a

MEASUREMENTS:
- 42f3669b3a7f (commit 0): 2146.54 µs
- 2e1bb6526a1f (commit 1): 2170.01 µs
- abaf96f8a4c1 (commit 3): 2166.27 µs
- 69417eb28730 (commit 4): 2383.99 µs
- d08d3aca601f (commit 8): 2398.85 µs

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

• Culprit: 69417eb28730
• Commit Message: Use SearchValues to scan for characters to escape with default Json options (Use SearchValues to scan for characters to escape with default Json options #129781)
• Relevance: RELATED
• Explanation: This commit modifies System.Text.Json.Writer.JsonWriterHelper.Escaping.cs, replacing the default JavaScriptEncoder.Default.FindFirstCharacterToEncode path with SearchValues<char>.IndexOfAnyExcept() for the NeedsEscaping method. The WriteDeepUtf16 benchmark directly exercises this escaping code path when writing JSON strings. The regression likely stems from the SearchValues approach having higher overhead for the specific input patterns used in this deep-write benchmark.

Helix jobs

Job ID	Submitted by bisection	Last status
4214a559-6f7f-4f74-b240-337ada133b9b	✅	submitted
696f62bd-37fb-4985-9dff-ae395c843acb	✅	submitted

Per-commit measurements

Commit	Position	Measurements
818cfd5edcda	baseline	2.265 ms
2e1bb6526a1f	q1-second	2.276 ms
42f3669b3a7f	q1-first	2.267 ms
4e28c96c3bd3	compare	2.456 ms
69417eb28730	culprit	2.467 ms
abaf96f8a4c1	pre-culprit	2.271 ms
d08d3aca601f	post-culprit	2.470 ms

Bisection details

…[earlier output omitted]

However, since this is an "optimization" PR using SearchValues, it's surprising it caused a regression. The measured values show:

• Baseline (abaf96f8): 2270.75 μs
• Culprit (69417eb2): 2467.32 μs
• Delta: ~196.6 μs (+8.7%)

This is well above the 2σ noise floor (2.44%), so it's a real regression.

Now let me compile the final result with all measurements.

BISECTION RESULT:
- Culprit Commit: 69417eb2873045ca96458ad632170a306e225c36
- Commit Message: Use SearchValues to scan for characters to escape with default Json options (#129781)
- Commits Tested: 8
- Confidence: 4
- Commit Relevance: RELATED
- Relevance Explanation: This commit modifies `System.Text.Json.Writer.JsonWriterHelper.Escaping.cs`, replacing the default `JavaScriptEncoder.Default.FindFirstCharacterToEncode` path with `SearchValues<char>.IndexOfAnyExcept()` for the `NeedsEscaping` method. The `WriteDeepUtf16` benchmark directly exercises this escaping code path when writing JSON strings. The regression likely stems from the `SearchValues` approach having higher overhead for the specific input patterns used in this deep-write benchmark.
- Baseline Value: 2270.75 us
- Culprit Value: 2467.32 us
- Summary: Bisected 17 commits in 2 Helix jobs. The regression is clearly introduced at commit 69417eb2 (PR #129781), which changes the JSON writer's character escaping scan to use SearchValues instead of JavaScriptEncoder.Default. All prior commits measure ~2265-2277 μs (baseline), while 69417eb2 and all subsequent commits measure ~2456-2470 μs (regressed).

HELIX JOBS:
- 4214a559-6f7f-4f74-b240-337ada133b9b
- 696f62bd-37fb-4985-9dff-ae395c843acb

MEASUREMENTS:
- 818cfd5edcda (baseline): 2265.00 us
- 42f3669b3a7f (Q1-first): 2267.09 us
- 2e1bb6526a1f (Q1-second): 2276.07 us
- abaf96f8a4c1 (pre-culprit): 2270.75 us
- 69417eb28730 (culprit): 2467.32 us
- d08d3aca601f (post-culprit): 2469.77 us
- 4e28c96c3bd3 (compare): 2456.08 us

🤖 Proposed Next Actions

Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

Verified regression attributed to commit 69417eb (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

#	Action	Status	Details
1	Back-reference cause PR	⏭️ Skipped (dry-run)	on culprit PR for 69417eb28730
2	Transfer issue	⏭️ Skipped (dry-run)	→ dotnet/runtime
3	Ping owners	⏭️ Skipped (dry-run)	no owners resolved
4	Remove label	⏭️ Skipped (dry-run)	untriaged

🦆 Pre-bisection Re-analysis (advisory)

✅ Agrees with the pre-bisection plan — advisory only, not applied

Rationale: Both findings show high change confidence (5/5) with consistent finding confidence (4/5), the same likely culprit commit, and relevant stack traces. The two tests are parameter variants of the same benchmark targeting the same code path, so bisecting both is reasonable to confirm the regression is real and consistent across SkipValidation settings.

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

✅ Agrees with the analysis — advisory only, not applied

Rationale: Two related benchmarks both bisected successfully to the same culprit commit with no alignment warnings, no bimodal history, and consistent RELATED stack relevance. The 5/5 finding and change confidence scores are well-supported by the convergent bisection evidence across both test variants.

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

---

Generated by PerfTriageAgent on 2026-06-30 15:35 UTC

[LoopedBard3]

Likely due to #129781, FYI @MihaZupan.

Related Regressions:

• [Perf] Windows/x64: 2 Regressions on 6/25/2026 3:50:04 PM +00:00 perf-autofiling-issues#75182
• [Perf] Linux/x64: 1 Regression on 6/25/2026 8:16:25 PM +00:00 perf-autofiling-issues#75229

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

[MihaZupan]

@MihuBot benchmark SerializeToWriter,WriteDeepUtf16 818cfd5...4e28c96

[MihaZupan]

@MihuBot benchmark SerializeToWriter;WriteDeepUtf16 818cfd5...4e28c96