Skip to content

Filter out SPMI metric diffs smaller than ±0.001%#125684

Open
Copilot wants to merge 6 commits intomainfrom
copilot/fix-synchronous-metric-differences
Open

Filter out SPMI metric diffs smaller than ±0.001%#125684
Copilot wants to merge 6 commits intomainfrom
copilot/fix-synchronous-metric-differences

Conversation

Copy link
Contributor

Copilot AI commented Mar 17, 2026

Description

SPMI metric diff reporting shows noise as changes even when there are no actual code diffs. The write_metricdiff_markdown_summary function used a bare != equality check, so any tiny numeric variance was reported as significant.

Inlined an abs(compute_pct(...)) >= 0.01 threshold check directly in the list comprehension filter in superpmi.py to suppress metric diffs below ±0.01%. This is consistent with the existing is_significant_pct threshold pattern already used in write_tpdiff_markdown_summary.

A dummy comment change was also added to src/coreclr/jit/compiler.cpp to trigger the SPMI pipeline and validate the fix.

Original prompt

This section details on the original issue you should resolve

<issue_title>JIT: Metrics show changes even with no-diff changes</issue_title>
<issue_description>E.g. https://dev.azure.com/dnceng-public/public/_build/results?buildId=1339686&view=ms.vss-build-web.run-extensions-tab has no diffs, yet several metrics show up as changing. We probably should increase the expected variance of these metrics so that they don't show up unless they are actually changing.

cc @EgorBo </issue_description>

<agent_instructions>Please do not report diffs for SPMI metric diffs if the difference is smaller than +/- 0.01%</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

@dotnet-policy-service
Copy link
Contributor

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Add a significance threshold to write_metricdiff_markdown_summary so
that metric diffs are only reported when the percentage change is at
least 0.01%. This prevents noise from expected variance when there
are no actual code diffs.

Co-authored-by: EgorBo <523221+EgorBo@users.noreply.github.com>
Copilot AI changed the title [WIP] [JIT] Fix metrics reporting for no-diff changes Filter out SPMI metric diffs smaller than ±0.01% Mar 17, 2026
Copilot AI requested a review from EgorBo March 17, 2026 21:47
Co-authored-by: EgorBo <523221+EgorBo@users.noreply.github.com>
Copilot AI requested a review from EgorBo March 17, 2026 21:57
@EgorBo
Copy link
Member

EgorBo commented Mar 17, 2026

@copilot please make a dummy change in jit to trigger the spmi pipeline

Co-authored-by: EgorBo <523221+EgorBo@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 17, 2026

@copilot please make a dummy change in jit to trigger the spmi pipeline

Done in 97e1984 — added a comment line to compiler.cpp to trigger the SPMI pipeline.

@EgorBo EgorBo changed the title Filter out SPMI metric diffs smaller than ±0.01% Filter out SPMI metric diffs smaller than ±0.001% Mar 18, 2026
@EgorBo EgorBo marked this pull request as ready for review March 18, 2026 22:30
Copilot AI review requested due to automatic review settings March 18, 2026 22:30
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to reduce noise in SuperPMI “metricdiff” markdown reporting by suppressing insignificant metric changes that are due to tiny numeric variance rather than real codegen differences.

Changes:

  • Updated write_metricdiff_markdown_summary to filter metric diffs using a percent-change threshold instead of a raw != comparison.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

JIT: Metrics show changes even with no-diff changes

3 participants