Skip to content

Conversation

@jbrinkman
Copy link
Collaborator

This PR implements the nightly matrix failure reporter described in .vs/error-workflow.md and addresses GH issue #71.

Closes #71

What’s included

  • tests.yml: Ensure each matrix leg emits TRX and a normalized failures.json; upload both in a stable per-leg artifact name. Run name set to “C# Matrix Tests” for scheduled and manual full‑matrix runs.
  • report-failures.yml: New workflow_run reporter that triggers after the "C# tests" workflow, but only for schedule/workflow_dispatch runs that failed. Aggregates artifacts across all legs and posts to a single rolling issue.
  • Aggregation details: Prefer failures.json; fall back to TRX parsing via xmlstarlet. Build a single Markdown section per failing leg with a concise header (framework, server, host) and a bulleted list of failed tests with short error excerpts. Totals across all legs are summarized at the top. If the pipeline failed with zero detected test failures, report as an infrastructure/job failure.
  • Permissions: Least privilege (actions: read, contents: read, issues: write).
  • Reliability: Artifacts uploaded with if: always() so reporter can fetch even on failures; reporter guards comment duplication using the workflow_run id marker.
  • Build hygiene: Suppress NETSDK1138 in benchmarks to keep CI green on net6.0.

Notes

  • Current reporter scope is schedule/workflow_dispatch full‑matrix runs; it ignores PR/push runs by design to reduce noise.
  • Follow‑ups (optional, separate PR): auto-close the rolling issue on recovery, chunking for very large outputs, and widening branch scope after bedding-in.

Validation

  • YAML lint‑clean locally; reporter exercised in a feature branch with synthetic artifacts. Next nightly will validate end‑to‑end; manual full‑matrix run can also be triggered.

@jbrinkman jbrinkman requested a review from a team as a code owner September 10, 2025 11:21
@jbrinkman jbrinkman force-pushed the jbrinkman/matrix-errors branch from dfd3eef to ae0b7b4 Compare September 10, 2025 16:24
@jbrinkman jbrinkman changed the base branch from main to release-0.9 September 10, 2025 16:26
@jbrinkman jbrinkman force-pushed the jbrinkman/matrix-errors branch from ae0b7b4 to 3a17f6c Compare September 10, 2025 16:27
…r scheduled/full-matrix; remove empty paths filters

Signed-off-by: jbrinkman <joe.brinkman@improving.com>

# Conflicts:
#	.github/workflows/tests.yml
… (aggregate artifacts and update rolling issue)

Signed-off-by: jbrinkman <joe.brinkman@improving.com>
…echo JSON; keep TRX+artifact contract

Signed-off-by: jbrinkman <joe.brinkman@improving.com>
Signed-off-by: jbrinkman <joe.brinkman@improving.com>
# Conflicts:
#	.github/workflows/report-failures.yml

# Conflicts:
#	benchmarks/csharp_benchmark.csproj
Signed-off-by: jbrinkman <joe.brinkman@improving.com>
…ob-level gating; always report even if no individual test failures (infra fail)

Signed-off-by: jbrinkman <joe.brinkman@improving.com>
This reverts commit 606a453.

Signed-off-by: jbrinkman <joe.brinkman@improving.com>
…x via inputs (fallback to title)

Signed-off-by: jbrinkman <joe.brinkman@improving.com>
@jbrinkman jbrinkman force-pushed the jbrinkman/matrix-errors branch from 3a17f6c to bd1c02b Compare September 12, 2025 12:20
@jbrinkman jbrinkman merged commit 7cb74f2 into release-0.9 Sep 12, 2025
13 checks passed
@jbrinkman jbrinkman deleted the jbrinkman/matrix-errors branch September 12, 2025 12:21
jbrinkman added a commit that referenced this pull request Sep 15, 2025
…tor) (#74)

Signed-off-by: jbrinkman <joe.brinkman@improving.com>
alexr-bq pushed a commit that referenced this pull request Oct 21, 2025
…tor) (#74)

Signed-off-by: jbrinkman <joe.brinkman@improving.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automated Matrix CI Failure Reporting for Nightly Builds

4 participants