Skip to content

CI: Fix "Publish Test Results" job to actually fail when tests fail#708

Closed
Copilot wants to merge 3 commits intomainfrom
copilot/analyze-ci-action-failures
Closed

CI: Fix "Publish Test Results" job to actually fail when tests fail#708
Copilot wants to merge 3 commits intomainfrom
copilot/analyze-ci-action-failures

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Feb 25, 2026

Summary

The EnricoMi/publish-unit-test-result-action with fail_on: test failures only marks the "NUnit Tests" check run (created via the Checks API) as failed — it does not fail the GitHub Actions job step. The "Publish Test Results" job therefore always concluded success, making it useless as a branch protection guard. PRs with failing tests could be merged unimpeded.

Two changes to CI.yml:

  • comment_mode conditional — prevents the publish step from posting/updating comments on already-closed PRs when CI runs on push to main (previously always, now off for non-PR events)
  • Explicit failure propagation — adds a final step that exits 1 if debug_build_and_test did not succeed, so the "Publish Test Results" job actually fails and branch protection blocks the merge:
- name: Fail if build or tests failed
  if: ${{ needs.debug_build_and_test.result != 'success' }}
  run: |
    echo "::error::Build or test failures detected in 'Build Debug and run managed tests' job"
    exit 1

No branch protection rule changes needed — the existing "Publish Test Results" requirement now works correctly.

CI-ready checklist

  • Commit messages follow .github/commit-guidelines.md (subject ≤ 72 chars, no trailing punctuation; if body present, blank line then ≤ 80-char lines).
  • No whitespace warnings locally:
    git fetch origin
    git log --check --pretty=format:"---% h% s" origin/<base>..
    git diff --check --cached
  • Builds/tests pass locally (or I've run the CI-style build via Bash script or MSBuild).
  • For any Src/** folders touched, corresponding AGENTS.md files are updated or explicitly confirmed still accurate.

Notes for reviewers (optional)

The fail_on: test failures setting is kept — it still correctly marks the "NUnit Tests" check run as failed, which surfaces individual test failure annotations. The new explicit step is additive: it propagates the upstream job result to the "Publish Test Results" job conclusion so branch protection actually enforces passing tests.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.


This change is Reviewable

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Feb 25, 2026

@jasonleenaylor Unfortunately, I hit an error while trying to use the custom Copilot setup steps configured for this repository and had to close this PR. The error I am seeing is:

no `copilot-setup-steps` job found in your `copilot-setup-steps.yml` workflow file. Please ensure you have a single job named `copilot-setup-steps`. For more details, see https://gh.io/copilot/actions-setup-steps

Once you or someone with the necessary access fixes the problem, please unassign and then reassign issue #0 to me and I'll retry. Thanks!

Co-authored-by: jasonleenaylor <2295227+jasonleenaylor@users.noreply.github.com>
Copilot AI changed the title [WIP] Analyze CI action failures during main merge CI: Don't comment on closed PRs when push-to-main tests fail Feb 25, 2026
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Feb 25, 2026

@jasonleenaylor Unfortunately, I hit an error while trying to use the custom Copilot setup steps configured for this repository and had to close this PR. The error I am seeing is:

no `copilot-setup-steps` job found in your `copilot-setup-steps.yml` workflow file. Please ensure you have a single job named `copilot-setup-steps`. For more details, see https://gh.io/copilot/actions-setup-steps

Once you or someone with the necessary access fixes the problem, please unassign and then reassign issue #0 to me and I'll retry. Thanks!

Co-authored-by: jasonleenaylor <2295227+jasonleenaylor@users.noreply.github.com>
Copilot AI changed the title CI: Don't comment on closed PRs when push-to-main tests fail CI: Fix "Publish Test Results" job to actually fail when tests fail Feb 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants