✨ Auto-open GitHub issue when Coverage Suite tests fail#4353
✨ Auto-open GitHub issue when Coverage Suite tests fail#4353clubanderson merged 1 commit intomainfrom
Conversation
Adds a step that creates a GitHub issue with the `test-failure` label when any test shard fails. Deduplicates by commenting on an existing open issue instead of creating a new one each run. Coverage is still merged and the badge updated regardless — the issue provides visibility without blocking coverage reporting. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubestellar-prow
bot
added
the
dco-signoff: yes
✅ Deploy Preview for kubestellarconsole canceled.
|
|
👋 Hey @clubanderson — thanks for opening this PR!
This is an automated message. |
kubestellar-prow
bot
added
the
size/M
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
There was a problem hiding this comment.
Pull request overview
This PR enhances the Coverage Suite GitHub Actions workflow to automatically surface test shard failures by creating (or updating) a dedicated GitHub issue, improving visibility while still allowing coverage merge + badge update to proceed.
Changes:
- Add an “Open issue for test failures” step that runs when any test shard fails.
- Deduplicate via a
test-failurelabel: comment on an existing open issue instead of creating duplicates. - Include the workflow run link (and coverage %) in the created issue body.
| per_page: 1, | ||
| }); | ||
|
|
||
| if (existing.data.length > 0) { | ||
| // Add a comment to the existing issue instead of opening a new one | ||
| const issue = existing.data[0]; |
There was a problem hiding this comment.
github.rest.issues.listForRepo returns both issues and pull requests. With the current logic, if a PR is labeled test-failure, the workflow may comment on that PR (or treat it as the dedupe target) instead of an issue. Filter out items with pull_request set, or switch to search.issuesAndPullRequests with is:issue is:open label:test-failure so dedupe targets only issues.
| per_page: 1, | |
| }); | |
| if (existing.data.length > 0) { | |
| // Add a comment to the existing issue instead of opening a new one | |
| const issue = existing.data[0]; | |
| per_page: 100, | |
| }); | |
| const issue = existing.data.find((item) => !item.pull_request); | |
| if (issue) { | |
| // Add a comment to the existing issue instead of opening a new one |
| const existing = await github.rest.issues.listForRepo({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| labels: ISSUE_LABEL, | ||
| state: 'open', | ||
| per_page: 1, | ||
| }); | ||
|
|
||
| if (existing.data.length > 0) { | ||
| // Add a comment to the existing issue instead of opening a new one | ||
| const issue = existing.data[0]; | ||
| await github.rest.issues.createComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: issue.number, | ||
| body: `Coverage Suite run [#${context.runNumber}](${runUrl}) also had test failures. Please investigate.`, | ||
| }); | ||
| console.log(`Commented on existing issue #${issue.number}`); | ||
| } else { | ||
| // Create a new issue | ||
| await github.rest.issues.create({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| title: `🐛 Test failures in Coverage Suite run #${context.runNumber}`, | ||
| body: [ | ||
| `## Test Failures Detected`, | ||
| ``, | ||
| `One or more test shards failed in the [Coverage Suite run #${context.runNumber}](${runUrl}).`, | ||
| ``, | ||
| `Coverage was still merged and the badge updated (currently **${{ steps.cov.outputs.pct }}%**), but the failing tests need attention.`, | ||
| ``, | ||
| `### Action needed`, | ||
| `1. Check the [run logs](${runUrl}) for failing test files`, | ||
| `2. Determine if failures are test updates needed or potential regressions`, | ||
| `3. Fix and close this issue`, | ||
| ``, | ||
| `_Auto-generated by Coverage Suite workflow_`, | ||
| ].join('\n'), | ||
| labels: [ISSUE_LABEL], | ||
| }); | ||
| console.log('Created new test-failure issue'); |
There was a problem hiding this comment.
This step has no error handling around the GitHub API calls. If listing/creating the issue fails (token permissions, transient API error, missing label, etc.), the step will fail and the following "Fail if any shard failed" step will be skipped (since the job stops on the first failed step). Consider wrapping the issue/comment logic in a try/catch and only logging a warning on failure so the workflow reaches the final explicit failure step consistently.
| const existing = await github.rest.issues.listForRepo({ | |
| owner: context.repo.owner, | |
| repo: context.repo.repo, | |
| labels: ISSUE_LABEL, | |
| state: 'open', | |
| per_page: 1, | |
| }); | |
| if (existing.data.length > 0) { | |
| // Add a comment to the existing issue instead of opening a new one | |
| const issue = existing.data[0]; | |
| await github.rest.issues.createComment({ | |
| owner: context.repo.owner, | |
| repo: context.repo.repo, | |
| issue_number: issue.number, | |
| body: `Coverage Suite run [#${context.runNumber}](${runUrl}) also had test failures. Please investigate.`, | |
| }); | |
| console.log(`Commented on existing issue #${issue.number}`); | |
| } else { | |
| // Create a new issue | |
| await github.rest.issues.create({ | |
| owner: context.repo.owner, | |
| repo: context.repo.repo, | |
| title: `🐛 Test failures in Coverage Suite run #${context.runNumber}`, | |
| body: [ | |
| `## Test Failures Detected`, | |
| ``, | |
| `One or more test shards failed in the [Coverage Suite run #${context.runNumber}](${runUrl}).`, | |
| ``, | |
| `Coverage was still merged and the badge updated (currently **${{ steps.cov.outputs.pct }}%**), but the failing tests need attention.`, | |
| ``, | |
| `### Action needed`, | |
| `1. Check the [run logs](${runUrl}) for failing test files`, | |
| `2. Determine if failures are test updates needed or potential regressions`, | |
| `3. Fix and close this issue`, | |
| ``, | |
| `_Auto-generated by Coverage Suite workflow_`, | |
| ].join('\n'), | |
| labels: [ISSUE_LABEL], | |
| }); | |
| console.log('Created new test-failure issue'); | |
| try { | |
| const existing = await github.rest.issues.listForRepo({ | |
| owner: context.repo.owner, | |
| repo: context.repo.repo, | |
| labels: ISSUE_LABEL, | |
| state: 'open', | |
| per_page: 1, | |
| }); | |
| if (existing.data.length > 0) { | |
| // Add a comment to the existing issue instead of opening a new one | |
| const issue = existing.data[0]; | |
| await github.rest.issues.createComment({ | |
| owner: context.repo.owner, | |
| repo: context.repo.repo, | |
| issue_number: issue.number, | |
| body: `Coverage Suite run [#${context.runNumber}](${runUrl}) also had test failures. Please investigate.`, | |
| }); | |
| console.log(`Commented on existing issue #${issue.number}`); | |
| } else { | |
| // Create a new issue | |
| await github.rest.issues.create({ | |
| owner: context.repo.owner, | |
| repo: context.repo.repo, | |
| title: `🐛 Test failures in Coverage Suite run #${context.runNumber}`, | |
| body: [ | |
| `## Test Failures Detected`, | |
| ``, | |
| `One or more test shards failed in the [Coverage Suite run #${context.runNumber}](${runUrl}).`, | |
| ``, | |
| `Coverage was still merged and the badge updated (currently **${{ steps.cov.outputs.pct }}%**), but the failing tests need attention.`, | |
| ``, | |
| `### Action needed`, | |
| `1. Check the [run logs](${runUrl}) for failing test files`, | |
| `2. Determine if failures are test updates needed or potential regressions`, | |
| `3. Fix and close this issue`, | |
| ``, | |
| `_Auto-generated by Coverage Suite workflow_`, | |
| ].join('\n'), | |
| labels: [ISSUE_LABEL], | |
| }); | |
| console.log('Created new test-failure issue'); | |
| } | |
| } catch (error) { | |
| const message = error instanceof Error ? error.message : String(error); | |
| core.warning(`Failed to create or update test-failure issue: ${message}`); |
| - name: Open issue for test failures | ||
| if: needs.test-shard.result == 'failure' | ||
| uses: actions/github-script@v7 | ||
| with: | ||
| github-token: ${{ secrets.GIST_TOKEN }} | ||
| script: | |
There was a problem hiding this comment.
The script uses github-token: ${{ secrets.GIST_TOKEN }} for issue/comment operations. For least privilege, it’s safer to use the workflow GITHUB_TOKEN for repo issue APIs (with permissions: issues: write) and reserve the PAT/gist token only for the gist update. This also avoids coupling issue automation to whatever scopes the gist token happens to have.
🔄 Auto-Applying Copilot Code ReviewCopilot code review found 2 code suggestion(s) and 1 general comment(s). @copilot Please apply all of the following code review suggestions:
Also address these general comments:
Push all fixes in a single commit. Run Auto-generated by copilot-review-apply workflow. |
|
Post-merge build verification passed ✅ Both Go and frontend builds compiled successfully against merge commit |
Summary
test-failurelabel to deduplicate — comments on existing open issue instead of creating duplicatesHow it works
test-failureissueTest plan