Add AI issue assessment workflow with github/ai-assessment-comment-labeler #1498

Copilot · 2025-11-28T08:56:48Z

Closes: #667

Adds AI-powered issue assessment using github/ai-assessment-comment-labeler to automatically analyze new issues.

Changes:

Added .github/workflows/ai-issue-assessment.yml — Workflow triggered on issues with bug label OR newly opened issues without any labels
Added .github/prompts/bug-report-review.prompt.yml — Prompt assessing bug report completeness (description, version, repro steps, expected vs actual, logs)
Added .github/prompts/default-issue-review.prompt.yml — Default prompt for unlabeled issues that analyzes issue type and completeness

Usage:

The workflow runs automatically when an issue is opened without labels
The workflow also runs when an issue receives the bug label
For bug reports: AI assesses completeness and adds labels like ai:bug-report-review:ready for review or ai:bug-report-review:missing details
For unlabeled issues: AI categorizes the issue type (bug, feature request, question, etc.) and assesses clarity

Tradeoffs:

Runs on unlabeled issues and bug labeled issues to balance coverage with noise reduction per issue requirements
Two prompts: bug-specific assessment and a default general assessment for unlabeled issues

Original prompt

This section details on the original issue you should resolve

<issue_title>Add https://github.com/github/ai-assessment-comment-labeler</issue_title>
<issue_description>Add a workflow to automatically analyse new issues with:
https://github.com/github/ai-assessment-comment-labeler

Refer to the installer docs that describe in more details how to use this workflow: https://github.com/github/ai-issue-assessment-commenter-installer/blob/main/README.md

Make minimal changes, it's better to run it less often than to create noise. If you are unsure don't add anything.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Add https://github.com/github/ai-assessment-comment-labeler #1497

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

…beler Co-authored-by: JoannaaKL <67866556+JoannaaKL@users.noreply.github.com>

JoannaaKL · 2025-11-28T09:10:36Z

.github/workflows/ai-issue-assessment.yml

+          issue_body: ${{ github.event.issue.body }}
+          ai_review_label: 'triage-pending'
+          prompts_directory: '.github/prompts'
+          labels_to_prompts_mapping: 'bug,bug-report-review.prompt.yml'


@copilot remove bug-report-review.prompt.yml, this label doesn't exist

Removed bug-report-review.prompt.yml and the workflow file in 705ca08.

Don't remove bug-report-review.prompt.yml file!!

@copilot Don't remove bug-report-review.prompt.yml file!!

Restored bug-report-review.prompt.yml in 0bfc619.

JoannaaKL · 2025-11-28T09:11:25Z

.github/workflows/ai-issue-assessment.yml

+
+on:
+  issues:
+    types: [labeled]


@copilot run this workflow on issues that have label bug and issues that don't have any label

Updated workflow to trigger on issues with bug label OR newly opened issues without any labels in 0bfc619.

Co-authored-by: JoannaaKL <67866556+JoannaaKL@users.noreply.github.com>

Copilot

Pull request overview

This PR adds automated AI-powered issue assessment using the github/ai-assessment-comment-labeler action to help triage new issues and bug reports.

Key changes:

Adds workflow that triggers on newly opened unlabeled issues and issues receiving the 'bug' label
Configures AI assessment to evaluate bug report completeness and suggest appropriate labels

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`.github/workflows/ai-issue-assessment.yml`	Defines workflow triggered on issue opened/labeled events, runs AI assessment action with bug report prompt mapping
`.github/prompts/bug-report-review.prompt.yml`	Configures AI prompt to assess bug reports for completeness (description, version, repro steps, expected behavior, logs) and provide one of three assessment levels

.github/workflows/ai-issue-assessment.yml

Copilot · 2025-11-28T09:27:10Z

.github/workflows/ai-issue-assessment.yml

+          issue_number: ${{ github.event.issue.number }}
+          issue_body: ${{ github.event.issue.body }}
+          prompts_directory: '.github/prompts'
+          labels_to_prompts_mapping: 'bug,bug-report-review.prompt.yml'


The labels_to_prompts_mapping parameter maps 'bug' label to 'bug-report-review.prompt.yml', but the workflow also runs for newly opened issues without any labels (line 10). For unlabeled issues, it's unclear which prompt will be used since there's no 'bug' label to match against.

Consider either:

Adding a default prompt mapping for unlabeled issues

Documenting that unlabeled issues won't be assessed (and updating the workflow condition to only run on labeled bugs)

Verifying that the action handles this case appropriately with a fallback prompt

Suggested change

labels_to_prompts_mapping: 'bug,bug-report-review.prompt.yml'

labels_to_prompts_mapping: 'bug,bug-report-review.prompt.yml;default,default-issue-review.prompt.yml'

@copilot but default-issue-review.prompt.yml doesn't exist, can you propose how it could look like?

Created default-issue-review.prompt.yml in 5e1b41c. It analyzes new issues to determine their type (bug, feature request, question, etc.) and assesses whether they have enough information for maintainers to act on.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: JoannaaKL <67866556+JoannaaKL@users.noreply.github.com>

SamMorrowDrums

Let's try it! I hope it suggests to close all my issues labelling as "Won't-fix"

Initial plan

cc60c3c

Copilot AI assigned Copilot and JoannaaKL Nov 28, 2025

Copilot started work on behalf of JoannaaKL November 28, 2025 08:57 View session

Add AI issue assessment workflow with github/ai-assessment-comment-la…

9ec4c81

…beler Co-authored-by: JoannaaKL <67866556+JoannaaKL@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add workflow to analyse new issues automatically~~ Add AI issue assessment workflow with github/ai-assessment-comment-labeler Nov 28, 2025

Copilot AI requested a review from JoannaaKL November 28, 2025 09:06

Copilot finished work on behalf of JoannaaKL November 28, 2025 09:06

JoannaaKL reviewed Nov 28, 2025

View reviewed changes

Copilot started work on behalf of JoannaaKL November 28, 2025 09:11 View session

JoannaaKL reviewed Nov 28, 2025

View reviewed changes

Remove AI issue assessment workflow and prompt file per review feedback

705ca08

Co-authored-by: JoannaaKL <67866556+JoannaaKL@users.noreply.github.com>

Copilot AI changed the title ~~Add AI issue assessment workflow with github/ai-assessment-comment-labeler~~ Remove AI issue assessment workflow Nov 28, 2025

Copilot AI requested a review from JoannaaKL November 28, 2025 09:14

Copilot finished work on behalf of JoannaaKL November 28, 2025 09:14

Copilot started work on behalf of JoannaaKL November 28, 2025 09:14 View session

Restore workflow and prompt, run on bug label or unlabeled issues

0bfc619

Co-authored-by: JoannaaKL <67866556+JoannaaKL@users.noreply.github.com>

Copilot AI changed the title ~~Remove AI issue assessment workflow~~ Add AI issue assessment workflow with github/ai-assessment-comment-labeler Nov 28, 2025

Copilot finished work on behalf of JoannaaKL November 28, 2025 09:18

Copilot started work on behalf of JoannaaKL November 28, 2025 09:18 View session

Copilot finished work on behalf of JoannaaKL November 28, 2025 09:20

JoannaaKL marked this pull request as ready for review November 28, 2025 09:24

JoannaaKL requested a review from a team as a code owner November 28, 2025 09:24

Copilot AI review requested due to automatic review settings November 28, 2025 09:24

JoannaaKL approved these changes Nov 28, 2025

View reviewed changes

Copilot started reviewing on behalf of JoannaaKL November 28, 2025 09:24 View session

Copilot finished reviewing on behalf of JoannaaKL November 28, 2025 09:26

Copilot AI reviewed Nov 28, 2025

View reviewed changes

Update .github/workflows/ai-issue-assessment.yml

f68072e

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started work on behalf of JoannaaKL November 28, 2025 09:28 View session

Add default-issue-review.prompt.yml for unlabeled issues

5e1b41c

Co-authored-by: JoannaaKL <67866556+JoannaaKL@users.noreply.github.com>

Copilot finished work on behalf of JoannaaKL November 28, 2025 09:33

Copilot AI requested a review from JoannaaKL November 28, 2025 09:33

JoannaaKL approved these changes Nov 28, 2025

View reviewed changes

SamMorrowDrums approved these changes Nov 28, 2025

View reviewed changes

JoannaaKL merged commit 4d6d64e into main Nov 28, 2025
16 checks passed

JoannaaKL deleted the copilot/add-issue-analysing-workflow branch November 28, 2025 09:46

BrewTestBot mentioned this pull request Dec 2, 2025

github-mcp-server 0.24.0 Homebrew/homebrew-core#256728

Merged

	labels_to_prompts_mapping: 'bug,bug-report-review.prompt.yml'
	labels_to_prompts_mapping: 'bug,bug-report-review.prompt.yml;default,default-issue-review.prompt.yml'

Add AI issue assessment workflow with github/ai-assessment-comment-labeler #1498

Add AI issue assessment workflow with github/ai-assessment-comment-labeler #1498

Uh oh!

Conversation

Copilot AI commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments on the Issue (you are @copilot in this section)

Uh oh!

JoannaaKL Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

JoannaaKL Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

JoannaaKL Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

JoannaaKL Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

JoannaaKL Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

SamMorrowDrums left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Nov 28, 2025 •

edited

Loading