Add test workflow for project-related safe output token failure paths #14682

Copilot · 2026-02-09T15:56:20Z

Create test workflow for project-related safe output token failures
Add workflow documentation and test cases
Update project URLs to real GitHub project
Recompile workflow to reflect updated configuration

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

…lures Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

github-actions · 2026-02-09T18:29:01Z

🔍 PR Triage Results

Category: test | Risk: low | Priority: 55/100

Scores Breakdown

Impact: 30/50 - Quality improvement through better test coverage of failure paths (token validation for Projects v2)
Urgency: 10/30 - Test coverage gap for 2.5 hours, quality-focused change aligns with release mode priorities
Quality: 15/20 - Detailed description with comprehensive test cases documented, CI pending

📋 Recommended Action: batch_review

Rationale: This PR adds important test coverage for project-related safe output token failure paths. The changes are low-risk (test files only) and directly contribute to quality and stability by validating error handling. In release mode, test improvements like this align perfectly with our focus on quality.

What This Tests:

update_project - Validates github-actions[bot] detection
create_project_status_update - Tests INSUFFICIENT_SCOPES handling
create_project - Confirms clear messaging for missing permissions

Review Recommendation: Ready for review once CI completes. The comprehensive description and clear test cases make this suitable for efficient batch review with other test-related PRs.

Triaged by PR Triage Agent on 2026-02-09 - Run #21836303013

AI generated by PR Triage Agent

github-actions · 2026-02-10T00:53:43Z

🔍 PR Triage Results - Updated

Category: test | Risk: medium | Priority: 39/100

Scores Breakdown

Impact: 25/50 - Quality improvement through comprehensive failure path testing
Urgency: 8/30 - Age 8.8 hours, quality-focused change aligned with release mode
Quality: 6/20 - CI pending, draft status, detailed description with test cases

📋 Recommended Action: BATCH_REVIEW

✅ Batched with PR #14702 as batch-test-001 - Test workflow additions that can be reviewed together for efficiency.

Changes: Adds test workflow for project-related safe output token failure paths. Validates handling of missing tokens, invalid tokens, and insufficient permissions (1,471 lines across 2 files).

Batch Context: Part of test infrastructure improvements. Both PRs in this batch enhance test coverage for different aspects of the system.

Next Steps:

Review alongside PR Apply strict matching to slash commands (startsWith + exact equality) #14702 in coordinated batch review
Verify test cases cover all token failure scenarios
Confirm safe output error handling is robust
Batch merge after approval

Status: Previously triaged on 2026-02-09, re-triaged with updated batch assignment.

Triaged by PR Triage Agent on 2026-02-10 00:45 UTC

AI generated by PR Triage Agent

mnkiefer · 2026-02-10T06:35:29Z

@copilot Recompile workflow

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot

Pull request overview

Adds a new agentic test workflow intended to validate failure-path messaging and guardrails for project-related safe outputs when the required Projects-capable token is not provided.

Changes:

Introduces a new .md workflow manifest to document and drive manual workflow_dispatch testing for project token failures.
Adds the compiled .lock.yml workflow that runs the agent and processes safe outputs for the three project-related handlers.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/test-project-token-failure.md	New workflow manifest documenting token-failure test cases for `update_project`, `create_project_status_update`, and `create_project`.
.github/workflows/test-project-token-failure.lock.yml	Compiled workflow that executes the agent and runs the safe-output processing step for the new test workflow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-10T06:40:35Z

.github/workflows/test-project-token-failure.lock.yml

+          GH_AW_PROJECT_URL: "https://github.com/orgs/example-org/projects/1"
+          GH_AW_PROJECT_GITHUB_TOKEN: ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN }}
+        with:
+          github-token: ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN }}


The safe_outputs job passes github-token: ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN }} to actions/github-script. When that secret is intentionally unset (as described in this workflow), this evaluates to an empty string and overrides github-script’s default ${{ github.token }}, so the step will run unauthenticated rather than using the default GITHUB_TOKEN. To actually exercise the “default token lacks Projects v2” failure path, omit github-token or set it to a fallback like ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN || github.token }} (or explicitly ${{ github.token }} to force the default token).

Suggested change

github-token: ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN }}

github-token: ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN || github.token }}

Copilot · 2026-02-10T06:40:35Z

.github/workflows/test-project-token-failure.md

+    project: "https://github.com/orgs/github/projects/24068"
+  create-project-status-update:
+    max: 1
+    project: "https://github.com/orgs/github/projects/24068"


The workflow frontmatter hard-codes a real project URL (https://github.com/orgs/github/projects/24068), but the test case examples use https://github.com/orgs/example-org/projects/1 (and the compiled .lock.yml also uses the example-org URL). This inconsistency makes it unclear which project URL is intended for the test and can lead to misleading results if someone follows the examples. Align the frontmatter safe-outputs.*.project values and the example JSON to use the same (preferably placeholder) project URL.

Suggested change

project: "https://github.com/orgs/github/projects/24068"

create-project-status-update:

max: 1

project: "https://github.com/orgs/github/projects/24068"

project: "https://github.com/orgs/example-org/projects/1"

create-project-status-update:

max: 1

project: "https://github.com/orgs/example-org/projects/1"

Copilot · 2026-02-10T06:40:35Z

.github/workflows/test-project-token-failure.md

+1. **Token Detection**: The system should detect that authentication is `github-actions[bot]`
+2. **Early Failure**: Should fail fast with clear error message BEFORE attempting GraphQL queries
+3. **Actionable Guidance**: Error message should explain:


The “Early Failure” expectation says the run should fail before attempting GraphQL queries, but the current project handlers use GraphQL calls (e.g., viewer { login } for update_project) as part of token identity / permission detection. Consider rewording this to something precise like “fail before project resolution/mutation calls” (or “fail before write operations”) so the success criteria matches the actual execution path.

Copilot · 2026-02-10T06:40:42Z

@copilot Recompile workflow

Workflow recompiled in 28c400e. The lock file now reflects the updated project URLs (github/projects/24068) and includes compilation improvements.

Initial plan

7e73bbc

Copilot AI assigned Copilot and mnkiefer Feb 9, 2026

Copilot started work on behalf of mnkiefer February 9, 2026 15:56 View session

Add test-project-token-failure workflow for testing project token fai…

4f92f0f

…lures Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add workflow to test token failure path for safe outputs~~ Add test workflow for project-related safe output token failure paths Feb 9, 2026

Copilot finished work on behalf of mnkiefer February 9, 2026 16:59

github-actions bot added pr-action:batch-review pr-agent:manual pr-priority:medium pr-risk:low pr-type:test labels Feb 9, 2026

github-actions bot mentioned this pull request Feb 9, 2026

[PR Triage Report] PR Triage Report - February 9, 2026 #14688

Closed

github-actions bot added the pr-batch:batch-test-001 label Feb 10, 2026

This was referenced Feb 10, 2026

Apply strict matching to slash commands (startsWith + exact equality) #14702

Merged

[PR Triage Report] PR Triage Report - 2026-02-10 #14706

Closed

Smoke Test: Copilot - 21847538932 #14712

Closed

Smoke Test: Copilot - 21848135068 #14717

Closed

Update project URLs in workflow file

10eeb6b

mnkiefer approved these changes Feb 10, 2026

View reviewed changes

mnkiefer marked this pull request as ready for review February 10, 2026 06:35

Copilot AI review requested due to automatic review settings February 10, 2026 06:35

Merge branch 'main' into copilot/add-token-failure-test-workflow

6e56669

Copilot started work on behalf of mnkiefer February 10, 2026 06:35 View session

Copilot started reviewing on behalf of mnkiefer February 10, 2026 06:36 View session

github-actions bot mentioned this pull request Feb 10, 2026

[PR Triage Report] PR Triage Report - February 10, 2026 #14734

Open

Recompile workflow with updated project URLs

28c400e

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot AI reviewed Feb 10, 2026

View reviewed changes

Copilot AI requested a review from mnkiefer February 10, 2026 06:41

Copilot finished work on behalf of mnkiefer February 10, 2026 06:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test workflow for project-related safe output token failure paths #14682

Add test workflow for project-related safe output token failure paths #14682

Uh oh!

Copilot AI commented Feb 9, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 9, 2026

Uh oh!

github-actions bot commented Feb 10, 2026

Uh oh!

mnkiefer commented Feb 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

Copilot AI commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	github-token: ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN }}
	github-token: ${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN \|\| github.token }}

Add test workflow for project-related safe output token failure paths #14682

Are you sure you want to change the base?

Add test workflow for project-related safe output token failure paths #14682

Uh oh!

Conversation

Copilot AI commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 9, 2026

🔍 PR Triage Results

Scores Breakdown

📋 Recommended Action: batch_review

Uh oh!

github-actions bot commented Feb 10, 2026

🔍 PR Triage Results - Updated

Scores Breakdown

📋 Recommended Action: BATCH_REVIEW

Uh oh!

mnkiefer commented Feb 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Feb 9, 2026 •

edited

Loading