Skip to content

Add create-check-run safe output type for multi-agent PR analysis#33852

Open
Copilot wants to merge 16 commits into
mainfrom
copilot/add-create-check-run-safe-output
Open

Add create-check-run safe output type for multi-agent PR analysis#33852
Copilot wants to merge 16 commits into
mainfrom
copilot/add-create-check-run-safe-output

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 21, 2026

Workflows can now surface agent results as GitHub Check Runs on the triggering commit/PR — a first-class signal in the PR checks UI, distinct from comments, that supports pass/fail state with title/summary/details.

Config

safe-outputs:
  create-check-run:
    name: "My Agent"  # check run name shown in the UI
    max: 1

Requires checks: write + contents: read permissions (for SHA resolution).

Agent output

{"type": "create_check_run", "conclusion": "failure", "title": "3 issues found", "summary": "...", "text": "..."}

Valid conclusions: success, failure, neutral, cancelled, timed_out, action_required, skipped.

Changes

  • Go config/compiler (create_check_run.go, compiler_types.go, safe_outputs_config.go) — CreateCheckRunConfig struct with Name + BaseSafeOutputConfig; wired into SafeOutputsConfig parser
  • Handler plumbing (safe_output_handlers.go, compiler_safe_outputs_handlers.go, safe_outputs_state.go, safe_outputs_tools_computation.go, safe_outputs_max_validation.go, compiler_safe_outputs_job.go) — follows the same registration pattern as create_issue / create_code_scanning_alert
  • Permissions (permissions_factory.go) — new NewPermissionsContentsReadChecksWrite() factory
  • Schema (main_workflow_schema.json) — create-check-run block with name, max, github-token, staged
  • MCP tool (both safe_outputs_tools.json copies) — create_check_run tool with required conclusion/title/summary + optional text
  • JS runtime (create_check_run.cjs) — calls rest.checks.create(); uses GITHUB_SHA for head_sha; truncates fields at 65535 chars; classified as THREAT_WARNING_REVIEWABLE_TYPES
  • Prompt integration (unified_prompt_step.go, tool_description_enhancer.go) — tools budget entry + constraint descriptions (max count, check run name)


✨ PR Review Safe Output Test - Run 26267023456

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● 7.8M ·



✨ PR Review Safe Output Test - Run 26267820166

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● 6M ·



✨ PR Review Safe Output Test - Run 26269860246

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● 5.8M ·


pr-sous-chef: automatic branch update via run https://github.com/github/gh-aw/actions/runs/26271114276

Generated by 👨‍🍳 PR Sous Chef · ● 198.6K ·

Copilot AI and others added 2 commits May 21, 2026 20:53
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add create-check-run safe output for multi-agent PR analysis Add create-check-run safe output type for multi-agent PR analysis May 21, 2026
Copilot AI requested a review from pelikhan May 21, 2026 21:12
@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot merge main and recompile

add support for github-token and github-app custom configuration fields, similar to other safe-outputs.

Copilot AI and others added 2 commits May 21, 2026 22:35
…eck-run-safe-output

# Conflicts:
#	.github/workflows/ai-moderator.lock.yml
#	.github/workflows/changeset.lock.yml
#	.github/workflows/codex-github-remote-mcp-test.lock.yml
#	.github/workflows/daily-cache-strategy-analyzer.lock.yml
#	.github/workflows/daily-fact.lock.yml
#	.github/workflows/daily-observability-report.lock.yml
#	.github/workflows/duplicate-code-detector.lock.yml
#	.github/workflows/grumpy-reviewer.lock.yml
#	.github/workflows/issue-arborist.lock.yml
#	.github/workflows/necromancer.lock.yml
#	.github/workflows/schema-feature-coverage.lock.yml
#	.github/workflows/smoke-call-workflow.lock.yml
#	.github/workflows/smoke-codex.lock.yml

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
- Add GitHubApp field to BaseSafeOutputConfig (all safe-output types inherit it)
- Parse github-app in parseBaseSafeOutputConfig for manual-parse handlers
- Wire create-check-run handler to use minted token expression when github-app is set
- Add per-handler token minting/invalidation steps in buildHandlerManagerStep
- Update JSON schema to allow github-app in create-check-run config block

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 21, 2026

@copilot merge main and recompile

add support for github-token and github-app custom configuration fields, similar to other safe-outputs.

Done in 77543ad:

  • Merged main and resolved lock file conflicts
  • Added github-app support per-handler: BaseSafeOutputConfig now has a GitHubApp field (inherited by all safe-output types including create-check-run); parseBaseSafeOutputConfig parses it; the compiler mints a create-check-run-app-token step before the handler manager step and invalidates it after, injecting ${{ steps.create-check-run-app-token.outputs.token }} as github-token in the handler config
  • Updated main_workflow_schema.json to accept github-app in the create-check-run block
  • Recompiled all 233 lock files (make recompile)

@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot update the smoke-copilot, and smoke-claude prompt and frontmatter to include a check run status safe-outputs.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 21, 2026

@copilot update the smoke-copilot, and smoke-claude prompt and frontmatter to include a check run status safe-outputs.

Done in the latest commit. Both smoke-copilot.md and smoke-claude.md now include:

  • A create-check-run block in safe-outputs (with a descriptive name and max: 1)
  • A Check Run Safe Output Testing step in the prompt that calls create_check_run with conclusion: "success" and a run-specific title/summary

Lock files for both workflows were recompiled (233 total).

@github-actions github-actions Bot mentioned this pull request May 22, 2026
@pelikhan pelikhan added the smoke label May 22, 2026
@pelikhan pelikhan marked this pull request as ready for review May 22, 2026 03:27
Copilot AI review requested due to automatic review settings May 22, 2026 03:27
@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot merge main and recompile

Copilot AI and others added 2 commits May 22, 2026 03:34
…eck-run-safe-output

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 22, 2026

@copilot review and apply #33852 (comment)

Applied in the latest commit. create_check_run.cjs now auto-appends (Result) to the check run name whenever it would collide with GITHUB_WORKFLOW — covering both the no-config-name fallback case and the case where the configured name explicitly matches the workflow name. Three tests updated/added to cover all three branches (collision via fallback, collision via explicit config, no collision).

@pelikhan pelikhan added smoke and removed smoke labels May 22, 2026
@github-actions
Copy link
Copy Markdown
Contributor

📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing...

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 22, 2026

⚠️ Smoke Pi failed. Pi encountered unexpected challenges...

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 22, 2026

🚀 Smoke Gemini MISSION COMPLETE! Gemini has spoken. ✨

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

The threat detection results could not be parsed.

Review the workflow run logs for details.

Verifying connectivity to safeoutputs

@github-actions github-actions Bot removed the smoke label May 22, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 22, 2026

🎬 THE ENDSmoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 22, 2026

🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation...

@github-actions
Copy link
Copy Markdown
Contributor

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

The threat detection results could not be parsed.

Review the workflow run logs for details.

Smoke Test: Gemini - 26269860165

  • GitHub MCP Testing: ✅
  • Web Fetch Testing: ✅
  • File Writing Testing: ✅
  • Bash Tool Testing: ✅
  • Build gh-aw: ❌
  • Overall Status: FAIL
    View Run

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

Smoke Gemini — Powered by Gemini ·

@github-actions
Copy link
Copy Markdown
Contributor

Agent Container Tool Check ✅

All required development tools are available:

Tool Status Version
bash 5.2.21
sh available
git 2.54.0
jq 1.7
yq 4.53.2
curl 8.5.0
gh 2.92.0
node 22.22.3
python3 3.14.5
go 1.24.13
java 21.0.11
dotnet 10.0.300

Result: 12/12 tools available ✅ PASS

🔧 Tool validation by Agent Container Smoke Test · ● 242.3K ·

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Claude — Run §26269860246

Core #1#13: ✅ all pass · PR-review #14#19: ✅ all pass · #20 ⚠️ skipped (no safe PR to close)

Overall: ✅ PASS

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● 5.8M ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💥 Automated smoke test review - all systems nominal!

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● 5.8M

Comment thread pkg/workflow/create_check_run.go Outdated
// CreateCheckRunConfig holds configuration for creating GitHub Check Runs from agent output
type CreateCheckRunConfig struct {
BaseSafeOutputConfig `yaml:",inline"`
Name string `yaml:"name,omitempty"` // Check run name shown in the GitHub Checks UI
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: consider adding a brief godoc comment on the Name field clarifying that it appears in the GitHub Checks UI (you already mention it inline, but a // Name is ... style doc would also surface via go doc).

} else {
// If configData is nil or not a map (e.g., "create-check-run:" with no value),
// still set the default max of 1
createCheckRunLog.Print("No config map provided, using defaults (max=1)")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider unifying the two branches: even when configData is a map, the explicit Max = defaultIntStr(1) fallback could be centralized in parseBaseSafeOutputConfig so the nil/non-map path doesn't need to duplicate that logic.

@github-actions
Copy link
Copy Markdown
Contributor

✅ Smoke Test PASSED

Run: §26269860137

Results: 16/16 tests ✅

  • GitHub MCP ✅
  • MCP Scripts ✅
  • Serena CLI ✅
  • Playwright ✅
  • File Ops ✅
  • Build & Artifact ✅
  • Discussions ✅
  • PR Review ✅
  • Sub-Agent ✅

Author: @app/copilot-swe-agent
Assignees: @pelikhan @Copilot

All Copilot engine features validated successfully! 🎉

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · ● 1.2M ·

@github-actions
Copy link
Copy Markdown
Contributor

🎭 The smoke test agent has arrived! 🎭

Just passing through on my automated rounds, validating all the wonderful MCP servers and safe-output tools! Everything's working beautifully!

Here's a little something I created while testing:

Smoke test was here
CLI agents at work
Testing all features

✨ Keep building amazing things! ✨

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · ● 1.2M ·

@github-actions
Copy link
Copy Markdown
Contributor

Comment Memory

Test memory file content

Smoke test was here
CLI agents at work
Testing all features

Note

This comment is managed by comment memory.

It stores persistent context for this thread in the code block at the top of this comment.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · ● 1.2M ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smoke tests look comprehensive! Tests cover MCP servers, file ops, GitHub integration, and safe outputs. All required functionality appears well-tested.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

  • accounts.google.com
  • android.clients.google.com
  • clients2.google.com
  • contentautofill.googleapis.com
  • safebrowsingohttpgateway.googleapis.com
  • www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · ● 1.2M

Comments that could not be inline-anchored

.github/workflows/smoke-copilot.md:10

Nice smoke test setup! The workflow structure looks solid.

actions/setup/js/create_check_run.cjs:5

Good addition of upload_artifact to the test suite!

@github-actions
Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot add support in safe_output to specify output.title/output.summary. Setup max sizes, sanitize markdown as with other safe outputs. Optional.

Copilot AI and others added 2 commits May 22, 2026 05:45
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…k-run

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 22, 2026

@copilot add support in safe_output to specify output.title/output.summary. Setup max sizes, sanitize markdown as with other safe outputs. Optional.

Done. The create-check-run config now accepts an optional output block:

safe-outputs:
  create-check-run:
    name: "My Agent"
    output:
      title: "Static fallback title"    # max 256 chars
      summary: "Static fallback summary" # max 65535 chars

Behaviour:

  • Agent-provided title/summary always take precedence; config values are used only when the agent omits them
  • All values (both config-level and agent-provided) are now run through sanitizeContent, which neutralizes @mentions, redacts blocked URLs, and enforces max lengths
  • text is also sanitized the same way
  • Schema enforces maxLength: 256 (title) and maxLength: 65535 (summary) at the YAML level

Changes: create_check_run.go, compiler_safe_outputs_handlers.go, main_workflow_schema.json, create_check_run.cjs, create_check_run.test.cjs.

@github-actions
Copy link
Copy Markdown
Contributor

``
@copilot review all comments
Please address unresolved review feedback or propose a plan.

``

Merge state is UNSTABLE — please update/rebase the branch to re-run checks and request reviewer attention. If CI still fails, run tests locally and report results.

Generated by 👨‍🍳 PR Sous Chef · ● 198.6K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add create-check-run safe output for multi-agent PR analysis

3 participants