Skip to content

feat: compile-time warning when slash_command and bots are configured together#29008

Merged
pelikhan merged 5 commits intomainfrom
copilot/add-compile-time-validation
Apr 29, 2026
Merged

feat: compile-time warning when slash_command and bots are configured together#29008
pelikhan merged 5 commits intomainfrom
copilot/add-compile-time-validation

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 28, 2026

Combining slash_command with bots: creates a potential concurrency clash. When a bot listed in bots: posts a comment that starts with the slash command text (e.g., /command-name), the check_command_position check passes and the bot triggers the workflow — occupying the concurrency slot and potentially blocking a simultaneous manual invocation (since cancel-in-progress is disabled for command-trigger workflows).

Changes

  • pkg/workflow/compiler_validators.go — Emits a warning in validateToolConfiguration when workflowData.Command (slash_command) and workflowData.Bots are both non-empty, pointing users to remove bots: if they want exclusively user-driven slash-command execution.
  • pkg/workflow/slash_command_bots_warning_test.go — Integration tests covering: both configured → warning, slash_command only → no warning, bots only → no warning, multiple bots → warning.

Example

A frontmatter like this now produces a compile-time warning:

on:
  slash_command:
    name: rust-review
    events: [pull_request, pull_request_comment]
  bots:
    - "copilot[bot]"
warning: Both slash_command and bots triggers are configured. If a bot listed in bots:
posts a comment that starts with the slash command text (e.g., /command-name), it will
trigger the workflow and occupy the concurrency slot, potentially blocking simultaneous
manual invocations. To ensure the workflow only runs on explicit user commands,
remove the 'bots:' field.

[!WARNING]

Copilot AI changed the title [WIP] Add compile time validation for front matter trigger conditions feat: compile-time warning when slash_command and bots are configured together Apr 29, 2026
Copilot AI requested a review from pelikhan April 29, 2026 00:20
This was referenced Apr 29, 2026
@pelikhan pelikhan marked this pull request as ready for review April 29, 2026 01:13
Copilot AI review requested due to automatic review settings April 29, 2026 01:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a compile-time warning to help users avoid an unintended trigger interaction when on.slash_command and on.bots are configured together.

Changes:

  • Emit a compiler warning when both workflowData.Command (slash_command) and workflowData.Bots are non-empty.
  • Add an integration test suite covering warning/no-warning scenarios for slash_command and bots combinations.
Show a summary per file
File Description
pkg/workflow/compiler_validators.go Adds a new validation warning for conflicting slash_command + bots configuration.
pkg/workflow/slash_command_bots_warning_test.go Adds integration tests asserting the warning behavior across configuration permutations.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

pkg/workflow/slash_command_bots_warning_test.go:127

  • The read end of the pipe (r) isn't closed and io.Copy's error/byte count are ignored. This can leak file descriptors across the integration test suite. Close r after reading (prefer defer r.Close() right after creation) and check the io.Copy error to ensure stderr capture succeeded.
			w.Close()
			os.Stderr = oldStderr
			var buf bytes.Buffer
			io.Copy(&buf, r)
			stderrOutput := buf.String()
  • Files reviewed: 2/2 changed files
  • Comments generated: 1

Comment on lines +116 to +126
r, w, _ := os.Pipe()
os.Stderr = w

compiler := NewCompiler()
compiler.SetStrictMode(false)
err := compiler.CompileWorkflow(testFile)

w.Close()
os.Stderr = oldStderr
var buf bytes.Buffer
io.Copy(&buf, r)
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.Pipe() errors are being ignored and os.Stderr is reassigned without a defer/t.Cleanup restore. If CompileWorkflow (or any earlier call) fails the test via panic/fatal, this can leak the modified os.Stderr into other tests. Capture and handle the os.Pipe error, and restore os.Stderr via defer/t.Cleanup immediately after reassignment.

This issue also appears on line 123 of the same file.

Suggested change
r, w, _ := os.Pipe()
os.Stderr = w
compiler := NewCompiler()
compiler.SetStrictMode(false)
err := compiler.CompileWorkflow(testFile)
w.Close()
os.Stderr = oldStderr
var buf bytes.Buffer
io.Copy(&buf, r)
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stderr = w
t.Cleanup(func() {
os.Stderr = oldStderr
_ = w.Close()
_ = r.Close()
})
compiler := NewCompiler()
compiler.SetStrictMode(false)
err = compiler.CompileWorkflow(testFile)
if err := w.Close(); err != nil {
t.Fatal(err)
}
os.Stderr = oldStderr
var buf bytes.Buffer
if _, err := io.Copy(&buf, r); err != nil {
t.Fatal(err)
}

Copilot uses AI. Check for mistakes.
@github-actions
Copy link
Copy Markdown
Contributor

🧪 Test Quality Sentinel Report

Test Quality Score: 80/100

Excellent test quality

Metric Value
New/modified tests analyzed 4 (table-driven rows in 1 test function)
✅ Design tests (behavioral contracts) 4 (100%)
⚠️ Implementation tests (low value) 0 (0%)
Tests with error/edge cases 4 (100%)
Duplicate test clusters 0
Test inflation detected Yes (150 test lines / 14 production lines ≈ 10.7:1)
🚨 Coding-guideline violations None

Test Classification Details

Test File Classification Issues Detected
slash_command with bots produces conflict warning slash_command_bots_warning_test.go:24 ✅ Design Verifies observable stderr output and warning counter
slash_command without bots does not produce warning slash_command_bots_warning_test.go:46 ✅ Design Negative case — verifies no spurious warning
bots without slash_command does not produce warning slash_command_bots_warning_test.go:64 ✅ Design Negative case — verifies warning is trigger-specific
slash_command with multiple bots produces conflict warning slash_command_bots_warning_test.go:83 ✅ Design Verifies the warning applies regardless of bot count

Flagged Tests — Requires Review

⚠️ Test Inflation (slash_command_bots_warning_test.go)

Not a hard failure, but worth noting: the test file adds 150 lines against 14 lines of production code (≈10.7:1). The bulk of the test lines are inline YAML fixture content, which is the idiomatic approach for workflow integration tests in this codebase, so the ratio is expected and acceptable. No action required.


Language Support

Tests analyzed:

  • 🐹 Go (*_test.go): 4 scenarios (1 table-driven function) — integration (//go:build integration)
  • 🟨 JavaScript (*.test.cjs, *.test.js): 0 tests

Verdict

Check passed. 0% of new tests are implementation tests (threshold: 30%). All four test scenarios verify observable compiler behavior (stderr warning message and warning count), covering both the positive trigger condition and three negative/edge cases. No coding-guideline violations detected.


📖 Understanding Test Classifications

Design Tests (High Value) verify what the system does:

  • Assert on observable outputs, return values, or state changes
  • Cover error paths and boundary conditions
  • Would catch a behavioral regression if deleted
  • Remain valid even after internal refactoring

Implementation Tests (Low Value) verify how the system does it:

  • Assert on internal function calls (mocking internals)
  • Only test the happy path with typical inputs
  • Break during legitimate refactoring even when behavior is correct
  • Give false assurance: they pass even when the system is wrong

Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.

References: §25085998668

🧪 Test quality analysis by Test Quality Sentinel · ● 544.7K ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 80/100. Test quality is excellent — 0% of new tests are implementation tests (threshold: 30%). All four scenarios verify observable compiler behavior with proper positive and negative cases. No coding-guideline violations.

…and + bots conflict

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

Commit pushed: ae4aff7

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions
Copy link
Copy Markdown
Contributor

🏗️ Design Decision Gate — ADR Required

This PR makes significant changes to core business logic (164 new lines in pkg/) but does not have a linked Architecture Decision Record (ADR).

AI has analyzed the PR diff and generated a draft ADR to help you get started:

📄 Draft ADR: docs/adr/29008-compile-time-warning-slash-command-bots-conflict.md

What to do next

  1. Review the draft ADR committed to your branch — it was generated from the PR diff
  2. Complete the missing sections — add context the AI couldn't infer, refine the decision rationale, and list real alternatives you considered
  3. Commit the finalized ADR to docs/adr/ on your branch
  4. Reference the ADR in this PR body by adding a line such as:

    ADR: ADR-29008: Compile-Time Warning for slash_command + bots Conflict

Once an ADR is linked in the PR body, this gate will re-run and verify the implementation matches the decision.

Why ADRs Matter

"AI made me procrastinate on key design decisions. Because refactoring was cheap, I could always say 'I'll deal with this later.' Deferring decisions corroded my ability to think clearly."

ADRs create a searchable, permanent record of why the codebase looks the way it does. Future contributors (and your future self) will thank you.


📋 Michael Nygard ADR Format Reference

An ADR must contain these four sections to be considered complete:

  • Context — What is the problem? What forces are at play?
  • Decision — What did you decide? Why?
  • Alternatives Considered — What else could have been done?
  • Consequences — What are the trade-offs (positive and negative)?

All ADRs are stored in docs/adr/ as Markdown files numbered by PR number (e.g., 29008-compile-time-warning-slash-command-bots-conflict.md for PR #29008).

🔒 This PR cannot merge until an ADR is linked in the PR body.

References: §25085998674

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · ● 145.1K ·

@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot review all comments

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 29, 2026

@copilot review all comments

Addressed the PR review feedback in 8e82c88: os.Pipe() error is now handled, and os.Stderr is restored via t.Cleanup immediately after reassignment to prevent state leaks if the test panics or fatals.

@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot analyze how slash_commands and bots (copilot code review) may clash

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 29, 2026

@copilot analyze how slash_commands and bots (copilot code review) may clash

Here's how the clash happens:

The trigger sequence

  1. slash_command (e.g. events: [pull_request, pull_request_comment]) compiles to issue_comment + workflow_dispatch. A human must post /command-name in a PR comment to start the workflow.

  2. bots: ["copilot[bot]"] adds copilot[bot] to the allow-list in check_membership.cjs. Because copilot[bot] automatically posts PR review comments (code review activity), it satisfies the same pull_request_comment / issue_comment event — no /command text required.

Why the slash command invocation then gets blocked

  • Both runs end up in the same concurrency group, keyed on the PR number.
  • For command-trigger workflows, cancel-in-progress is explicitly disabled (shouldEnableCancelInProgress returns false when isCommandTrigger == true).
  • So the bot-triggered run holds the concurrency slot; the user's manual /command run is queued and silently waits (or is dropped if the queue fills).

The practical impact with copilot code review

copilot[bot] fires on pull_requestopened / review_requested and leaves a review comment — exactly the events most slash_command workflows listen on. So any repo where Copilot Code Review is enabled and the workflow has bots: ["copilot[bot]"] will see automatic bot runs that race ahead of and block every subsequent manual invocation on that PR.

What the warning targets

The compile-time warning fires precisely when workflowData.Command (slash_command) and workflowData.Bots are both non-empty, which is the only configuration where this race can happen. Removing bots: keeps the bot's activity blocked by the default bot-denial check, leaving the concurrency slot free for explicit slash commands.

@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot the comment posted by the bot should also comply with the /slash-command string matching requirement

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 29, 2026

@copilot the comment posted by the bot should also comply with the /slash-command string matching requirement

Correct — the check_command_position step checks that the triggering comment body starts with /command-name, so a regular Copilot code review comment (which doesn't contain the slash command text) won't pass that check and won't trigger the workflow.

Updated the warning text in 6e7ba1b to accurately reflect this: the risk is only when a bot in bots: posts a comment that starts with the slash command text, at which point check_command_position passes, the bot occupies the concurrency slot, and a simultaneous manual invocation could be blocked.

New text:

Both slash_command and bots triggers are configured. If a bot listed in bots: posts a comment that starts with the slash command text (e.g., /command-name), it will trigger the workflow and occupy the concurrency slot, potentially blocking simultaneous manual invocations. To ensure the workflow only runs on explicit user commands, remove the 'bots:' field.

@pelikhan pelikhan merged commit ca70827 into main Apr 29, 2026
@pelikhan pelikhan deleted the copilot/add-compile-time-validation branch April 29, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature request: compile time validation for front matter trigger conditions

3 participants