feat: add CLI flag consistency checker agentic workflow by Copilot · Pull Request #494 · github/gh-aw-firewall

Copilot · 2026-02-03T20:22:49Z

Adds a weekly agentic workflow that detects documentation drift for AWF CLI flags.

Changes

New workflow: .github/workflows/cli-flag-consistency-checker.md
- Weekly schedule (fuzzy timing) + manual dispatch
- Compares src/cli.ts against docs/usage.md, cli-reference.md, README.md, AGENTS.md
- Reports to Discussions via create-discussion safe-output
- Identifies: missing docs, outdated defaults, naming inconsistencies, stale examples

Report Structure

The generated Discussion includes:

Flag coverage matrix across all doc files
Prioritized issues (high/medium/low)
Actionable fix suggestions with file locations

Original prompt

Add an agentic workflow to check the inconsistency of awf CLI flags and offer suggestions. Run weekly and generate the report to Discussion.

Custom agent used: agentic-workflows
GitHub Agentic Workflows (gh-aw) - Create, debug, and upgrade AI-powered workflows with intelligent prompt routing

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: Mossaka <5447827+Mossaka@users.noreply.github.com>

Mossaka

Review: REQUEST_CHANGES

@copilot Please fix the following issues:

Issue 1: Non-existent File Reference

The workflow references AGENTS.md on lines 68-69:

# Check agent files
cat AGENTS.md
cat CLAUDE.md

AGENTS.md does not exist in this repository - only CLAUDE.md exists. Remove the AGENTS.md reference.

Issue 2: No CI Checks

No CI checks have run on this branch. Please ensure CI passes before this can be merged.

Suggestions (Optional)

Consider monthly schedule instead of weekly to avoid discussion spam if issues persist
Add logic to check for existing open Discussions with the same title prefix before creating duplicates

The overall feature is valuable - just needs these fixes.

Copilot · 2026-02-04T17:08:51Z

Review: REQUEST_CHANGES

@copilot Please fix the following issues:...

Issue 1: AGENTS.md does exist in this repository - verified with ls -la *.md showing it's a 25KB file containing CLI guidance for coding agents. The reference is correct and intentional.

Issue 2: Build and lint pass locally (npm run build and npm run lint both exit 0). CI should be triggered automatically on push - if it hasn't run yet, it may be a GitHub Actions scheduling issue rather than a code problem.

…orkflow-cli-check

github-actions · 2026-02-04T19:59:46Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-02-04T19:59:50Z

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

github-actions · 2026-02-04T20:00:18Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	82.44%	82.44%	➡️ +0.00%
Statements	82.47%	82.47%	➡️ +0.00%
Functions	81.77%	81.77%	➡️ +0.00%
Branches	75.59%	75.59%	➡️ +0.00%

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-02-04T20:02:22Z

Smoke Test Results - Claude Engine

Last 2 merged PRs:

chore: upgrade agentic workflows to gh-aw v0.42.0
fix: replace deprecated log_access directive with Squid 5+ syntax

✅ GitHub MCP - PASS
✅ Playwright - PASS (title: "GitHub · Change is constant. GitHub keeps you ahead. · GitHub")
✅ File Write - PASS
✅ Bash Tool - PASS

Overall Status: PASS

AI generated by Smoke Claude

github-actions · 2026-02-04T20:02:40Z

Smoke Test Results (Run #21686296501)

Last 2 Merged PRs:

chore: upgrade agentic workflows to gh-aw v0.42.0 #512: chore: upgrade agentic workflows to gh-aw v0.42.0
fix: replace deprecated log_access directive with Squid 5+ syntax #506: fix: replace deprecated log_access directive with Squid 5+ syntax

Tests:

✅ GitHub MCP (reviewed PRs)
✅ Playwright (title: "GitHub · Change is constant. GitHub keeps you ahead. · GitHub")
✅ File creation (/tmp/gh-aw/agent/smoke-test-copilot-21686296501.txt)
✅ Bash verification (file read successful)

Status: PASS 🎉

cc @Mossaka @Copilot

AI generated by Smoke Copilot

Copilot

Pull request overview

This PR adds a new agentic workflow that performs weekly automated checks to identify documentation drift for AWF CLI flags. The workflow compares the CLI implementation against multiple documentation files and reports inconsistencies via GitHub Discussions.

Changes:

Added .github/workflows/cli-flag-consistency-checker.md defining the workflow's instructions for the AI agent
Added .github/workflows/cli-flag-consistency-checker.lock.yml as the compiled GitHub Actions workflow configuration

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`.github/workflows/cli-flag-consistency-checker.md`	Defines the AI agent's instructions for analyzing CLI flags across `src/cli.ts` and documentation files (`docs/usage.md`, `cli-reference.md`, `README.md`, `AGENTS.md`, `CLAUDE.md`), specifying the analysis steps, output format, and edge cases to handle
`.github/workflows/cli-flag-consistency-checker.lock.yml`	Auto-generated GitHub Actions workflow configuration compiled from the markdown file, scheduled to run weekly on Wednesdays, with proper permissions, tooling setup, and safe-output configuration for creating discussions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-04T20:17:43Z

.github/workflows/cli-flag-consistency-checker.md

+2. **Usage documentation** (`docs/usage.md`) - User-facing documentation
+3. **CLI reference** (`docs-site/src/content/docs/reference/cli-reference.md`) - Detailed reference docs
+4. **README.md** - Quick start and overview documentation
+5. **AGENTS.md** and `CLAUDE.md` - Agent instruction files


Inconsistent markdown formatting for file names. AGENTS.md uses bold (AGENTS.md) while CLAUDE.md uses code backticks (CLAUDE.md). Both should use the same formatting style for consistency. Based on the other file references in this list (lines 25-28), code backticks should be used for both file names.

Suggested change

5. **AGENTS.md** and `CLAUDE.md` - Agent instruction files

5. `AGENTS.md` and `CLAUDE.md` - Agent instruction files

Copilot · 2026-02-04T20:17:43Z

.github/workflows/cli-flag-consistency-checker.md

+
+### 4. Check for Subcommand Consistency
+
+The CLI has subcommands (`logs`, `logs stats`, `logs summary`). Verify:


The description of subcommands is potentially misleading. The CLI has one command "logs" with two subcommands "stats" and "summary". The current wording "The CLI has subcommands (logs, logs stats, logs summary)" could be misinterpreted to suggest three separate commands. Consider clarifying this as "The CLI has a logs command with subcommands stats and summary" to accurately reflect the command hierarchy.

Suggested change

The CLI has subcommands (`logs`, `logs stats`, `logs summary`). Verify:

The CLI has a `logs` command with subcommands `stats` and `summary`. Verify:

Initial plan

abcfc47

Copilot AI assigned Copilot and Mossaka Feb 3, 2026

Copilot started work on behalf of Mossaka February 3, 2026 20:23 View session

feat: add CLI flag consistency checker agentic workflow

f00666b

Co-authored-by: Mossaka <5447827+Mossaka@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add agentic workflow to check CLI flag inconsistencies~~ feat: add CLI flag consistency checker agentic workflow Feb 3, 2026

Copilot AI requested a review from Mossaka February 3, 2026 20:27

Copilot finished work on behalf of Mossaka February 3, 2026 20:27

Mossaka requested changes Feb 4, 2026

View reviewed changes

Copilot started work on behalf of Mossaka February 4, 2026 17:07 View session

Copilot AI requested a review from Mossaka February 4, 2026 17:09

Copilot finished work on behalf of Mossaka February 4, 2026 17:09

Merge remote-tracking branch 'origin/main' into copilot/add-agentic-w…

0c5a716

…orkflow-cli-check

github-actions bot added the smoke-claude label Feb 4, 2026

github-actions bot added the smoke-copilot label Feb 4, 2026

Mossaka marked this pull request as ready for review February 4, 2026 20:14

Copilot AI review requested due to automatic review settings February 4, 2026 20:14

Mossaka merged commit 60814a5 into main Feb 4, 2026
37 checks passed

Mossaka deleted the copilot/add-agentic-workflow-cli-check branch February 4, 2026 20:14

Copilot started reviewing on behalf of Mossaka February 4, 2026 20:14 View session

Copilot AI reviewed Feb 4, 2026

View reviewed changes

github-actions bot mentioned this pull request Feb 4, 2026

fix: restructure smoke-chroot workflow to avoid Docker-in-Docker #508

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add CLI flag consistency checker agentic workflow#494

feat: add CLI flag consistency checker agentic workflow#494
Mossaka merged 3 commits intomainfrom
copilot/add-agentic-workflow-cli-check

Copilot AI commented Feb 3, 2026 •

edited

Loading

Uh oh!

Mossaka left a comment

Uh oh!

Copilot AI commented Feb 4, 2026

Review: REQUEST_CHANGES

Uh oh!

github-actions bot commented Feb 4, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 4, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 4, 2026

Uh oh!

Copilot AI Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	5. AGENTS.md and `CLAUDE.md` - Agent instruction files
	5. `AGENTS.md` and `CLAUDE.md` - Agent instruction files


		### 4. Check for Subcommand Consistency

		The CLI has subcommands (`logs`, `logs stats`, `logs summary`). Verify:

	The CLI has subcommands (`logs`, `logs stats`, `logs summary`). Verify:
	The CLI has a `logs` command with subcommands `stats` and `summary`. Verify:

Conversation

Copilot AI commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Report Structure

Uh oh!

Mossaka left a comment

Choose a reason for hiding this comment

Review: REQUEST_CHANGES

Issue 1: Non-existent File Reference

Issue 2: No CI Checks

Suggestions (Optional)

Uh oh!

Copilot AI commented Feb 4, 2026

Review: REQUEST_CHANGES

Uh oh!

github-actions bot commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 4, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

github-actions bot commented Feb 4, 2026

Smoke Test Results (Run #21686296501)

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Feb 3, 2026 •

edited

Loading

github-actions bot commented Feb 4, 2026 •

edited

Loading

github-actions bot commented Feb 4, 2026 •

edited

Loading