Skip to content

Reduce CLI Consistency Checker token usage via pre-agent help capture and prompt/frontmatter trimming#33363

Merged
pelikhan merged 4 commits into
mainfrom
copilot/copilot-token-optimizer-cli-consistency-checker
May 19, 2026
Merged

Reduce CLI Consistency Checker token usage via pre-agent help capture and prompt/frontmatter trimming#33363
pelikhan merged 4 commits into
mainfrom
copilot/copilot-token-optimizer-cli-consistency-checker

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 19, 2026

This workflow was spending most tokens on repetitive CLI help discovery despite being a read-only inspection task. This change shifts deterministic data collection to pre-agent execution and trims unnecessary workflow context to reduce per-run token cost.

  • Pre-agent deterministic help capture

    • Added pre-agent-steps to build gh-aw, collect --help output for top-level commands and discovered subcommands, and aggregate into /tmp/gh-aw/agent/all-help.txt.
    • Prompt now treats that file as the authoritative CLI dataset instead of re-running command discovery in-agent.
  • Frontmatter context reduction

    • Removed unused tools: cli-proxy, edit, web-fetch.
    • Tightened network policy from broad allowlist to allowed: [defaults].
  • Prompt streamlining

    • Removed unused delegated-agent sections.
    • Collapsed verbose execution/template instructions into concise analysis + reporting requirements while preserving severity breakdown and consolidated issue format expectations.
  • Script hardening

    • Improved command extraction robustness for help parsing.
    • Added warning output for failed per-command help collection.
    • Added safe glob handling and explicit empty-output guard before concatenation.
pre-agent-steps:
  - name: Build CLI and pre-collect help output
    run: |
      cd /home/runner/work/gh-aw/gh-aw
      make build
      # collect help output -> /tmp/gh-aw/agent/all-help.txt

Copilot AI and others added 2 commits May 19, 2026 17:09
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Optimize token usage for CLI Consistency Checker Reduce CLI Consistency Checker token usage via pre-agent help capture and prompt/frontmatter trimming May 19, 2026
Copilot AI requested a review from pelikhan May 19, 2026 17:13
@pelikhan pelikhan marked this pull request as ready for review May 19, 2026 18:33
Copilot AI review requested due to automatic review settings May 19, 2026 18:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces token usage for the CLI Consistency Checker workflow by moving deterministic CLI help discovery into pre-agent-steps, trimming workflow context/frontmatter, and simplifying the in-agent prompt to analyze a pre-collected help corpus instead of re-running CLI discovery.

Changes:

  • Add pre-agent-steps to build gh-aw and aggregate --help output into /tmp/gh-aw/agent/all-help.txt.
  • Tighten frontmatter context (remove unused tools, reduce network allowlist to allowed: [defaults]).
  • Streamline the markdown prompt to reference the pre-collected help file and simplify reporting instructions.
Show a summary per file
File Description
.github/workflows/cli-consistency-checker.md Adds pre-agent help capture + updates prompt/frontmatter to rely on the pre-collected dataset.
.github/workflows/cli-consistency-checker.lock.yml Regenerates the compiled workflow to include the new pre-agent step and updated network/tool configuration.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 3

strict: false
network:
allowed: [defaults, node, "api.github.com", "proxy.golang.org", "sum.golang.org"]
allowed: [defaults]
Comment on lines +32 to +36
/^[[:space:]]+[[:alnum:]_-]+([[:space:]]|$)/ {
cmd=$1
gsub(/:$/, "", cmd)
if (cmd != "" && cmd != "Commands") print cmd
}
}
'

./gh-aw --help > "${output_dir}/main.txt"
@pelikhan
Copy link
Copy Markdown
Collaborator

@copilot merge main and recompile

Copy link
Copy Markdown
Contributor Author

Copilot AI commented May 19, 2026

@copilot merge main and recompile

Merged main and recompiled. Addressed in commit c8e977f.

@pelikhan pelikhan merged commit c2b1d13 into main May 19, 2026
25 checks passed
@pelikhan pelikhan deleted the copilot/copilot-token-optimizer-cli-consistency-checker branch May 19, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[copilot-token-optimizer] CLI Consistency Checker - Token Optimization (2.5M-3.2M tokens/run savings)

3 participants