Skip to content

perf(lockfile-stats): enforce single-script cached analysis to cut token/turn cost#31415

Merged
pelikhan merged 3 commits into
mainfrom
copilot/optimize-lockfile-token-usage
May 11, 2026
Merged

perf(lockfile-stats): enforce single-script cached analysis to cut token/turn cost#31415
pelikhan merged 3 commits into
mainfrom
copilot/optimize-lockfile-token-usage

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 11, 2026

✨ Enhancement

The lockfile-stats workflow prompt was driving high-cost runs (11–15M tokens, 51 bash turns) by repeatedly reading raw lock YAML across hundreds of files. This change reshapes the prompt into a bounded, cache-first analysis flow targeting ≤1M tokens and ≤5 bash turns using compact JSON summaries.

  • What does this improve?

    • Forces a single first-turn analyzer execution that parses all lockfiles once and emits one compact summary JSON.
    • Shifts reasoning from raw YAML to /tmp/gh-aw/agent/lockfile-stats-summary.json only.
    • Adds explicit run budgets and stop behavior (token/turn ceilings + noop fallback).
  • Why is this valuable?

    • Eliminates repeated raw lockfile reads from the model context.
    • Reduces turn churn and token amplification from iterative shell/YAML inspection.
    • Makes runtime behavior deterministic via bounded output and explicit truncation priorities.
  • Implementation approach:

    • Updated .github/workflows/lockfile-stats.md to define:
      • strict performance contract (effective tokens, bash-turn counting, budget stop condition),
      • cache-memory script reuse/versioning (lockfile_stats_v1.py, version bump rules),
      • compact summary constraints (size target + deterministic truncation order),
      • JSON-only downstream analysis and optional historical delta flow.
### 1) First turn: run one command that caches + executes the analyzer
...
4. Produces `/tmp/gh-aw/agent/lockfile-stats-summary.json` (compact, target ≤50KB; if larger, reduce examples before writing).
5. If the prompt version is bumped (for example to `lockfile_stats_v2.py`), do not reuse older script versions; use the version referenced in this prompt.

Copilot AI and others added 2 commits May 11, 2026 04:50
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…rompt

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Optimize lockfile stats to reduce token usage perf(lockfile-stats): enforce single-script cached analysis to cut token/turn cost May 11, 2026
Copilot AI requested a review from pelikhan May 11, 2026 04:56
@pelikhan pelikhan marked this pull request as ready for review May 11, 2026 05:01
Copilot AI review requested due to automatic review settings May 11, 2026 05:01
@pelikhan pelikhan merged commit 4f49ca9 into main May 11, 2026
1 check passed
@pelikhan pelikhan deleted the copilot/optimize-lockfile-token-usage branch May 11, 2026 05:01
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors the lockfile-stats workflow prompt to enforce a bounded, cache-first analysis flow that avoids repeated raw lockfile reads and constrains bash turns/tokens.

Changes:

  • Adds a strict performance contract (effective token + bash turn ceilings and a stop/noop fallback).
  • Requires a single first-turn analyzer script run that writes a compact JSON summary for downstream reasoning.
  • Defines cache-memory/script versioning and optional historical delta reporting from cached summaries.
Show a summary per file
File Description
.github/workflows/lockfile-stats.md Replaces the open-ended, multi-phase instructions with a single-script cached JSON analysis flow + explicit runtime budgets.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 1/1 changed files
  • Comments generated: 2

Analyze all .lock.yml files in the `.github/workflows/` directory to identify usage patterns, popular triggers, safe outputs, step sizes, and other interesting structural characteristics. Generate comprehensive statistical reports and publish findings to the "audits" discussion category.
- Target **effective tokens ≤ 1M** (the sum of input and output tokens as reported by the engine usage metrics for this workflow run).
- Use **≤ 5 bash turns total** (each bash command execution counts as one turn).
- If you are about to exceed either limit, call the `noop` safe-output action exposed by the runtime import (`{{#runtime-import shared/noop-reminder.md}}`) with a short reason and stop. Do not create a discussion in that case.
Comment on lines +72 to 75
### 2) Second turn: read summary JSON only

Analyze the collected data to generate insights:
Read only `/tmp/gh-aw/agent/lockfile-stats-summary.json` and derive insights from it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[q] perf(lockfile-stats): reduce token usage from 14M → ≤1M via single-script analysis

3 participants