Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 39 additions & 39 deletions .github/workflows/agentic-token-audit.lock.yml

Large diffs are not rendered by default.

26 changes: 15 additions & 11 deletions .github/workflows/agentic-token-audit.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Daily audit of AI credit spend across all agentic workflows with historical trend tracking
description: Daily audit of AI Credit (AIC) usage across all agentic workflows with historical trend tracking
on:
schedule:
- cron: "daily around 12:00 on weekdays"
Expand All @@ -9,6 +9,10 @@ permissions:
actions: read
issues: read
pull-requests: read
network:
allowed:
- defaults
- python
tracker-id: agentic-token-audit
safe-outputs:
create-issue:
Expand All @@ -25,12 +29,12 @@ tools:
- "*"
repo-memory:
branch-name: "memory/token-audit"
description: "Historical daily workflow token usage snapshots (shared with agentic-token-optimizer)"
description: "Historical daily workflow AIC snapshots (shared with agentic-token-optimizer)"
file-glob: ["*.json", "*.jsonl", "*.csv", "*.md"]
max-file-size: 102400
max-patch-size: 51200
steps:
- name: Setup Python runtime
- name: Setup Python
uses: actions/setup-python@v6.2.0
with:
python-version: "3.12"
Expand Down Expand Up @@ -71,15 +75,15 @@ steps:
timeout-minutes: 25
---

# Daily Agentic Workflow AI Credit Spend Audit
# Daily Agentic Workflow AIC Usage Audit

You are the Agentic Workflow Auditor — a workflow that tracks daily AI credit spend and token consumption across all agentic workflows in this repository and maintains a historical record for trend analysis.
You are the Agentic Workflow Auditor — a workflow that tracks daily AI Credit (AIC) spend and token consumption across all agentic workflows in this repository and maintains a historical record for trend analysis.

## Mission

1. Parse the pre-downloaded agentic workflow logs and compute per-workflow AI credit spend and token usage metrics.
1. Parse the pre-downloaded agentic workflow logs and compute per-workflow AIC spend and token usage metrics.
2. Persist today's snapshot to repo-memory so the optimizer (and future runs of this audit) can read historical data.
3. Publish a concise audit issue summarizing today's AI credit spend and trend highlights.
3. Publish a concise audit issue summarizing today's AIC spend and trend highlights.

## Data Sources

Expand All @@ -103,9 +107,9 @@ Each element of `.runs` is a `RunData` object with (among others):
|---|---|---|
| `workflow_name` | string | Human-readable name |
| `workflow_path` | string | `.github/workflows/....lock.yml` |
| `ai_credits` | float | AI credits consumed (primary billing metric; 1 AIC = $0.01 USD) |
| `aic` | float | AI Credits (AIC) consumed (primary billing metric; 1 AIC = $0.01 USD) |
| `token_usage` | int | Total tokens (`omitempty` — treat missing/null as 0) |
| `effective_tokens` | int | Legacy normalized token metric (deprecated; use `ai_credits` for billing) |
| `effective_tokens` | int | Legacy normalized token metric (deprecated; use `aic` for billing) |
| `action_minutes` | float | Billable GitHub Actions minutes |
| `turns` | int | Number of agent turns |
| `duration` | string | Human-readable duration |
Expand Down Expand Up @@ -163,7 +167,7 @@ Write a Python script to `/tmp/gh-aw/token-audit/process_audit.py` and run it. T
}
```

Handle null/missing `ai_credits` and `token_usage` by treating them as 0.
Handle null/missing `aic` and `token_usage` by treating them as 0.

## Phase 2 — Persist Snapshot to Repo-Memory

Expand Down Expand Up @@ -258,7 +262,7 @@ Summarize AI credit, token, and active-workflow changes from `rolling-summary.js

## Important Notes

- Use `// 0` (null coalescing) in jq and `.get(field, 0)` in Python for nullable numeric fields (`ai_credits`, `token_usage`).
- Use `// 0` (null coalescing) in jq and `.get(field, 0)` in Python for nullable numeric fields (`aic`, `token_usage`).
- Distinguish between these two cases in the issue:
- the raw `.runs` array is empty
- the raw `.runs` array is non-empty but none of the runs are `status == "completed"`
Expand Down
60 changes: 30 additions & 30 deletions .github/workflows/agentic-token-optimizer.lock.yml

Large diffs are not rendered by default.

30 changes: 15 additions & 15 deletions .github/workflows/agentic-token-optimizer.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Daily optimizer that identifies high-AI-credit-spend agentic workflows, audits their runs, and recommends efficiency improvements including inline sub-agent refactors when warranted
description: Daily optimizer that identifies a high-AIC agentic workflow, audits its runs, and recommends efficiency improvements including inline sub-agent refactors when warranted
on:
schedule:
- cron: "daily around 14:00 on weekdays"
Expand All @@ -18,7 +18,7 @@ tools:
- "*"
repo-memory:
branch-name: "memory/token-audit"
description: "Historical daily workflow token usage snapshots (shared with agentic-token-audit)"
description: "Historical daily workflow AIC snapshots (shared with agentic-token-audit)"
file-glob: ["*.json", "*.jsonl", "*.csv", "*.md"]
max-file-size: 102400
max-patch-size: 51200
Expand Down Expand Up @@ -58,7 +58,7 @@ steps:
echo '{"runs":[],"summary":{}}' > /tmp/gh-aw/token-audit/all-runs.json
fi

- name: Aggregate top workflows by token usage
- name: Aggregate top workflows by AIC usage
run: |
set -euo pipefail
mkdir -p /tmp/gh-aw/token-audit
Expand All @@ -71,7 +71,7 @@ steps:
| select(.status == "completed")
| {
workflow_name: .workflow_name,
ai_credits: (.ai_credits // 0),
ai_credits: (.aic // 0),
tokens: (.token_usage // 0),
turns: (.turns // 0),
action_minutes: (.action_minutes // 0)
Expand Down Expand Up @@ -110,14 +110,14 @@ steps:
fi
---

# Agentic Workflow AI Credit Spend Optimizer
# Agentic Workflow AIC Usage Optimizer

You are the Agentic Workflow Optimizer. Pick one high-AI-credit-spend workflow, audit recent runs, and create a conservative optimization issue with measurable improvements. Your recommendations may include prompt, tool, reliability, setup-prefix, and inline sub-agent improvements when the evidence supports them.
You are the Agentic Workflow Optimizer. Pick one high-AIC workflow, audit recent runs, and create a conservative optimization issue with measurable improvements. Your recommendations may include prompt, tool, reliability, setup-prefix, and inline sub-agent improvements when the evidence supports them.

## Objectives

1. Select one workflow using repo-memory and pre-aggregated data.
2. Analyze tokens, turns, errors, tool usage patterns, and prompt structure across multiple runs.
2. Analyze AIC, tokens, turns, errors, tool usage patterns, and prompt structure across multiple runs.
3. Propose safe, high-impact optimizations with evidence, including inline sub-agent refactors only when they are a clear fit.
4. Publish one issue and update optimization history.

Expand Down Expand Up @@ -151,11 +151,11 @@ Prefer `--jq` on `gh api` calls over a separate `| jq` step when the filter is s
## Data Inputs

- `/tmp/gh-aw/token-audit/all-runs.json`: full 7-day run data (`gh aw logs --json`).
- `/tmp/gh-aw/token-audit/top-workflows.json`: pre-aggregated top 10 workflows by total AI credits.
- `/tmp/gh-aw/token-audit/top-workflows.json`: pre-aggregated top 10 workflows by total AIC.
- `/tmp/gh-aw/repo-memory/default/YYYY-MM-DD.json`: daily audit snapshots.
- `/tmp/gh-aw/repo-memory/default/optimization-log.json`: prior optimizations (if present).

Treat missing numeric fields (`ai_credits`, `token_usage`, `turns`, `action_minutes`) as `0`.
Treat missing numeric fields (`aic`, `token_usage`, `turns`, `action_minutes`) as `0`.

## Phase 1 — Select Target

Expand All @@ -168,7 +168,7 @@ Treat missing numeric fields (`ai_credits`, `token_usage`, `turns`, `action_minu
Then collect run-level data for the selected workflow:

- run count
- total and average AI credits
- total and average AIC
- total and average tokens
- total and average turns
- conclusions/error patterns
Expand All @@ -180,8 +180,8 @@ Use this compact analysis matrix:
| Area | Required checks | Output |
|---|---|---|
| Tool usage | Compare configured tools from workflow source vs observed usage across multiple runs | Keep / Consider removing / Remove |
| AI credit spend | Evaluate AI credits, token totals, cache efficiency, turns | Top spend drivers |
| Reliability | Repeated errors, warnings, retries, missing tools | AI credit waste from failures |
| AI credit spend | Evaluate AIC, token totals, cache efficiency, turns | Top spend drivers |
| Reliability | Repeated errors, warnings, retries, missing tools | AIC waste from failures |
| Prompt efficiency | Redundant instructions, overlong sections, avoidable iteration | Prompt reduction opportunities |
| Structural optimization | Repeated setup/tool-call prefixes and sections suited for inline sub-agents | Extract setup / Add sub-agent / Keep in main agent |

Expand Down Expand Up @@ -283,10 +283,10 @@ Create one issue with:

- **Target workflow + reason selected**
- **Analysis period + runs analyzed**
- **Spend profile table** (total AI credits, avg AI credits/run, total tokens, avg turns/run, cache efficiency)
- **Spend profile table** (total AIC, avg AIC/run, total tokens, avg turns/run, cache efficiency)
- **Ranked recommendations** with:
- title
- estimated AI credit savings per run
- estimated AIC savings per run
- concrete action
- evidence from observed runs
- **Optional structural optimizations** for shared setup prefixes and inline sub-agents when supported by the analysis
Expand All @@ -304,7 +304,7 @@ Create one issue with:

Append one entry to `/tmp/gh-aw/repo-memory/default/optimization-log.json`:

`{"date":"YYYY-MM-DD","workflow_name":"...","total_ai_credits_analyzed":F,"total_tokens_analyzed":N,"runs_audited":N,"recommendations_count":N,"subagent_candidates":N,"estimated_ai_credit_savings_per_run":F}`
`{"date":"YYYY-MM-DD","workflow_name":"...","total_ai_credits_analyzed":F,"total_tokens_analyzed":N,"runs_audited":N,"recommendations_count":N,"subagent_candidates":N,"estimated_aic_savings_per_run":F}`

Use `subagent_candidates` for the count of inline sub-agent candidates you actually recommend in the issue body.

Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,13 @@ Changelog entries should describe what changed for users of these workflows, not
- Omit vague filler like "various fixes" when the actual effect can be named.
- Do not include contributor handles, reviewer names, or other attribution in release entries.

## [0.3.1] - 2026-06-09

### Fixed

- Kept the published audit and optimizer workflows compatible with the current upstream `gh-aw` AIC/runtime log schema updates without changing the existing snapshot and summary fields that downstream reporting depends on.
- Restored the audit workflow's explicit Python network allowlist so chart dependency installation remains supported in the packaged workflow.

## [0.3.0] - 2026-06-08

### Changed
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ It is built for platform engineers, developer productivity teams, and repository
## Key Features

- **Clear operational visibility** with a daily audit that captures AI credit spend, usage trends, and workflow-level hotspots.
- **Actionable optimization guidance** that identifies high-AI-credit-spend workflows and proposes safe, conservative improvements.
- **Actionable optimization guidance** that identifies high-AIC workflows and proposes safe, conservative improvements.
- **Faster efficiency improvements** by helping teams find waste before it becomes recurring operational overhead.
- **Built for real GitHub workflows** using GitHub Agentic Workflows, so installation and adoption fit naturally into existing repositories.
- **Useful historical context** through shared snapshots that support trend analysis instead of one-off debugging.
Expand Down Expand Up @@ -50,8 +50,8 @@ Included workflows:

| Workflow | What it does |
| ----- | --- |
| [`Daily Agentic Workflow AI Credit Spend Audit`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-audit.md?plain=1) | Collects recent agentic workflow usage and creates a daily AI credit spend snapshot. |
| [`Agentic Workflow AI Credit Spend Optimizer`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-optimizer.md?plain=1) | Analyzes high-AI-credit-spend workflows and proposes conservative efficiency changes, including inline sub-agent opportunities when they are a strong fit. |
| [`Daily Agentic Workflow AIC Usage Audit`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-audit.md?plain=1) | Collects recent agentic workflow usage and creates a daily AIC spend snapshot. |
| [`Agentic Workflow AIC Usage Optimizer`](https://github.com/githubnext/agentic-ops/blob/main/workflows/agentic-token-optimizer.md?plain=1) | Analyzes high-AIC workflows and proposes conservative efficiency changes, including inline sub-agent opportunities when they are a strong fit. |

## License

Expand Down
Loading