feat(agentic): add agent-run-cost skill - #91
Conversation
Scripted automation announces its cost in wall-clock time; agentic automation does not. A fan-out of forty subagents and a single call are the same shape in a diff, and the difference surfaces later on a bill attributed to nothing in particular. The token-spend counterpart to `sentry-quota`, and deliberately the same posture: operate on code and PRs before the spend exists, and produce figures rather than adjectives. Names the amplifier triad — fan-out × trigger frequency × no-kill-switch — where one alone is usually fine and all three together is the shape that produces a surprise. Requires the arithmetic be shown, and the worst case stated separately from the expected case, since the budget conversation is about the PR that touches 400 files rather than the normal one. Renders no ship verdict: whether a cost is worth paying belongs to whoever owns the budget. Raised as an open question during ADR-0058 review (MetaMask/decisions#173), where agent token consumption had no estimate.
Context budgetWhat this PR costs an agent, measured from an install rather than read from the diff. Three tiers, and only the first is unavoidable.
Frontmatter is the only tier paid unconditionally — every agent loads it on every run once the skill is installed, used or not, because it is what the agent reads to decide relevance. The 28 skills across the eleven open skill PRs sit at a median of ~1,716 tokens selected and ~1,860 with references followed. All are within the 1,536-character description budget. Selected is paid only when the agent picks the skill. + refs & knowledge is the ceiling if every bundled reference is then read; it is a worst case, not an expectation. Method
These figures are pinned to the commit above and drift on every push; #96 tracks automating them. |
`sentry-quota` was named twice — once as the framing analogy, once in `## Related` — but it exists on no branch of this repo. The analogy is kept as a description of the guard; the `## Related` entry is removed. Also updates `pr-validate` to its current name, `evidence`.
Removed in the previous commit on the claim that no such skill existed. It does — `sentry-quota` ships in the analytics domain. The check that "proved" its absence passed a bare PR number where a ref was required, so every lookup errored into a silenced zero and the skill appeared to exist nowhere. A skill defined in a concurrent pull request is a forward reference that resolves on merge, which is why the reference linter treats it as a warning rather than an error.
1 participant
Summary
agent-run-costtoagentic— estimate what an agentic workflow costs to run, before it merges.The problem it addresses
Scripted automation announces its cost in wall-clock time. Agentic automation does not — a fan-out of forty subagents and a single call are the same few lines in a diff, and the difference appears later on a bill attributed to nothing in particular.
Shape
The token-spend counterpart to
sentry-quota, deliberately the same posture: operate on code and PRs, before the spend exists, and produce figures rather than adjectives.The amplifier triad — cost is not the per-agent price, it's the product:
One alone is usually fine. All three together is the shape that produces a surprise.
The estimate must show its arithmetic — agents per run, tokens per agent (context dominates output), × frequency, with the run-count assumption stated because it's the number most likely to be wrong. Worst case is reported separately from expected: the budget conversation is about the PR that touches 400 files, not the normal one.
It also asks for the comparison against a deterministic alternative where one exists. An agentic approach can still win — adversarial review, exploration, fuzzing, anything without a fixed oracle — but the case is made by the comparison, not by the capability.
No ship/no-ship verdict. Whether a cost is worth paying is a budget decision belonging to whoever owns the budget.
Notes
agentic, which README already documents as "experimental recipe workflows and runtime proof tools" but which currently ships no skills onmain.evidence, which weighs AEP run cost when picking an evidence lane and tears the stack down afterwards. This is the review-side version, for workflows other people will run.CHANGELOG.mdentry: skill-only, no CLI change.Validation runs
Trial runs of this PR's skills against merged
metamask-extensionPRs nobody flagged. Every claim was re-verified against the real diff before posting. Clean results are included on purpose — a skill that only ever reports problems cannot be calibrated.agent-run-costEach comment carries a trial-run disclaimer and links back here for feedback.