Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
154 commits
Select commit Hold shift + click to select a range
4a4cd85
Add analytics domain: Sentry quota, MCP queries, instrumentation
MajorLift Jun 5, 2026
78d8342
Add performance skills: measurement + React/Redux anti-pattern reviews
MajorLift Jun 5, 2026
98e8963
Add platform domain plus testing and utility skills
MajorLift Jun 5, 2026
4e97c4b
Add 30D+ query fidelity guidance and performance-attribution skill
MajorLift Jun 22, 2026
b72ded1
Merge branch 'main' into add/analytics
MajorLift Jun 26, 2026
16df6f5
Merge branch 'main' into add/performance
MajorLift Jun 26, 2026
50d1c5c
Merge branch 'main' into add/platform-skills
MajorLift Jun 26, 2026
837ba90
sentry-quota: catch incidental instrumentation — memoized-selector fa…
MajorLift Jul 15, 2026
0061f40
CHANGELOG: note expanded sentry-quota detection (selector fan-out, in…
MajorLift Jul 15, 2026
8eb0970
Add `typescript-typing` skill for `any`-handling and type derivation
MajorLift Jul 16, 2026
9e20763
Ground the derive rule in a real `metamask-extension` #42583 countere…
MajorLift Jul 16, 2026
be1dc98
Split typescript-typing into a `typescript` domain: avoid-any, derive…
MajorLift Jul 16, 2026
a89095d
Add "why decompose first" rationale to `decompose-large-files`: bound…
MajorLift Jul 17, 2026
388dd98
Add `migration-context-cost` skill (fan-in/fan-out) + avoid-any secon…
MajorLift Jul 17, 2026
8483d91
Fix migration-context-cost inaccuracies, reframe decompose how-to aro…
MajorLift Jul 17, 2026
801758f
Reframe decompose how-to: identifying the boundaries is the key; extr…
MajorLift Jul 17, 2026
71bf451
feat(analytics): add grafana-tempo-queries skill
MajorLift Jul 27, 2026
5590e2d
feat(analytics): add sentry-grafana-cross-ref skill
MajorLift Jul 27, 2026
4d0aab2
feat(coding): add memory-leak-hunt skill
MajorLift Jul 30, 2026
cfd6a37
feat(security): add security domain with supply-chain-audit and lavam…
MajorLift Jul 30, 2026
c635aa8
feat(pr-workflow): add pr-validate and falsifying-test skills
MajorLift Jul 30, 2026
c3c3b21
Drop the CHANGELOG entry — it is for the CLI package, not skills
MajorLift Jul 30, 2026
712d17a
Drop the CHANGELOG entry — it is for the CLI package, not skills
MajorLift Jul 30, 2026
349efe5
Drop the CHANGELOG entry — it is for the CLI package, not skills
MajorLift Jul 30, 2026
6e6bbb7
Make knowledge/ the single source for the React perf pattern taxonomy
MajorLift Jul 30, 2026
f5aa6f9
Consolidate the shift-left performance work into one domain PR
MajorLift Jul 30, 2026
ed3d6eb
Restore the internal planning-ticket references
MajorLift Jul 30, 2026
26a7daf
Restore the MetaMask-planning link in the step-waiver item
MajorLift Jul 30, 2026
6459be4
Move `memory-leak-hunt` from `coding` to a new `stability` domain
MajorLift Jul 30, 2026
8dc5b86
Fold in `typescript-compiler-blindspots`, moved from the `testing` do…
MajorLift Jul 30, 2026
a66e72a
Move `falsifying-test` to the `testing` domain
MajorLift Jul 30, 2026
e1cd009
Move the benchmarking skills out to the performance PR
MajorLift Jul 30, 2026
8f6762f
Take in the benchmarking skills
MajorLift Jul 30, 2026
0f740f2
Drop `resilient-api-collection`
MajorLift Jul 30, 2026
45f78e1
Make the keepalive claim re-verifiable instead of asserted
MajorLift Jul 30, 2026
b576c7e
Point the extension overlays at `main`, not `develop`
MajorLift Jul 30, 2026
78a93b0
Add `D6` substitution A/B lane and sync `pr-validate` references
MajorLift Jul 30, 2026
3fd9bda
Add `privacy-egress-diligence` skill
MajorLift Jul 30, 2026
dfd0012
Add `agent-run-cost` skill
MajorLift Jul 30, 2026
36ea9c8
Move the AEP run procedure behind a reference
MajorLift Jul 30, 2026
2af1db1
Restore the full `pr-validate` description
MajorLift Jul 31, 2026
0b4bb26
Actually restore the `pr-validate` description
MajorLift Jul 31, 2026
95e6d8b
Add `race-condition-proof` skill for ordering guarantees under concur…
MajorLift Jul 31, 2026
7032425
Add `debug` orchestrator as the symptom-first sibling of `pr-validate`
MajorLift Jul 31, 2026
704128a
Rename `pr-validate` to `evidence`
MajorLift Jul 31, 2026
6e15178
Shorten and normalise skill names
MajorLift Jul 31, 2026
615c3ab
Rename `memory-leak-hunt` to `memory-leak`
MajorLift Jul 31, 2026
ab926b8
Rename `typescript-compiler-blindspots` to `compiler-blindspots`
MajorLift Jul 31, 2026
4d330e4
Rename `analytics-instrumentation` to `instrumentation`
MajorLift Jul 31, 2026
952d543
Ship `hooks/`, and check that a description names the installed command
MajorLift Jul 31, 2026
14196b2
Rename the antipattern skills from `-review` to `-scan`
MajorLift Jul 31, 2026
81dcf48
fix(race-condition-proof): name the evidence category instead of inde…
MajorLift Jul 31, 2026
843a49e
Name the evidence category in `react-render-proof` instead of indexin…
MajorLift Jul 31, 2026
d9c6231
Update `pr-validate` references to `evidence` in the security skills
MajorLift Jul 31, 2026
e7e9248
Repair unresolvable references in `evidence` and `falsifying-test`
MajorLift Jul 31, 2026
698d4f1
Drop references to a skill that does not exist from `agent-run-cost`
MajorLift Jul 31, 2026
e170b39
Rename `compiler-blindspots` to `tsc-blindspots`
MajorLift Jul 31, 2026
0da684b
Rename `sentry-grafana-cross-ref` to `sentry-grafana-correlation`
MajorLift Jul 31, 2026
6a93434
Rename `race-condition-proof` to `race-condition-repro`
MajorLift Jul 31, 2026
c18c751
Rename `react-render-proof` to `react-render-delta`
MajorLift Jul 31, 2026
1e927d5
Follow the engine renames in the catalog and `falsifying-test`
MajorLift Jul 31, 2026
affac8f
Update `pr-validate` references to `evidence` in `memory-leak`
MajorLift Jul 31, 2026
c579591
feat(cli): resolve hook registration at install, and on demand (#100)
MajorLift Jul 31, 2026
d04bd59
Restore the `sentry-quota` references
MajorLift Jul 31, 2026
14e670a
Add a lane index to the evidence catalog, and fix lane placement
MajorLift Jul 31, 2026
eceaf37
Add build-duration lanes `D7` and `G6`
MajorLift Jul 31, 2026
7b3eeaa
Widen the §3 grep to catch result functions returning fresh literals
MajorLift Aug 1, 2026
90d7082
Inline the publishing non-negotiables, which a real run ignored entirely
MajorLift Aug 1, 2026
d412caf
Add `falsify-probe.sh` — the runner that makes a lane reproducible
MajorLift Aug 1, 2026
559b4c0
Add `capture.sh` so the C9 and D3 analyses stop needing an operator t…
MajorLift Aug 1, 2026
3a21ee7
Detect named-subscription listeners, which the scanner could not see
MajorLift Aug 1, 2026
85a575c
Add `selector-recompute.sh` — lane C4 gets a runner
MajorLift Aug 1, 2026
94a7640
Add `tsc-substitution.sh` — a runner for the tsc-blindspots lane
MajorLift Aug 1, 2026
6e38d6d
Diff the error sets rather than requiring a clean baseline
MajorLift Aug 1, 2026
3f861c7
Add `attest-gate.sh` — eight mechanical checks before anything is pub…
MajorLift Aug 1, 2026
2c0f286
Standardise the provenance marker across every runner
MajorLift Aug 1, 2026
51d6f94
Add `render-count.sh` — the component half of lane C4
MajorLift Aug 1, 2026
73d4b17
Audit override scope, and escalate rather than rule on the critical g…
MajorLift Aug 1, 2026
e5b6ce1
Guard the runners against the failures that masquerade as findings
MajorLift Aug 1, 2026
477009f
Add `egress-delta.py` — what a diff newly exposes, and what it stoppe…
MajorLift Aug 1, 2026
d84f304
Give the orchestrator a runner registry, with limits and synthesis rules
MajorLift Aug 1, 2026
0d8935f
State the bar: float concerns, do not close them
MajorLift Aug 1, 2026
6bdffd1
Report what each measurement runner did not cover
MajorLift Aug 1, 2026
30a03b6
Require a validation run to float something for review
MajorLift Aug 1, 2026
d5c6a61
Stop the runners from cutting the part a reader needed
MajorLift Aug 1, 2026
a3f1590
Reference artifacts by name in the publishable block
MajorLift Aug 1, 2026
d55b324
Stop the gate failing runs whose environment is not the repo's
MajorLift Aug 1, 2026
4ae5373
Let the caller say where its tool puts the finding
MajorLift Aug 1, 2026
d32bdf7
Write for the reviewer, not for a user of this skill
MajorLift Aug 1, 2026
7dee1f9
Move the output template into the skill, where corrections can land
MajorLift Aug 1, 2026
f0eaf41
Separate the grants someone chose from the ones a new package brought
MajorLift Aug 1, 2026
2c72486
Require a finding, not a printout
MajorLift Aug 1, 2026
4132951
Check where the disclaimer is, not just that it is there
MajorLift Aug 1, 2026
e58d1c0
Stop a script's summary of itself passing as a capture
MajorLift Aug 1, 2026
3c86466
Ask for a medium, not for better text
MajorLift Aug 1, 2026
dbc2754
Move the measurement to CI, where the run URL is the capture
MajorLift Aug 2, 2026
bdf5f39
Pin the runner source, and say so when it is missing
MajorLift Aug 2, 2026
5c41bfb
Require the target repo, make install opt-out, drop the consumer-inst…
MajorLift Aug 2, 2026
8d1ec29
Make every runner say whether a reader can verify it
MajorLift Aug 2, 2026
56578ce
Cite what exists; capture what you ran
MajorLift Aug 2, 2026
f6d76e8
Fold the runner-workflow fixes back from where they were found
MajorLift Aug 2, 2026
0b874cf
Name the commit in the command line, so an A/B pair is not one line t…
MajorLift Aug 2, 2026
a64ddb8
Stop a broken substitution reading as a divergence
MajorLift Aug 2, 2026
bfdf6b4
Require a run to measure the PR's range, and to say where its reach ends
MajorLift Aug 2, 2026
faff610
Require an instrument to publish the effect it had, not the instructi…
MajorLift Aug 3, 2026
5086ffe
Add two PR-audit skills that read the description in opposite orders
MajorLift Aug 3, 2026
f046789
Add `silent-failure`, which tests detectability rather than correctness
MajorLift Aug 3, 2026
4fa12d7
Add five reasoning audits and two diagnostic skills
MajorLift Aug 3, 2026
46372af
Point `falsifying-test` at the harness, and separate it from its sibling
MajorLift Aug 3, 2026
1e7ef54
Rename `falsifying-test` to `red-on-base`
MajorLift Aug 3, 2026
50b52dc
Move `red-on-base` out; this PR is the instrument
MajorLift Aug 3, 2026
ca9111f
Take `red-on-base` from #84, where it was the odd one out
MajorLift Aug 3, 2026
c30e575
Say what this skill does not cover
MajorLift Aug 3, 2026
0558f9e
Point the engine table at the renamed skills
MajorLift Aug 3, 2026
594a1a7
Check where the run is going, not only what it says
MajorLift Aug 3, 2026
d50d94f
Add exit-tracing, a runtime declaration check, and a dead-module check
MajorLift Aug 3, 2026
539c58b
Add `lane-graphs`: route to a small executable graph, not a prose pro…
MajorLift Aug 3, 2026
e1dcfbe
Merge remote-tracking branch 'origin/jongsun/add/pr-validate-falsifyi…
MajorLift Aug 3, 2026
107fd78
Merge remote-tracking branch 'origin/jongsun/add/audit-skills' into t…
MajorLift Aug 3, 2026
36872f1
Merge remote-tracking branch 'origin/jongsun/add/lane-graphs' into tr…
MajorLift Aug 3, 2026
42270a0
Name the installed command in `red-on-base`'s description
MajorLift Aug 3, 2026
0dee42c
Bring the runner fixes back from the branch CI was actually running
MajorLift Aug 3, 2026
cae0c08
Check that a figure in the prose traces to an exhibit
MajorLift Aug 4, 2026
a5bfb98
Stop check 12 passing when it could not run
MajorLift Aug 4, 2026
8dbb50a
Merge remote-tracking branch 'origin/jongsun/add/memory-leak-hunt-ski…
MajorLift Aug 4, 2026
1ebbfbd
Merge remote-tracking branch 'origin/jongsun/add/race-condition-proof…
MajorLift Aug 4, 2026
fbbf96a
Merge remote-tracking branch 'fork/add/performance' into trial/combined
MajorLift Aug 4, 2026
b33927b
Merge remote-tracking branch 'fork/add/analytics-tracing-cross-ref' i…
MajorLift Aug 4, 2026
2ed9054
Merge security-domain, keeping both new CODEOWNERS lines
MajorLift Aug 4, 2026
783b6a2
Merge updated #84 and #106 into the combined trial tree
MajorLift Aug 4, 2026
549904d
Merge the remaining skill PRs into the combined trial tree
MajorLift Aug 4, 2026
a1ea24a
Add false precision as a divergence shape and the `IsAny` probe that …
MajorLift Aug 4, 2026
7da6ee1
Point references at the renamed `lavamoat-policy` skill
MajorLift Aug 4, 2026
1706f0f
Point references at the renamed `lavamoat-policy` skill
MajorLift Aug 4, 2026
af1a58f
Rename `lavamoat-policy-diligence` to `lavamoat-policy` and restore t…
MajorLift Aug 4, 2026
79d7478
Merge the lavamoat-policy rename and updated #69 into the trial tree
MajorLift Aug 4, 2026
609dd9c
End the diligence comment with an applicable override diff, and defin…
MajorLift Aug 4, 2026
38316ee
Give the diligence format a gate with `attest-gate.sh --diligence`
MajorLift Aug 4, 2026
1837b75
Merge the diligence gate mode and override-diff contract into the tri…
MajorLift Aug 4, 2026
76ecd0a
Remove private-repo and personal references from a public skill
MajorLift Aug 4, 2026
2d4b69d
Merge the private-reference scrub into the trial tree
MajorLift Aug 4, 2026
fbf8909
Take the artifact bucket and test fork out of the published text
MajorLift Aug 4, 2026
11bc1c0
Merge the bucket and test-fork scrub into the trial tree
MajorLift Aug 4, 2026
6634203
Restore what the privacy scrub broke: a working jq filter and bucket …
MajorLift Aug 4, 2026
08bff2d
Merge the scrub-repair commits into the trial tree
MajorLift Aug 4, 2026
d5fa3a5
Enforce the evidence rules where the model cannot route around them
MajorLift Aug 4, 2026
27be7ae
Add `attest` — the publish gate, ported from a personal command
MajorLift Aug 4, 2026
bbf746e
Require the removal test to exercise the capability, not the app
MajorLift Aug 4, 2026
34b935c
Name the installed command in `react-render-delta`'s description
MajorLift Aug 4, 2026
ea4f501
Name the installed command in `debug`'s description
MajorLift Aug 4, 2026
42d69a4
Name the installed command in `memory-leak`'s description
MajorLift Aug 4, 2026
43b5e92
Name the installed command in `race-condition-repro`'s description
MajorLift Aug 4, 2026
995f3c8
Name the installed command in `tsc-blindspots`'s description
MajorLift Aug 4, 2026
07bbe3d
Merge the attest port, the description fixes, and the lavamoat scenar…
MajorLift Aug 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@
/domains/performance/ @MetaMask/extension-platform @MetaMask/mobile-platform
/domains/perps/ @MetaMask/perps
/domains/pr-workflow/ @MetaMask/extension-platform @MetaMask/mobile-platform
<<<<<<< HEAD
/domains/stability/ @MetaMask/extension-platform @MetaMask/mobile-platform
||||||| 4063bf1
=======
/domains/security/ @MetaMask/extension-platform @MetaMask/mobile-platform
>>>>>>> origin/jongsun/add/security-domain
/domains/swaps/ @MetaMask/swaps-engineers
/domains/testing/ @MetaMask/qa
/domains/ui/ @MetaMask/design-system-engineers
16 changes: 16 additions & 0 deletions .github/scripts/lint-skill-entry.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,22 @@ export function lintSkill(skill) {
}
}

// The installer prefixes every emitted skill, so a description advertising `/<name>`
// names a command no operator exposes. The description IS the discovery surface, so a
// wrong trigger string is a selection failure, not a typo.
//
// The lookbehind keeps a scoped package (`@metamask/gator-cli`) or a path
// (`skills/gator-cli`) from being read as a slash command.
if (raw.description && raw.name) {
const bare = new RegExp(`(?<![\\w@/-])/${raw.name}\\b`, 'u');
const prefixed = new RegExp(`(?<![\\w@/-])/${INSTALLED_PREFIX}${raw.name}\\b`, 'u');
if (bare.test(raw.description) && !prefixed.test(raw.description)) {
errors.push(
`\`description\` advertises \`/${raw.name}\` but the installer emits \`${INSTALLED_PREFIX}${raw.name}\`; name the installed form`,
);
}
}

for (const section of RECOMMENDED_SECTIONS) {
// A trailing `\b` let `## When To Use Cases` satisfy `When To Use` — a different
// section. Anchoring to end-of-line fixes that but rejects `## Workflows` and
Expand Down
52 changes: 52 additions & 0 deletions bin/metamask-skills.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Usage:
metamask-skills describe <skill|domain/skill> [options]
metamask-skills sync [options]
metamask-skills postinstall [options]
metamask-skills hooks [options]
metamask-skills install [options]

Options:
Expand Down Expand Up @@ -799,6 +800,55 @@ function invokedDirectly() {
}
}


/**
* Print the Claude Code registration for every hook an installed skill ships.
*
* The installer copies `hooks/` like any other bundle directory, but a hook does nothing
* until it is registered in settings.json — and the path to register is absolute, so it
* differs per machine and per consumer repo and cannot be documented as a constant. This
* resolves it against the actual install.
*/
function printHookRegistration(args) {
const { target } = parseGlobalArgs(args);
const skillsDir = path.join(target, '.claude', 'skills');

let entries = [];
try {
for (const skill of readdirSync(skillsDir, { withFileTypes: true })) {
if (!skill.isDirectory()) continue;
const hooks = path.join(skillsDir, skill.name, 'hooks');
if (!dirExists(hooks)) continue;
for (const file of readdirSync(hooks)) {
if (file.endsWith('.py')) entries.push(path.join(hooks, file));
}
}
} catch {
warn(`no installed skills found under ${skillsDir}`);
return 1;
}

if (entries.length === 0) {
process.stdout.write('No installed skill ships a hook.\n');
return 0;
}

const commands = entries
.map((f) => ` { "type": "command", "command": "python3 ${f}" }`)
.join(',\n');

process.stdout.write(
`${entries.length} hook(s) installed. Copying a hook does not activate it — Claude Code\n` +
`runs one only once it is registered. Add this to ~/.claude/settings.json, or to\n` +
`${path.join(target, '.claude', 'settings.json')} to scope it to this repo:\n\n` +
' {\n "hooks": {\n "PreToolUse": [\n {\n "matcher": "Bash",\n "hooks": [\n' +
`${commands}\n` +
' ]\n }\n ]\n }\n }\n',
);
return 0;
}


if (invokedDirectly()) {
const [command, ...args] = process.argv.slice(2);
if (!command || command === '-h' || command === '--help') {
Expand All @@ -817,6 +867,8 @@ if (invokedDirectly()) {
exitCode = sync(args);
} else if (command === 'postinstall') {
exitCode = postinstall(args);
} else if (command === 'hooks') {
exitCode = printHookRegistration(args);
} else if (command === 'install') {
exitCode = install(args);
} else {
Expand Down
108 changes: 108 additions & 0 deletions domains/agentic/skills/agent-run-cost/skill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
---
name: agent-run-cost
maturity: experimental
description: >-
Estimate what an agentic workflow costs to run before it merges — fan-out × trigger
frequency × no kill-switch — and say so in figures rather than adjectives. Agent token
spend is invisible in a diff: a workflow that spawns one agent and one that spawns forty
are the same few lines, and the difference only appears on a bill nobody reads during
review. Produces a per-run and per-month estimate with its arithmetic shown, flags the
three amplifiers, and proposes the cheapest mitigation that preserves the intent. Use
when a PR adds or widens an agentic workflow, an AEP task class, a verification recipe,
or a schedule that runs agents unattended.
---

# Agent run cost

Scripted automation announces its cost in wall-clock time; agentic automation does not.
A fan-out of forty subagents and a single call are the same shape in a diff, and the
difference surfaces later, on a bill, attributed to nothing in particular.

This is the token-spend counterpart to `sentry-quota`, which guards span volume. Same
posture: operate on **code and PRs**, before the spend exists, and produce figures.

## When to use

- A PR adds or widens an agentic workflow, AEP task class, or verification recipe.
- A workflow gains fan-out — an agent per file, per finding, per test, per PR.
- Something agentic moves from opt-in to automatic (a CI trigger, a cron, a git hook).
- An ADR or design proposes agents for work a script already does — the estimate is the
argument, and its absence is usually the tell.

## Do not use when

- The workflow is developer-invoked, one agent, no loop — the ceiling is a person's patience.
- The change only narrows fan-out or adds a gate.

## The amplifier triad

Cost is not the per-agent price. It is the product of three things, any one of which can be
the whole problem:

| Amplifier | What it looks like | Effect |
|---|---|---|
| **Fan-out** | an agent per item — per file, per finding, per dimension, per round; nested `parallel` inside `pipeline`; a loop-until-dry with no ceiling | N× per run, and N is often data-dependent rather than fixed |
| **Trigger frequency** | runs on every push rather than on demand; a cron; a label that re-fires on each commit; a retry that respawns the fleet | turns a one-off into a rate |
| **No kill-switch** | no env var, feature flag, or budget cap; nothing to stop it mid-run; no way to disable without a revert | a runaway costs whatever it costs until someone merges a fix |

One alone is usually fine. **Fan-out × frequency with no kill-switch is the shape that
produces a surprise**, and it is worth naming explicitly in review when all three are present.

## Producing the estimate

Show the arithmetic. An estimate whose derivation is hidden is an adjective.

1. **Count agents per run.** Read the fan-out literally — how many items feed the widest
stage, and whether that number is bounded by the code or by the data. A `pipeline` over
changed files is unbounded by the code; `Array.from({length: 3})` is not.
2. **Estimate tokens per agent.** Prompt + the context it will read + its output. The context
dominates: an agent that reads three files is not an agent that greps a repo.
3. **Multiply, then apply frequency.** Per-run cost × runs per week. State the assumption
about run count — it is the number most likely to be wrong, and naming it lets a reviewer
correct it.
4. **State the worst case separately from the expected case.** The expected case is what it
costs on a normal PR; the worst case is what it costs on the PR that touches 400 files.
Budget conversations are about the second one.
5. **Compare against the alternative.** If a deterministic script covers the same ground, the
estimate belongs next to that script's cost. An agentic approach can still win — for
adversarial review, exploration, fuzzing, or anything with no fixed oracle — but the case
is made by the comparison, not by the capability.

Report the figures, the assumptions behind them, and the mitigation. **Do not render a
ship/no-ship verdict** — whether a cost is worth paying is a budget decision, and it belongs
to whoever owns the budget.

## Mitigation ladder

Cheapest first; stop at the rung that fits.

1. **Cap the fan-out.** A literal ceiling on items, with a `log()` of what was dropped —
silent truncation reads as full coverage and is worse than the cost.
2. **Narrow the trigger.** On-demand or label-gated instead of every push; on the changed
subset instead of the tree.
3. **Right-size the model per stage.** Mechanical stages rarely need the top tier; reserve it
for the judgement stages.
4. **Add a budget guard.** A token ceiling the workflow checks between stages, so it degrades
instead of running to completion at any price.
5. **Add a kill-switch.** An env var or flag that disables it without a revert. Cheap to add
up front and unavailable exactly when it is needed most.

## Common pitfalls

| Mistake | Correct approach |
|---|---|
| "It's just a few agents" | Count them. Data-dependent fan-out has no "just" |
| Estimating output tokens only | Context dominates — an agent that reads the repo costs more than one that answers at length |
| Quoting an average with no worst case | The worst case is the budget conversation |
| Treating a retry as free | A retried fleet is a second fleet |
| Assuming a concurrency cap bounds cost | It bounds *parallelism*, not total spend — queued agents still run |
| Adding a kill-switch after launch | It is needed during the incident it would have prevented |
| Comparing capability instead of cost | "Agents can do this" is not "agents should do this at this price" |

## Related

- `sentry-quota` — the same guard for span volume; `fan-out × ungated × no-kill-switch`.
- `evidence` — weighs AEP run cost when choosing an evidence lane, and tears the stack
down after; this skill is the review-side version for workflows others will run.
- [`MetaMask/decisions#173`](https://github.com/MetaMask/decisions/pull/173) — ADR-0058
review, where the missing token-cost estimate was raised as an open question.
41 changes: 41 additions & 0 deletions domains/analytics/knowledge/metrametrics-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
name: metrametrics-identity
domain: analytics
description: isOptIn:true unconditionally strips user identity in MetaMetricsController — always sends as anonymous ID
---

# MetaMetrics Identity Stripping

## The Mechanism

In `MetaMetricsController` (`app/scripts/controllers/metametrics-controller.ts`):

```typescript
if (excludeMetaMetricsId || (isOptIn && !metaMetricsIdOverride)) {
idType = 'anonymousId';
idValue = METAMETRICS_ANONYMOUS_ID; // 0x0000000000000000
}
```

When `isOptIn: true` with no `metaMetricsIdOverride`:
- The user's real `metaMetricsId` is discarded
- ALL such events share a single anonymous ID (`0x0000000000000000`) in Segment
- User-level attribution is completely lost

This is **unconditional** — it applies to fully opted-in users with valid IDs, not just anonymous users.

## Intended Use

The onboarding opt-in flow (`creation-successful.tsx`) — where the user hasn't committed to MetaMetrics yet and no `metaMetricsId` has been persisted. The event must fire regardless of opt-in state.

## The Misuse Pattern

Post-opt-in `trackEvent` calls with `{ isOptIn: true }` without `metaMetricsIdOverride`. Defeats the purpose of Segment user-level dimensions (account types, feature flags).

## Detection

```bash
grep -r "isOptIn: true" app/scripts/ ui/ --include="*.ts" --include="*.tsx"
```

Any occurrence outside `creation-successful.tsx` (or the onboarding flow) is suspect.
40 changes: 40 additions & 0 deletions domains/analytics/knowledge/segment-governance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
name: segment-governance
domain: analytics
description: Segment event governance via segment-schema is advisory — no CI enforcement prevents unregistered events from shipping
---

# Segment Event Governance

## Architecture

| Component | Location |
|-----------|----------|
| Tracking plan | `Consensys/segment-schema` → `tracking-plans/metamask-extension.yaml` |
| Event registry | `shared/constants/metametrics.ts` → `MetaMetricsEventName` enum (300+ entries) |
| Review process | `CONTRIBUTING.md` in segment-schema; Data Council review |
| Governance channel | `#metamask-metametrics`, `@consensys/data-council` |

## The Gap

There is **no CI enforcement** in the extension repo. A developer can:

1. Add entry to `MetaMetricsEventName` enum
2. Call `trackEvent` with it
3. Merge and ship to production

...without registering in segment-schema or going through Data Council review.

## Implications

- Schema drift between tracking plan and production events
- No property schema validation for unregistered events
- Billing impact goes unreviewed
- Data Council review is bypassable by omission

## Recommended Fix

CI check that:
1. Parses `MetaMetricsEventName` entries
2. Validates each against `tracking-plans/metamask-extension.yaml`
3. Fails build if event is missing from the plan
72 changes: 72 additions & 0 deletions domains/analytics/knowledge/span-sub-sampling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
name: span-sub-sampling
domain: analytics
description: Deterministic per-trace sub-sampling for high-frequency custom spans — global tracesSampleRate × span sub-rate, traceId-hash bucketed
---

# Span Sub-Sampling

Durable fix for a custom span that fans out and eats the span budget. Layer a per-trace sub-rate **under** the global `tracesSampleRate`, keyed on the trace id so every span in a trace is kept-or-dropped together. Source: [PR #39891](https://github.com/MetaMask/metamask-extension/pull/39891) (`shared/lib/wrapper-sampling.ts`).

## Rate Math

```
effective rate = global tracesSampleRate × span sub-rate
```

- Global `tracesSampleRate` is already small (extension prod: 0.75%).
- The sub-rate cuts the custom span on top: `0.75% × 1% = 0.0075%`.
- PR #39891 ships a sub-rate of 0.5% (`WRAPPER_SAMPLE_RATE = 0.005`) — a conservative pilot — and names 5% as the step-up once the denylist is confirmed effective in production.

Pick the sub-rate from how many sampled traces the metric needs to stay useful — not from the quota alone. Too low and the metric goes dark.

## Pattern

```ts
const WRAPPER_SAMPLE_RATE = 0.005;

// Deterministic: same answer for the same traceId, so all spans in a trace
// are kept or dropped together — clean waterfalls, no partial gaps.
export function shouldSampleWrappers(traceId: string | undefined): boolean {
if (!traceId || traceId.length < 8) {
return false;
}
const hashBucket = parseInt(traceId.slice(0, 8), 16) % 10000;
return hashBucket < WRAPPER_SAMPLE_RATE * 10000;
}
```

**Why deterministic, not `Math.random()` per call:** independent per-span sampling shreds a trace into partial waterfalls (some spans present, siblings missing) — useless for attribution. Hashing the trace id makes keep/drop a property of the whole trace.

## Gate Order (cheapest check first)

```ts
const traceId = sentryGetActiveSpan()?.spanContext().traceId;
if (!traceId || isReadOnlyAction(action) || !shouldSampleWrappers(traceId)) {
return doWorkWithoutSpan();
}
return trace({ name, op, data }, doWorkWithSpan);
```

1. No active trace → no span.
2. Denylist → skip noise (below).
3. Sub-sample miss → skip this trace's spans.

## Denylist: cut before you sample

Drop spans with no timing/attribution signal before sub-sampling. In PR #39891, read-only verbs are ~90% of `messenger.call` volume:

```ts
const READ_ONLY_VERB = /^(?:get|has|find|is|peek)(?:[A-Z]|$)/u;
```

Removing ~90% of volume before the sample multiplies headroom — a higher sub-rate then yields the same span budget, so kept traces are denser and more useful.

## Where the Gate Goes

- **Consumer (extension):** spans go through `trace()`. Gate at the call site, or for a whole span family inside the wrapper. `traceId` from `sentryGetActiveSpan()?.spanContext().traceId`.
- **Controller package (core):** controllers call an injected `trace` callback. Gate in the package's trace util or the callback so every consumer inherits the cap. Pull the trace id from the controller's tracing context, not a fresh Sentry import.

## Kill Switch

Ship every always-on span family with an env disable flag (PR #39891: `SENTRY_DISTRIBUTED_TRACING_DISABLED` returns the messenger un-wrapped). It turns a future emergency cut into a config flip instead of a cherry-pick.
Loading
Loading