feat(analytics): add Sentry, MetaMetrics, Segment, and Tempo skills - #76
feat(analytics): add Sentry, MetaMetrics, Segment, and Tempo skills#76MajorLift wants to merge 10 commits into
Conversation
sentry-mcp-queries: document data-fidelity loss on older releases for longer-range (30D+) queries (sample-rate drift, extrapolation hiding thin samples, retention downsampling) and percentile (p75+) sample-size/quality filtering; add table rows + pitfalls for stored-span-count and superseded-patch releases. performance-attribution: new skill for attributing release-over-release p75/p95 movements to code changes via black-box diff analysis, with an extension repo file (Trace Explorer queries, key transactions, highest-sample-patch version selection, 90d-vs-30d empirics, hot-path files, core-package changelog analysis, worked v13.11->v13.15 catalogue). analytics-instrumentation: cross-link volume-estimation caveats to the new fidelity guidance. CHANGELOG updated.
…n-out, latent trace? param, trace-arg PR-review scan, per-name budget
…cidental instrumentation)
CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (MetaMask#80, MetaMask#78, MetaMask#70, MetaMask#62, MetaMask#61 all touch zero changelog lines). Carrying one here bought nothing and was the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block. Restoring the file to its merge-base state makes the branch conflict-free without a merge commit.
MajorLift
changed the title
MajorLift
changed the title
Context budgetWhat this PR costs an agent, measured from an install rather than read from the diff. Three tiers, and only the first is unavoidable.
Frontmatter is the only tier paid unconditionally — every agent loads it on every run once the skill is installed, used or not, because it is what the agent reads to decide relevance. The 28 skills across the eleven open skill PRs sit at a median of ~1,716 tokens selected and ~1,860 with references followed. All are within the 1,536-character description budget. Selected is paid only when the agent picks the skill. + refs & knowledge is the ceiling if every bundled reference is then read; it is a worst case, not an expectation. Method
These figures are pinned to the commit above and drift on every push; #96 tracks automating them. |
The skill lives in the `analytics` domain, so the prefix repeated what the path and every discovery surface already show. Installed as `mms-instrumentation`.
"Cross-ref" describes the mechanism and reads as a generic pointer. The skill produces a correlation: one trace joined across two stores by `trace_id`, plus a reading of which absences are expected. The vendor names stay. They are load-bearing here rather than incidental — the skill is about the split between Sentry and Grafana Tempo specifically, and is not source-agnostic.
1 participant
Adds an
analyticsdomain for Sentry / MetaMetrics / Segment / Grafana Tempo observability.Supersedes #41, which is closed; this PR carries the whole domain.
Skills
sentry-quota— catch quota-risky span instrumentation in code and PRs: thefan-out × ungated × no-kill-switchproduct. Covers the incidental cases that are easy to miss — spans created inside memoized selectors (which fan out per row, forever), and the latent optionaltraceparameter that detonates the moment a caller wires it up.sentry-mcp-queries— query Sentry via MCP: error triage, volume estimation, release comparison, replay. Includes the 30D+ percentile-fidelity rules (filter superseded and low-sample releases; anchor on the highest-sample patch, never the.0).grafana-tempo-queries— query backend traces with TraceQL: search by service or span attribute, fetch by id, inspect the span tree, enumerate tag values. Documents the failure modes that produce confidently wrong conclusions: an expired session returns401with an empty body that parses as zero results — indistinguishable from "this data doesn't exist" — and a filter that silently fails to apply returns a full page of apparent matches. Hence a mandatory negative control before believing any negative result.sentry-grafana-correlation— join one trace across Sentry and Tempo by trace id, and diagnose why a half is missing. Built on the three rules that explain every missing half: the propagated sampled flag and the client's own retention are separate decisions (which is what manufactures orphaned backend spans), a not-sampled flag suppresses backend span creation rather than dropping it later, and an environment attribute decides whether backend spans reach Sentry at all.instrumentation— create/update Sentry spans, MetaMetrics events, and Segment events.performance-attribution— attribute release-over-release p75/p95 movement to specific code changes via black-box diff analysis, deliberately ignoring commit messages and claimed impact.Knowledge
metrametrics-identity(isOptIn:truesends events as anonymous) ·span-sub-sampling(deterministic per-trace sub-sampling for high-frequency spans) ·segment-governance(the event-governance gap with no CI enforcement).Notes
@metamask/skillsCLI package; skill-only PRs don't add one./domains/analytics/falls to the*catch-all. Tell me which team should own it and I'll add the line.grafana-tempo-queriesis the backend counterpart tosentry-mcp-queries;sentry-grafana-correlationdepends on both.Validation runs
Trial runs of this PR's skills against merged
metamask-extensionPRs nobody flagged. Every claim was re-verified against the real diff before posting. Clean results are included on purpose — a skill that only ever reports problems cannot be calibrated.sentry-quotasentry-quotaEach comment carries a trial-run disclaimer and links back here for feedback.