Is Strict Qualified Usage the right default for public metrics? #1
Unanswered
roy-tong
asked this question in
Metric Semantics
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Our measurement policy treats usage as UNKNOWN by default: an observation only upgrades to Qualified Usage when evidence supports it (see product/ and QUALITY.md).
Why this bias: CI, health checks, benchmarks and manual curl calls all look identical at the call boundary. Without the strict default, every public metric silently absorbs noise - and then the metric is worthless for the thing we care about: proving real agent usage.
The trade-off: strict defaults undercount at first. A tool genuinely used by agents will show Qualified Usage of 0% until identity evidence is collected. That is a feature (no inflated claims), but it makes early numbers look bad.
Questions for the community:
I lean toward: publish raw and qualified side by side, but gate claims (badges, comparison, billing) on Qualified. What breaks in that model?
All reactions