perf(tools): advertise the front door, not the whole catalogue - #97
Merged
Conversation
getnable
force-pushed
the
feat/tool-tiers
branch
from
August 8, 2026 23:32
7fef4ca to
496d502
Compare
Filtering by connected family narrowed the surface but left it flat. Measured on
main: 198 advertised tools, ~46,700 tokens riding every message, and 59 of those
tools with cost/spend/bill in the name. get_cost_summary, get_costs_by_service,
get_top_cost_drivers, get_cost_trends, get_service_cost and
get_total_spend_all_sources are not six capabilities. They are six phrasings, and
the model was left guessing between them instead of choosing between things it
can actually do.
Tools now carry a tier alongside their family, and only tier 1 is advertised.
Measured on an AWS-connected machine: 147 tools and 35,705 tokens becomes 17
tools and 5,584. An 84% cut, 30,121 tokens back on every message, with no
capability removed.
Nothing is unregistered. The MCP call path resolves against the registry, not
the advertised list, so a hidden tool runs the moment something names it. That
is only true in practice if something does name it, so:
- get_cost_summary returns next_tools: the drill-down tools that expand THIS
answer, picked from its own top services. A generic menu would just be the
flat surface moved into the response body, so Textract in the top spenders
names get_textract_costs and an RDS-heavy bill does not.
- The server instructions say the listed tools are the front door, and point
at what_can_nable_do(detailed=True) for the full map, with an explicit
instruction never to report a capability missing because its tool is not
listed.
- FINOPS_FLAT_TOOLS=1 restores the old surface. Narrower than FINOPS_ALL_TOOLS
on purpose: escaping the funnel should not also force an AWS-only user to
take the Azure tools.
Unlisted means tier 2, and that default fails SHUT rather than open. For
families an unmapped tool is still advertised because a missing capability is
worse than a noisy one; for tiers the reverse holds, since a tool quietly
joining the front door is what costs every user tokens on every message. The
guard is therefore not completeness but a budget: the front door has a token
ceiling and a tool count ceiling, and CI fails when either creeps.
Two tests earned their place immediately. TIER3 and the drill-down map both
named get_kendra_costs, get_documentdb_costs and get_marketplace_costs, which
live in _EXTRA_TOOLS and are not registered at all unless FINOPS_ALL_TOOLS is
set. A route to a tool the registry does not hold would have had the model
reporting a capability the user cannot have.
Five family tests in test_tool_surface.py were re-pointed rather than deleted:
they assert family behaviour, so they now run with FINOPS_FLAT_TOOLS=1 and test
the family layer alone. Without that they would have passed trivially, with the
tier gate hiding the tool before the family gate was ever consulted.
Full suite green at 2213.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No new secret. The tier work added lines above an existing baseline entry in cost_queries.py, so every recorded line number below it moved and the hook rewrote the file. Verified by diffing the hashed_secret set before and after: nothing added, nothing removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
getnable
force-pushed
the
feat/tool-tiers
branch
from
August 8, 2026 23:36
496d502 to
bfe2128
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phases 0 and 1 of the tool-hierarchy roadmap, with the
promotedecision taken:get_cost_summarybecomes the front door rather than a newcost_overviewname.The measurement
84% cut. 30,121 tokens back on every message. No capability removed.
Why flat was the problem
Family filtering (0.8.160) narrowed the surface but left it flat. 59 of 198 tools have
cost/spend/billin the name.get_cost_summary,get_costs_by_service,get_top_cost_drivers,get_cost_trends,get_service_cost,get_total_spend_all_sourcesare not six capabilities, they are six phrasings.Hidden is not gone
The MCP call path resolves against the registry, not the advertised list, so a hidden tool runs the moment something names it. That only helps if something does:
get_cost_summaryreturnsnext_tools— the drill-downs that expand this answer, picked from its own top services. A generic menu would just be the flat surface moved into the response body, so a Textract-heavy bill namesget_textract_costsand an RDS-heavy one does not.what_can_nable_do(detailed=True)for the full map, and explicitly forbid reporting a capability missing because its tool is not listed.FINOPS_FLAT_TOOLS=1restores the old surface. Narrower thanFINOPS_ALL_TOOLSon purpose: escaping the funnel shouldn't force an AWS-only user to also take the Azure tools.Unlisted fails shut, and the guard is a budget
For families, an unmapped tool is still advertised (missing capability is worse than a noisy one). For tiers the reverse holds: a tool quietly joining the front door is what costs every user tokens on every message. So the guard isn't completeness, it's a ceiling on tokens and tool count, and CI fails when either creeps.
Two tests earned their place immediately
TIER3and the drill-down map both namedget_kendra_costs,get_documentdb_costsandget_marketplace_costs. Those live in_EXTRA_TOOLSand aren't registered at all unlessFINOPS_ALL_TOOLSis set. A route to a tool the registry doesn't hold would have had the model reporting a capability the user cannot have.Re-pointed, not deleted
Five family tests in
test_tool_surface.pynow run withFINOPS_FLAT_TOOLS=1so they test the family layer alone. Without that they'd pass trivially, with the tier gate hiding the tool before the family gate was consulted.Full suite green at 2213 passed, 2 skipped.
Phases 2-4 (handles, synonym collapse, the recommendations funnel) are not in this PR.
🤖 Generated with Claude Code