Tool Reranking

Tool Reranking

With 50 built-in tools plus any user tools and MCP-server tools, the list of tool schemas sent to the LLM on every turn can consume significant prompt tokens. Tool reranking is an opt-in feature that filters the tool list down to the most relevant subset for each user message — reducing tokens (and, for MCP tools, schema-fetch round-trips) without changing the chat's actual capabilities.

Reranking is off by default. Enabling it is a tradeoff: fewer tokens per turn, but the LLM only sees the subset the reranker picked. Too narrow a Top N can break multi-step tasks that need tools the first message didn't hint at.

Enabling

Settings → Tool Reranking:

• Method — Off | Keyword (free, lexical) | LLM (semantic)
• Top N — how many non-pinned tools to send (default 15)
• Pinned tools — comma-separated tool names that are always included, regardless of score (e.g. list_objects, describe_object, take_screenshot)
• LLM reranker provider (visible only when method = LLM) — Provider / Base URL / API key / Model. Leave empty to inherit from the main LLM provider. Override individual fields to run the reranker through a different provider or model (e.g. a local Ollama model for fast, free reranking while the main chat uses a cloud provider).
• Model parameters (reranker) — always-visible, editable per-model params table for the reranker's model. Shares the same cfg.model_params dict as the main Model Parameters table, keyed by model name. When an override model is set, edits save to that model's slot. When inheriting the main model, the table is prefilled with the main model's current params and edits also apply to the main model (last-save-wins against the main table). Critical for small Ollama models (num_predict, top_k, repeat_penalty) and for providers with required settings. The label above the table tells you which mode you're in.
• Test Reranker button — sends a small canonical probe (5 tools, one clear query) to the reranker LLM using the current dialog values (no need to save first). Reports success with the picked tools or the exact error from the provider. Useful for diagnosing 4xx errors, auth failures, timeouts, and unparseable responses before sending a real chat message.

When enabled, the Report View will print a line per user turn showing which method ran and which tools were selected:

[FreeCAD AI] Reranker (keyword): 5 of 52 tools -> fillet_edges, chamfer_edges, list_objects, list_edges, describe_object

This is the easiest way to verify the feature is working and to tune Top N / pinned tools to your workflow.

Choosing a method

Method	Latency	Cost	Filter quality
Off	—	—	No filtering; all tools sent every turn
Keyword	None	None	Lexical only — misses synonyms, stemming, concepts
LLM	+1 round-trip per user turn	1 small LLM call	Semantic — understands synonyms, paraphrase, context

Use Keyword when you want a free token reduction and your tool names/descriptions are already well-chosen. Use LLM when you have many tools with similar-sounding names, conceptual queries ("make this hollow"), or MCP servers whose tool descriptions are sparse. Use Off if you have fewer than ~20 tools and token usage isn't a concern.

How it works (keyword reranker)

The keyword reranker is pure-Python scoring with IDF weighting. No extra LLM call, no extra cost, no extra latency. The pipeline runs once per user turn, right before the tool schemas are generated.

Stage 1 — Extract the user message

In _continue_send(), _extract_latest_user_text() walks the conversation history in reverse, skips [System] ... synthetic messages (framework chatter), and returns the text of the most recent user-authored message. Block-list content (with attached images or documents) is flattened to just the type: "text" parts.

Stage 2 — Tokenize query and tool corpus

Both the user message and every tool's name + description are tokenized by the same function (_tokenize):

• Lowercase, then extract [A-Za-z][A-Za-z0-9_]* tokens
• Drop tokens shorter than 2 characters
• Drop ~70 stopwords (a, the, is, with, ...). Domain vocabulary (circle, sketch, box) is deliberately kept.

A second pass (_expand_snake_case) splits snake_case tokens into constituent words, keeping the original. edit_sketch becomes [edit_sketch, edit, sketch]. This is critical: tool names are snake_case but user queries are prose — without this, "edit my sketch" wouldn't match the tool name edit_sketch at all.

Stage 3 — IDF across the tool corpus

Smoothed IDF: log((N + 1) / (df + 1)) + 1. A term that appears in every tool's description (object, create) carries almost no weight. A term that appears in only one tool (revolve appearing only in revolve_sketch) gets a high weight.

This is why querying "revolve around axis" surfaces revolve_sketch at rank 1 — the rare term dominates the score.

Stage 4 — Score and rank

For each non-pinned tool, sum the IDF weights of query tokens that also appear in the tool's tokens. Each query token contributes at most once, so repeating a word in the query can't dominate.

Sort by (-score, name) — score descending, name ascending as tiebreaker (makes output deterministic). Return pinned + top_n names.

Stage 5 — Filter the schemas

ToolRegistry.to_openai_schema(filter_names=...) / to_anthropic_schema(filter_names=...) skip any tool whose name isn't in the set. Excluded tools never have their deferred params resolved — so for MCP tools, we also skip the round-trip that would otherwise fetch the JSON Schema from the external server.

The filter operates on visibility to the LLM, not capability: the full registry is still passed to the agentic loop for execution. If the LLM manages to reference a tool that wasn't in the filtered set (which shouldn't happen unless the LLM hallucinates), the registry can still execute it.

Once per user turn — not per agentic step

The filter is computed once at the start of _continue_send() and reused for the entire agentic loop. Flickering the visible tool set mid-loop would confuse the LLM and potentially desync tool_use / tool_result pairings — so reranking happens at user-message granularity, not per LLM call.

User tools and MCP tools

The reranker operates on the ToolRegistry, which doesn't distinguish built-in tools from user tools (.py / .FCMacro in ~/.config/FreeCAD/FreeCADAI/tools/) or MCP-server tools. All registered tools are scored and filtered on the same footing.

This is intentional — any other choice would be worse:

• "Always include user tools" defeats the point when someone registers 20 user tools
• "Never include user tools" violates the reasonable expectation that custom tools participate in relevance ranking
• A per-source toggle adds config surface for a distinction the runtime doesn't actually care about

Quality tips for custom tools

Keyword reranking is only as good as a tool's name + description. Two practices matter:

• Descriptive snake_case names. The reranker expands hollow_shell into [hollow_shell, hollow, shell], so descriptive compound names pay off. Generic names like do_thing or helper have no signal and will almost never rank in, even for relevant queries.
• First-line docstring about the domain concept, not the parameters. "Hollow out a solid by shelling inward" scores better than "Take a body and return a shell".

Escape hatch: pinned tools

If a custom tool is critical to your workflow and the ranker keeps filtering it out, add its name to Pinned tools. Pinned tools bypass scoring entirely and are always included.

How it works (LLM reranker)

The LLM reranker sends a short prompt to a small/fast LLM, listing every tool's name and description plus the user's request, and asks for a JSON array of the top-N tool names by relevance. The returned list is validated against the real tool set (hallucinated names are dropped) and topped up from the keyword reranker if too few valid names came back.

Under the hood (rerank_tools_llm in freecad_ai/tools/reranker.py):

1. Separate pinned tools — same as keyword, pinned tools are returned unconditionally and don't go through the LLM.
2. Build the prompt — system prompt instructs the LLM to return ONLY a JSON array; user prompt lists - name: description lines and the user's request.
3. Call the LLM — reuses the standard LLMClient, with temperature=0.0 for determinism, max_tokens=1024 (plenty for a name list), thinking="off".
4. Parse the response — three fallbacks in order: direct JSON parse, strip markdown code fence and retry, regex-extract the first bracketed sequence. Handles any of these LLM output styles gracefully.
5. Validate — filter returned names against the known tool set (hallucinated names silently dropped), dedupe while preserving order.
6. Top up from keyword — if fewer than top_n valid names survived, fill the remaining slots using the keyword reranker. Guarantees you get top_n usable tools even from a flaky LLM response.
7. Cap — enforce top_n in case the LLM returned more than asked.

Robustness contract

The LLM reranker is designed to never make the chat experience worse than keyword reranking alone:

• LLM call fails (timeout, HTTP error, connection refused) → falls back to keyword reranker for that turn
• Unparseable response → falls back to keyword reranker
• Partial response (fewer valid names than top_n) → tops up with keyword picks
• Hallucinated tool names → silently dropped
• Config is broken (invalid provider name) → falls back to keyword reranker

A warning is logged via Python's logging module but the chat continues normally. You won't see dialogs or errors from a broken LLM reranker — just potentially worse filter quality until the issue is resolved.

Provider override

By default, the LLM reranker uses the same provider, base URL, API key, and model as the main chat. Override any of these four fields individually:

• Provider + Base URL + API key + Model — run reranking through a completely different provider (e.g. local Ollama while the main chat is Anthropic)
• Just Model — same provider, but a smaller/faster model (e.g. main = claude-opus-4-7, reranker = claude-haiku-4-5)
• Nothing set — pure inheritance; useful if you're just testing whether LLM reranking helps before tuning the provider

Reranker model parameters

Sampling params for the reranker's model come from the shared cfg.model_params[model] dict, so a given model's params are consistent regardless of which role it plays. Two cases:

• Inherited model — the reranker uses the main Model Parameters table's settings. Provider quirks like Moonshot/Kimi's locked temperature=1 are inherited automatically. The inline reranker params table is prefilled with those values and stays editable; edits also apply to the main model since both tables write to the same cfg.model_params[main_model] entry.
• Override model — the inline reranker params table edits params for that model specifically. Critical for small Ollama models (num_predict: 128, top_k: 40, repeat_penalty: 1.1) and for providers that reject the default temperature=0.0.

Use the Test Reranker button after configuring the provider/model/params to verify the LLM call works before sending a real chat message. The button sends a small probe and shows one of three outcomes:

• Green (OK) — LLM returned at least one valid tool name. The status shows which tools were picked and how many came from LLM vs. keyword top-up. A result like "2 from LLM, 3 from keyword top-up" means the LLM is contributing but the reranker is still topping up the remainder.
• Red (Error) — HTTP failure — the provider returned a 4xx/5xx. Status shows the exact error message. Fix the provider/URL/key/model config.
• Red (Error) — 0 valid names — the LLM responded but produced nothing usable (empty, malformed, all hallucinated names, or echoed control tags like /no_think). The raw response preview below the status message tells you what the model actually said. Pick a different model.

Choosing an Ollama reranker model

Not every small Ollama model is a good reranker. What you need: a model that reliably outputs JSON arrays when instructed, with minimal commentary. Gotchas we've seen:

• Models that echo control tags. Some fine-tunes (e.g. certain Gemma variants) will echo /no_think — a Qwen3 control tag FreeCAD AI appends to Ollama system prompts — as their entire response. The Test Reranker button flags this immediately (raw response = literally /no_think).
• Models without strong instruction-following. Conversational or chat-focused fine-tunes may prepend "Sure! Here are the most relevant tools:" or wrap output in markdown. The parser handles both of those, but some models produce longer prose that nothing extracts from.

Recommended starting points for Ollama reranking:

• Qwen2.5 / Qwen3 small variants (qwen2.5:3b, qwen3:1.7b) — strong JSON output, support the /no_think tag properly
• Llama3 / Llama3.2 (llama3.2:3b) — solid general instruction-following
• Any instruct-tuned model ≥ 1.5B parameters with good JSON benchmarks

Ollama Base URL gotcha: if you run reranking through a local or remote Ollama, the Base URL must end in /v1, not /api/. Example: http://localhost:11434/v1 or http://myhost:11434/v1. Ollama exposes its OpenAI-compatible endpoints under /v1/* and its native endpoints under /api/* — FreeCAD AI uses the OpenAI-compatible interface, so /api/ will return HTTP 404. See Configuration#Ollama (Local) for the full explanation.

A good rule of thumb: the reranker sees at most len(tools) * avg_description_length tokens per user turn. For 50 tools with ~80-character descriptions, that's ~5K tokens of input and ~100 tokens of output. That fits comfortably within the free tier of most providers and runs in under a second on a cold Ollama model. No need to over-engineer the setup.

Known limitations (keyword reranker only)

Keyword scoring is a lexical filter, not a semantic one. Expect these blind spots:

• Synonyms: "drill a hole" won't match pocket_sketch because "hole" and "pocket" share no tokens.
• Stemming: "circles" in the query doesn't match "circle" in a description — two different tokens.
• Concepts: "make this part hollow" should surface shell_object, but "hollow" and "shell" are unrelated strings.
• Paraphrase: "cut it in half and rotate each half" should probably surface mirror/revolve/boolean tools.
• Context: only the last user message is considered.

All of these are handled naturally by the LLM reranker — it understands meaning, not just strings. If you hit these blind spots with keyword, either pin the relevant tools or switch to LLM.

Tuning guide

A reasonable starting configuration for a 50-tool workbench:

• Top N: 15 — enough headroom for multi-step workflows, still a meaningful token reduction
• Pinned: list_objects, describe_object — ensures the LLM can always inspect the document

Tune down (Top N: 8) for single-shot tasks where you're confident the user request maps to a small set of tools. Tune up (Top N: 25) if you see the LLM asking for tools it doesn't have, or bailing out mid-task.

If you use MCP to expose many external tools (100+), consider enabling reranking unconditionally — that's the regime where the token savings become substantial.

Note on MCP deferred loading. MCP tools get an additional benefit from reranking: their schemas are deferred (only fetched on demand). If the reranker filters an MCP tool out, we skip the round-trip that would otherwise fetch its JSON Schema. So for MCP tools, reranking saves not just tokens but also latency on the first message after a cold server connect.

A theoretical failure mode worth flagging: a poorly-designed or malicious MCP server could ship tools with names stuffed with keywords to dominate rankings (create_sketch_pad_pocket_fillet_chamfer_shell). The reranker has no defense against this. Relevant if you ever connect to a registry of community-provided MCP servers whose quality you haven't vetted.

Code references

• Keyword reranker: freecad_ai/tools/reranker.py — rerank_tools(), _tokenize, _expand_snake_case, _compute_idf, _score
• LLM reranker: freecad_ai/tools/reranker.py — rerank_tools_llm(), _build_rerank_prompt(), _parse_rerank_response()
• Registry integration: freecad_ai/tools/registry.py — to_openai_schema(filter_names=...), to_anthropic_schema(filter_names=...), to_mcp_schema(filter_names=...), list_name_description_pairs()
• Call site: freecad_ai/ui/chat_widget.py — _continue_send(), _extract_latest_user_text(), _run_reranker(), _build_rerank_llm_client()
• Config: freecad_ai/config.py — rerank_method, rerank_top_n, rerank_pinned_tools, rerank_llm_provider_name, rerank_llm_base_url, rerank_llm_api_key, rerank_llm_model
• Tests: tests/unit/test_reranker.py