feat: Gemma 4 tool calling support via Nous inference API

[0xarkstar]

Summary

Gemma 4 (31B, 26B) models do not work with tool calling when using the Nous Research inference API. The model outputs tool calls in text format, but they are not parsed into structured tool_calls on the API response.

Problem

When using Gemma 4 via Nous inference API (https://inference-api.nousresearch.com/v1):

1. Request includes tools= parameter (OpenAI-compatible)
2. Gemma 4 generates tool calls in its native text format:

   <|tool_call>call:search(query: "blockchain news")<tool_call|>
   

3. API response returns tool_calls: null — the text remains in content only
4. Hermes Agent treats this as a final text response and does not execute the tool

Evidence

Tested with google/gemma-4-31b-it via Nous API:

# With native tools parameter → tool_calls is always None
payload = {"model": "google/gemma-4-31b-it", "tools": [...], "messages": [...]}
response.choices[0].message.tool_calls  # None
response.choices[0].message.content     # "<|tool_call>call:search(...)<tool_call|>"

# Same test with Qwen → tool_calls is populated correctly
payload = {"model": "qwen/qwen3.5-plus-02-15", "tools": [...], "messages": [...]}
response.choices[0].message.tool_calls  # [{"function": {"name": "search", ...}}]

6 different prompt patterns tested (single arg, multi arg, nested dict, multi tool call, integer args, Unicode/Korean) — Gemma consistently outputs valid tool calls in content but tool_calls field is always None.

Request

Please add server-side parsing for Gemma 4's tool call format (<|tool_call>call:name(args)<tool_call|>) in the Nous inference API, similar to how Qwen's <tool_call> format is already parsed into structured tool_calls.

Related

• Gemma 4 tool calling support (parser availability & required configuration) #6626 — Gemma 4 tool calling parser for vLLM (Phase 2)
• feat: add Gemma 4 tool call parser #7449 — PR adding client-side Gemma 4 parser + model-aware fallback in agent_loop.py

Workaround

Currently using Qwen 3.5 as an alternative. A client-side parser has been submitted in #7449 for Phase 2 (vLLM) and RL training fallback, but Phase 1 (Nous API) requires server-side support.

[raullenchai]

For local inference on Mac, this works today with Rapid-MLX — it parses Gemma 4's native tool call format server-side and returns standard OpenAI `tool_calls` JSON to Hermes. No client-side parser needed.

20/20 integration tests pass on Gemma 4 26B MoE: chat, single/parallel tool calls, multi-turn, streaming, code gen + test execution, file patching, git analysis.

```bash
pip install vllm-mlx && rapid-mlx serve gemma-4-26b
```

Details: https://github.com/raullenchai/Rapid-MLX/issues/7800#issuecomment-4230218340

[0xarkstar]

Closing. The implementation path for this (#7449) was closed after upstream removed environments/tool_call_parsers/ entirely, and a practical server-side workaround exists for local inference: Rapid-MLX parses Gemma 4's native tool-call format and returns standard OpenAI tool_calls JSON (thanks @raullenchai). With client-side parsing no longer the right layer for this, I'm not pursuing it further.