Skip to content

feat: pluggable tool-call adapter + OpenAI tool support (closes #95, #96, #97) - #98

Merged
pekkah merged 2 commits into
masterfrom
feat/tool-call-adapter
May 30, 2026
Merged

feat: pluggable tool-call adapter + OpenAI tool support (closes #95, #96, #97)#98
pekkah merged 2 commits into
masterfrom
feat/tool-call-adapter

Conversation

@pekkah

@pekkah pekkah commented May 30, 2026

Copy link
Copy Markdown
Owner

Summary

Completes the tool-calling feature across both HTTP wire formats by introducing a per-architecture translation layer between the model's free-text tool-call output and the structured tool_use / tool_calls shapes the endpoints emit.

  • New IToolCallAdapter + ToolCallAdapterRegistry in SharpInference.Core keyed on the GGUF general.architecture field. Adapter ships for Qwen2/3 (<tool_call> wrapper, JSON or XML inside), Qwen3-Coder (bare <function=name>...</function>, closes Tool-call parser misses Qwen3-Coder's bare <function=...> format (no <tool_call> wrapper) #95), Llama-3 (<|python_tag|>{json}<|eom_id|>), and DeepSeek-R1 multi-call envelope.
  • Anthropic /v1/messages streaming + non-streaming now drive through the adapter. Streaming open/close-marker scan replaces the hard-coded <tool_call> matcher; the buffer guard becomes adapter.MaxOpenTagLength - 1. tool_result history-replay goes through adapter.RenderToolResult so families that need a non-tool role (e.g. Llama-3 ipython) replay correctly.
  • OpenAI /v1/chat/completions gains full tool support (OpenAI /v1/chat/completions: tool-call wire-format support (parity with /v1/messages) #97): tools / tool_choice request fields, tool_calls array on the assistant message, tool_calls streaming deltas, finish_reason: "tool_calls", history-side role: "tool" echoes via tool_call_id. The buffering state machine activates only when tools are present so the existing no-tools per-chunk streaming cadence is preserved.

Test plan

  • 21 adapter unit tests in SharpInference.Tests.Core (registry + Qwen, Qwen3-Coder, Llama, DeepSeek wire formats).
  • 7 endpoint tests in SharpInference.Tests.Server: Qwen3-Coder /v1/messages non-stream + stream (closes Tool-call parser misses Qwen3-Coder's bare <function=...> format (no <tool_call> wrapper) #95), OpenAI tool roundtrip non-stream + stream, OpenAI tool history echo, OpenAI no-tools per-chunk cadence regression guard.
  • Existing 7 Anthropic tool tests remain green.
  • Full suite: 505/505 passing (Core 78, ForwardPass 261, Pipeline 33, TurboQuant 41, Server 92).

Closes #95, #96, #97.

🤖 Generated with Claude Code

, #97)

Introduces IToolCallAdapter (Core) and ToolCallAdapterRegistry keyed on GGUF
architecture. Ships adapters for Qwen2/3 (<tool_call> wrapper), Qwen3-Coder
(bare <function=...>), Llama-3 (<|python_tag|>...<|eom_id|>), and DeepSeek-R1
(<|tool_calls_begin|>...<|tool_calls_end|>). ChatTemplateRenderer resolves the
adapter once at engine-load time.

AnthropicEndpoints: replaces the hard-coded <tool_call> matcher in
HandleStreaming + non-streaming ParseToolCalls with the adapter's open/close
marker API. tool_result history-replay now goes through adapter.RenderToolResult
so families like Llama-3 (role: ipython) replay correctly.

OpenAiEndpoints: adds Tools / ToolChoice on ChatCompletionRequest, tool_calls on
OaiAssistantMessage + ChunkDelta, OaiToolCall/OaiToolCallDelta records, history
role=tool messages, finish_reason=tool_calls. Streaming buffering activates only
when tools are in the request so the no-tools per-chunk cadence is preserved.

Tests: 21 adapter unit tests (Qwen, Qwen3-Coder, Llama, DeepSeek + registry), 7
endpoint tests covering Qwen3-Coder /v1/messages non-stream + stream (closes
#95), OpenAI /v1/chat/completions non-stream + stream + tool history + no-tools
regression guard. Full suite green: 505/505.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a translation layer for LLM tool calls by implementing IToolCallAdapter and concrete adapters for Qwen, Llama, and DeepSeek model families, integrating them into the Anthropic and OpenAI endpoints. The review feedback identifies critical robustness issues, including potential ArgumentOutOfRangeException and NullReferenceException crashes when handling malformed model outputs or request payloads, as well as a memory leak from an undisposed JsonDocument in ParseJsonCallBlock.

Comment on lines +158 to +161
int nameEnd = block.IndexOf('>', funcStart);
if (nameEnd < 0) return null;
string funcName = block[(funcStart + funcTag.Length)..nameEnd].Trim();
if (funcName.Length == 0) return null;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If the model output is malformed and does not contain a valid function name (e.g., <function=>), nameEnd can be less than or equal to funcStart + funcTag.Length. This causes an ArgumentOutOfRangeException when slicing the string. Adding a guard prevents potential server crashes on untrusted model outputs.

        int nameEnd = block.IndexOf('>', funcStart);
        if (nameEnd < 0 || nameEnd <= funcStart + funcTag.Length) return null;
        string funcName = block[(funcStart + funcTag.Length)..nameEnd].Trim();

Comment on lines +175 to +178
int paramNameEnd = funcBody.IndexOf('>', paramStart);
if (paramNameEnd < 0) break;
string paramName = funcBody[(paramStart + paramTag.Length)..paramNameEnd].Trim();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Similarly to the function name parsing, if a parameter tag is malformed (e.g., <parameter=>), paramNameEnd can be less than or equal to paramStart + paramTag.Length. This leads to an ArgumentOutOfRangeException during slicing. A guard is needed here to safely handle malformed parameters.

            int paramNameEnd = funcBody.IndexOf('>', paramStart);
            if (paramNameEnd < 0 || paramNameEnd <= paramStart + paramTag.Length) break;
            string paramName = funcBody[(paramStart + paramTag.Length)..paramNameEnd].Trim();

Comment on lines +402 to +407
["function"] = new Dictionary<string, object?>(StringComparer.Ordinal)
{
["name"] = tc.Function.Name,
// OpenAI stringifies arguments; pass through as-is.
["arguments"] = tc.Function.Arguments,
},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If a client sends a history message with a tool_calls array where the function property is missing or null, accessing tc.Function.Name or tc.Function.Arguments directly will throw a NullReferenceException. Using null-conditional operators prevents server crashes on malformed request payloads.

                            ["function"] = new Dictionary<string, object?>(StringComparer.Ordinal)
                            {
                                ["name"]      = tc.Function?.Name,
                                // OpenAI stringifies arguments; pass through as-is.
                                ["arguments"] = tc.Function?.Arguments,
                            },

Comment on lines +427 to +432
["function"] = new Dictionary<string, object?>(StringComparer.Ordinal)
{
["name"] = t.Function.Name,
["description"] = t.Function.Description,
["parameters"] = t.Function.Parameters,
},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If an incoming tool definition in the request payload has a null or missing function object (e.g., {"type": "function"}), accessing t.Function.Name will result in a NullReferenceException. Null-conditional operators should be used to handle such malformed inputs gracefully.

                    ["function"] = new Dictionary<string, object?>(StringComparer.Ordinal)
                    {
                        ["name"]        = t.Function?.Name,
                        ["description"] = t.Function?.Description,
                        ["parameters"]  = t.Function?.Parameters,
                    },

Comment on lines +212 to +214
object? parsed = argsElem.ValueKind == JsonValueKind.String
? JsonElementToObject(JsonDocument.Parse(argsElem.GetString() ?? "{}").RootElement)
: JsonElementToObject(argsElem);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

JsonDocument.Parse allocates unmanaged resources and implements IDisposable. Failing to dispose of the parsed document results in memory leaks, which can degrade performance over time in high-throughput environments. Wrapping the parsing in a using block resolves this leak.

            object? parsed = null;
            if (argsElem.ValueKind == JsonValueKind.String)
            {
                using var innerDoc = JsonDocument.Parse(argsElem.GetString() ?? "{}");
                parsed = JsonElementToObject(innerDoc.RootElement);
            }
            else
            {
                parsed = JsonElementToObject(argsElem);
            }

…ol-call surfacing

- ToolCallAdapterRegistry: ConcurrentDictionary instead of Dictionary (was racy
  between request-time Get and startup-time Register).
- ParseJsonCallBlock: dispose the inner JsonDocument when arguments is a JSON
  string (leak under high QPS).
- ParseXmlFunctionBlock: skip <parameter=> tags with empty names instead of
  inserting an empty-string dict key.
- OaiToolCall.Function and OaiTool.Function are now nullable to match runtime
  reality of source-gen JSON; rich-message builder uses ?. so malformed history
  / tool defs no longer NRE the request.
- Anthropic + OpenAI streaming: when the stream ends mid-tool-call, flush the
  buffered bytes as a text content delta and signal truncation via
  finish_reason=length (OpenAI) / stop_reason=max_tokens (Anthropic) instead
  of silently dropping the partial call.
- Request-deserialization catches narrowed to JsonException +
  BadHttpRequestException; OperationCanceledException now propagates. Error
  body now includes the parse error message instead of being empty.
- Streaming finally-block catches narrowed to OperationCanceledException +
  IOException; non-aborted errors propagate instead of being silently
  swallowed.
- AnthropicEndpoints non-streaming tool input fallback narrowed to
  JsonException, and reuses the existing EmptyJsonObject instead of
  parsing "{}" per call.

Tests:
- New OpenAi_WithTools_Streaming_SplitOpenMarkerAcrossChunks regression guard
  for the maxOpenLen buffering — delivers <tool_ / call> across two engine
  chunks.
- Anthropic streaming test now parses the message_delta event and asserts
  stop_reason==tool_use explicitly rather than substring-matching.

Verified end-to-end against a running Qwen3-8B server: malformed JSON now
returns a 400 with the parse error in the body, null function on tool defs
no longer NREs, OpenAI tool-history echo path produces the expected reply,
Anthropic SSE shape unchanged for the non-tool case.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@pekkah
pekkah merged commit 6fa096d into master May 30, 2026
1 check passed
@pekkah
pekkah deleted the feat/tool-call-adapter branch May 30, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tool-call parser misses Qwen3-Coder's bare <function=...> format (no <tool_call> wrapper)

1 participant