Skip to content

[Bug]: cuga-lite silently mishandles native tool_calls: parallel calls dropped, preamble+tool_calls ignored, bind_tools NotImplementedError crashes the run #471

Description

@segevshlomovIBM

What happened

When cuga_lite_bind_tools_mode binds tools to the model (which the gpt-oss-20b runtime profile does automaticallymodel_runtime_profile.py:9-14), a model that responds with native AIMessage.tool_calls — the encoding function-calling-trained models naturally emit — is silently mishandled by cuga-lite's execution side. Three distinct defects:

  • D1 — parallel tool calls silently dropped. The recovery transpiler reads tool_calls[0] only (src/cuga/backend/cuga_graph/nodes/cuga_lite/adapter/response_utils.py, tool_call = tool_calls[0]) and converts it to result = await tool(args) for the sandbox. Every sibling call in the same turn is discarded with no error, no log, no trace — and the model then reports success for work that never happened.
  • D2 — text preamble + tool_calls → tool_calls ignored entirely. normalize_response only attempts tool-call recovery when response.content is empty (adapter/graph_adapter.py, if not content:). A natural preamble like "I'll notify both customers now." means zero tools execute and the announcement itself becomes the final answer.
  • D9 — models without bind_tools support crash the run. langchain_core.BaseChatModel.bind_tools raises NotImplementedError by default, and NotImplementedError is a subclass of RuntimeError — so the deliberate except RuntimeError: raise for cap errors in helpers/bind_tools.py re-raises it instead of falling back to the unbound model. call_model dies instead of degrading.

Observed demo output (scripted deterministic FC model, task "notify customers acme and globex", one notify(customer) tool, cuga_lite_bind_tools_mode="all"):

=== SCENARIO A: parallel tool_calls (empty content) ===
agent's final answer : 'Done — both acme and globex have been notified.'
tools ACTUALLY run   : ['acme']
VERDICT: FAIL — globex silently dropped, answer claims FALSE SUCCESS        (D1)

=== SCENARIO B: text preamble + tool_calls ===
agent's final answer : "I'll notify both customers now."
tools ACTUALLY run   : []
VERDICT: FAIL — ZERO tools ran; the announcement became the final answer   (D2)

=== SCENARIO C: baseline code-act (no binding) ===
tools ACTUALLY run   : ['acme', 'globex']
VERDICT: PASS — code-act handles it

Scenario A is the worst failure class an agent can exhibit: silent partial execution reported as full success. Scenario C shows the failures are specific to the native-FC encoding, not the task.

Motivation / why this matters now

  • The gpt-oss-20b profile enables binding with zero configuration, so every deployment on that model exercises these paths today.
  • The in-tree pad-to-cap measurements ("0 tool calls vs 5-7 without padding", bind_tools/cap.py) prove real bound models do emit native tool_calls — despite the system prompt forbidding it (prompts/mcp_prompt.jinja2:138), which is its own inconsistency.
  • The transpiler bridging native FC to execution has no test coverage (nothing imports response_utils; the empty-content branch of normalize_response is never exercised).
  • Full analysis, defect table (D1–D9), and a phased fix design: docs/reports/function_calling_cuga_lite.md on branch function_calling.

How to reproduce

Reproducing tests are on branch function_calling as strict-xfail tests (executable documentation — they assert the correct behavior, xfail today, and flip to hard failures once fixed):

git checkout function_calling
uv run pytest src/cuga/backend/cuga_graph/nodes/cuga_lite/tests/test_native_tool_calls_execution.py -rxX -q
# => 1 passed, 3 xfailed
#    XFAIL test_parallel_tool_calls_all_execute            (D1)
#    XFAIL test_text_preamble_with_tool_calls_still_executes (D2)
#    XFAIL test_model_without_bind_tools_falls_back_gracefully (D9)
#    PASSED test_code_act_control_executes_both            (control)

The tests use a deterministic scripted BaseChatModel (no API keys, no network) with CugaAgent(tools=[notify]) and config={"configurable": {"cuga_lite_bind_tools_mode": "all"}}.

Environment

  • Branch: main (df173678), reproduced on branch function_calling
  • Python 3.12, macOS (darwin), local executor (no Docker/e2b required)
  • No model API needed — deterministic scripted model

Expected behavior (fix sketch)

  1. All tool_calls in a turn execute (batch block; explicit per-call success/error reporting so partial execution is impossible to miss), not just [0].
  2. text + tool_calls responses keep the text as reasoning and still execute the calls.
  3. NotImplementedError from bind_tools falls back to the unbound model instead of crashing (catch it before the cap's RuntimeError re-raise, or use a dedicated cap exception type).
  4. Generated batch code must not splice tool names into source (await {name}(...)) — names may not be valid Python identifiers; call through a lookup map.

See docs/reports/function_calling_cuga_lite.md §5 for the full design (P0 correctness → P1 typed SDK surface → P2 modes/prompt).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcomponent: agentCore agent loop, DynamicAgentGraph, LLM node, tool execution, CugaLiteneeds-triageNewly filed, not yet reviewedpriority: highImportant, address soon

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    In progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions