What happened
When cuga_lite_bind_tools_mode binds tools to the model (which the gpt-oss-20b runtime profile does automatically — model_runtime_profile.py:9-14), a model that responds with native AIMessage.tool_calls — the encoding function-calling-trained models naturally emit — is silently mishandled by cuga-lite's execution side. Three distinct defects:
- D1 — parallel tool calls silently dropped. The recovery transpiler reads
tool_calls[0] only (src/cuga/backend/cuga_graph/nodes/cuga_lite/adapter/response_utils.py, tool_call = tool_calls[0]) and converts it to result = await tool(args) for the sandbox. Every sibling call in the same turn is discarded with no error, no log, no trace — and the model then reports success for work that never happened.
- D2 — text preamble + tool_calls → tool_calls ignored entirely.
normalize_response only attempts tool-call recovery when response.content is empty (adapter/graph_adapter.py, if not content:). A natural preamble like "I'll notify both customers now." means zero tools execute and the announcement itself becomes the final answer.
- D9 — models without
bind_tools support crash the run. langchain_core.BaseChatModel.bind_tools raises NotImplementedError by default, and NotImplementedError is a subclass of RuntimeError — so the deliberate except RuntimeError: raise for cap errors in helpers/bind_tools.py re-raises it instead of falling back to the unbound model. call_model dies instead of degrading.
Observed demo output (scripted deterministic FC model, task "notify customers acme and globex", one notify(customer) tool, cuga_lite_bind_tools_mode="all"):
=== SCENARIO A: parallel tool_calls (empty content) ===
agent's final answer : 'Done — both acme and globex have been notified.'
tools ACTUALLY run : ['acme']
VERDICT: FAIL — globex silently dropped, answer claims FALSE SUCCESS (D1)
=== SCENARIO B: text preamble + tool_calls ===
agent's final answer : "I'll notify both customers now."
tools ACTUALLY run : []
VERDICT: FAIL — ZERO tools ran; the announcement became the final answer (D2)
=== SCENARIO C: baseline code-act (no binding) ===
tools ACTUALLY run : ['acme', 'globex']
VERDICT: PASS — code-act handles it
Scenario A is the worst failure class an agent can exhibit: silent partial execution reported as full success. Scenario C shows the failures are specific to the native-FC encoding, not the task.
Motivation / why this matters now
- The gpt-oss-20b profile enables binding with zero configuration, so every deployment on that model exercises these paths today.
- The in-tree pad-to-cap measurements ("0 tool calls vs 5-7 without padding",
bind_tools/cap.py) prove real bound models do emit native tool_calls — despite the system prompt forbidding it (prompts/mcp_prompt.jinja2:138), which is its own inconsistency.
- The transpiler bridging native FC to execution has no test coverage (nothing imports
response_utils; the empty-content branch of normalize_response is never exercised).
- Full analysis, defect table (D1–D9), and a phased fix design:
docs/reports/function_calling_cuga_lite.md on branch function_calling.
How to reproduce
Reproducing tests are on branch function_calling as strict-xfail tests (executable documentation — they assert the correct behavior, xfail today, and flip to hard failures once fixed):
git checkout function_calling
uv run pytest src/cuga/backend/cuga_graph/nodes/cuga_lite/tests/test_native_tool_calls_execution.py -rxX -q
# => 1 passed, 3 xfailed
# XFAIL test_parallel_tool_calls_all_execute (D1)
# XFAIL test_text_preamble_with_tool_calls_still_executes (D2)
# XFAIL test_model_without_bind_tools_falls_back_gracefully (D9)
# PASSED test_code_act_control_executes_both (control)
The tests use a deterministic scripted BaseChatModel (no API keys, no network) with CugaAgent(tools=[notify]) and config={"configurable": {"cuga_lite_bind_tools_mode": "all"}}.
Environment
- Branch:
main (df173678), reproduced on branch function_calling
- Python 3.12, macOS (darwin), local executor (no Docker/e2b required)
- No model API needed — deterministic scripted model
Expected behavior (fix sketch)
- All
tool_calls in a turn execute (batch block; explicit per-call success/error reporting so partial execution is impossible to miss), not just [0].
text + tool_calls responses keep the text as reasoning and still execute the calls.
NotImplementedError from bind_tools falls back to the unbound model instead of crashing (catch it before the cap's RuntimeError re-raise, or use a dedicated cap exception type).
- Generated batch code must not splice tool names into source (
await {name}(...)) — names may not be valid Python identifiers; call through a lookup map.
See docs/reports/function_calling_cuga_lite.md §5 for the full design (P0 correctness → P1 typed SDK surface → P2 modes/prompt).
What happened
When
cuga_lite_bind_tools_modebinds tools to the model (which the gpt-oss-20b runtime profile does automatically —model_runtime_profile.py:9-14), a model that responds with nativeAIMessage.tool_calls— the encoding function-calling-trained models naturally emit — is silently mishandled by cuga-lite's execution side. Three distinct defects:tool_calls[0]only (src/cuga/backend/cuga_graph/nodes/cuga_lite/adapter/response_utils.py,tool_call = tool_calls[0]) and converts it toresult = await tool(args)for the sandbox. Every sibling call in the same turn is discarded with no error, no log, no trace — and the model then reports success for work that never happened.normalize_responseonly attempts tool-call recovery whenresponse.contentis empty (adapter/graph_adapter.py,if not content:). A natural preamble like "I'll notify both customers now." means zero tools execute and the announcement itself becomes the final answer.bind_toolssupport crash the run.langchain_core.BaseChatModel.bind_toolsraisesNotImplementedErrorby default, andNotImplementedErroris a subclass ofRuntimeError— so the deliberateexcept RuntimeError: raisefor cap errors inhelpers/bind_tools.pyre-raises it instead of falling back to the unbound model.call_modeldies instead of degrading.Observed demo output (scripted deterministic FC model, task "notify customers acme and globex", one
notify(customer)tool,cuga_lite_bind_tools_mode="all"):Scenario A is the worst failure class an agent can exhibit: silent partial execution reported as full success. Scenario C shows the failures are specific to the native-FC encoding, not the task.
Motivation / why this matters now
bind_tools/cap.py) prove real bound models do emit native tool_calls — despite the system prompt forbidding it (prompts/mcp_prompt.jinja2:138), which is its own inconsistency.response_utils; the empty-content branch ofnormalize_responseis never exercised).docs/reports/function_calling_cuga_lite.mdon branchfunction_calling.How to reproduce
Reproducing tests are on branch
function_callingas strict-xfail tests (executable documentation — they assert the correct behavior, xfail today, and flip to hard failures once fixed):The tests use a deterministic scripted
BaseChatModel(no API keys, no network) withCugaAgent(tools=[notify])andconfig={"configurable": {"cuga_lite_bind_tools_mode": "all"}}.Environment
main(df173678), reproduced on branchfunction_callingExpected behavior (fix sketch)
tool_callsin a turn execute (batch block; explicit per-call success/error reporting so partial execution is impossible to miss), not just[0].text + tool_callsresponses keep the text as reasoning and still execute the calls.NotImplementedErrorfrombind_toolsfalls back to the unbound model instead of crashing (catch it before the cap'sRuntimeErrorre-raise, or use a dedicated cap exception type).await {name}(...)) — names may not be valid Python identifiers; call through a lookup map.See
docs/reports/function_calling_cuga_lite.md§5 for the full design (P0 correctness → P1 typed SDK surface → P2 modes/prompt).