Bug: Tool calls with Object arguments are serialized as string

[TatuLund]

When using FastFlowLM with OpenCode and Gemma 4 E4B model, I observed frequent tool call failures, by inspecting the log I observed the likely cause to be wrong data serialization

        {
            "content": "",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": "{\"filePath\":\"./src/main/java/my/path/to/JavaClass.java\"}",
                        "name": "read"
                    },
                    "id": "call_1777830235_96",
                    "type": "function"
                },
                {
                    "function": {
                        "arguments": "{\"filePath\":\"./src/main/java/my/path/to/OtherClass.java\"}",
                        "name": "read"
                    },
                    "id": "call_1777830239_97",
                    "type": "function"
                }
            ]
        },

This results in error:

Invalid input for tool read: JSON parsing failed: Text: {{\\\"filePath\\\":\\\"./src/main/java/my/path/to/JavaClass.java\\\\"}}.
Error message: JSON Parse error: Expected '}'

Probably related to #483

[ZaneNi]

@TatuLund Thanks for your feedback. May I ask are you using stream or non-stream mode?

[TatuLund]

Thanks for your feedback. May I ask are you using stream or non-stream mode?

@ZaneNi I would assume OpenCode uses stream mode by default, I have not touched the settings. I observed the same issue also using Copilot CLI.

I did not see this problem when I was serving the model with LM Studio, but that setup had other problems with running models stable and that was the main motivation to try out FastFlowLM. FastFlowLM feels much more performant and stable compared to that what comes to inference, but the tool calls seem not to work, and it is essential for the workflow.

[ZaneNi]

Thanks for your feedback. May I ask are you using stream or non-stream mode?

@ZaneNi I would assume OpenCode uses stream mode by default, I have not touched the settings. I observed the same issue also using Copilot CLI.

I did not see this problem when I was serving the model with LM Studio, but that setup had other problems with running models stable and that was the main motivation to try out FastFlowLM. FastFlowLM feels much more performant and stable compared to that what comes to inference, but the tool calls seem not to work, and it is essential for the workflow.

@TatuLund We will look into it. Please stay tuned!

[jtmonroe]

+1 to this issue

If I had to hazard a guess, it would be due to the fact that there is no json_schema validation on the tool end. Looking at the base function for the generation, I do not think there is any implementation of structured output.

[ZaneNi]

@TatuLund @jtmonroe

Please try the latest release. Thank you!

[jtmonroe]

Using OpenCode it takes a minute longer, but I get the same issues. In particular, it looks like I either get an invalid initial token (an extra {) or it fails to close the initial brace. Either way, the JSON parsing is not quite right. I am working on integrating llguidance into the parser since OpenCode does include "type" : "json_schema" in some tool calls.

Here is the failure log

       {
            "content": "",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": "{\"tool\":\"todowrite\",\"error\":\"Invalid input for tool todowrite: JSON parsing failed: Text: {{\\\"todos\\\":[\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 1: Define the Task struct (ID, description, timestamp) in Rust.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 2: Establish central state structure for holding collection of tasks.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 3: Mockup/decide on basic in-memory persistence (Vec<Task>).\\\",\\n        \\\"priority\\\": \\\"medium\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 4: Design the main Slint UI to include task input field and list view.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 5: Implement backend logic to create and add new tasks from UI input.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"}\\n]}.\\nError message: JSON Parse error: Expected '}'\"}",
                        "name": "invalid"
                    },
                    "id": "call_1778692887_4",
                    "type": "function"
                }
            ]
        },

[ZaneNi]

Using OpenCode it takes a minute longer, but I get the same issues. In particular, it looks like I either get an invalid initial token (an extra {) or it fails to close the initial brace. Either way, the JSON parsing is not quite right. I am working on integrating llguidance into the parser since OpenCode does include "type" : "json_schema" in some tool calls.

Here is the failure log

       {
            "content": "",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": "{\"tool\":\"todowrite\",\"error\":\"Invalid input for tool todowrite: JSON parsing failed: Text: {{\\\"todos\\\":[\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 1: Define the Task struct (ID, description, timestamp) in Rust.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 2: Establish central state structure for holding collection of tasks.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 3: Mockup/decide on basic in-memory persistence (Vec<Task>).\\\",\\n        \\\"priority\\\": \\\"medium\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 4: Design the main Slint UI to include task input field and list view.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"\\n    },\\n    {\\n        \\\"content\\\": \\\"Implement Phase 1, Step 5: Implement backend logic to create and add new tasks from UI input.\\\",\\n        \\\"priority\\\": \\\"high\\\",\\n        \\\"status\\\": \\\"pending\\\"}\\n]}.\\nError message: JSON Parse error: Expected '}'\"}",
                        "name": "invalid"
                    },
                    "id": "call_1778692887_4",
                    "type": "function"
                }
            ]
        },

Can you show more details so that I could reproduce it?

[jtmonroe]

What details would you like? The basic outline that I go through to test is the following.

1. start flm on 172.17.0.1 (host.docker.internal)
2. inside docker devcontainer, create ~/.config/opencode/opencode.json with proper config.
3. Install and start opencode.
4. Ask it to read files, list files, and make todo lists.
5. The model eventually drifts and starts making growing errors.

[ZaneNi]

What details would you like? The basic outline that I go through to test is the following.

1. start flm on 172.17.0.1 (host.docker.internal)
2. inside docker devcontainer, create ~/.config/opencode/opencode.json with proper config.
3. Install and start opencode.
4. Ask it to read files, list files, and make todo lists.
5. The model eventually drifts and starts making growing errors.

Thank you! Are you on the latest version?

We tested with OpenCode and it worked well.

You might want to see the video https://youtu.be/56DSREp9RgU?si=iOWkmvIDK8-bKdOE.

[jtmonroe]

What details would you like? The basic outline that I go through to test is the following.

1. start flm on 172.17.0.1 (host.docker.internal)
2. inside docker devcontainer, create ~/.config/opencode/opencode.json with proper config.
3. Install and start opencode.
4. Ask it to read files, list files, and make todo lists.
5. The model eventually drifts and starts making growing errors.

Thank you! Are you on the latest version?

We tested with OpenCode and it worked well.

You might want to see the video https://youtu.be/56DSREp9RgU?si=iOWkmvIDK8-bKdOE.

Yeah. I updated immediately: 0.9.42. I am on a linux system so there could be some divergence. I will keep on my integration of llguidance for the time being.

[ZaneNi]

What details would you like? The basic outline that I go through to test is the following.

1. start flm on 172.17.0.1 (host.docker.internal)
2. inside docker devcontainer, create ~/.config/opencode/opencode.json with proper config.
3. Install and start opencode.
4. Ask it to read files, list files, and make todo lists.
5. The model eventually drifts and starts making growing errors.

Thank you! Are you on the latest version?
We tested with OpenCode and it worked well.
You might want to see the video https://youtu.be/56DSREp9RgU?si=iOWkmvIDK8-bKdOE.

Yeah. I updated immediately: 0.9.42. I am on a linux system so there could be some divergence. I will keep on my integration of llguidance for the time being.

@jtmonroe I was able to reproduce the issue!

This appears to be another corner case where the model generates an invalid schema from its chat template 😅. In your case, there was an extra "{". Another issue I encountered is that the model sometimes uses ` instead of " to terminate the JSON arguments, which can also cause problems.

I’ve submitted a PR to improve the JSON sanity checks for these corner cases. You can try the artifacts here and see if they resolve the issue.

Feel free to report any other issues you encounter — your feedback is always appreciated!

[jtmonroe]

What details would you like? The basic outline that I go through to test is the following.

1. start flm on 172.17.0.1 (host.docker.internal)
2. inside docker devcontainer, create ~/.config/opencode/opencode.json with proper config.
3. Install and start opencode.
4. Ask it to read files, list files, and make todo lists.
5. The model eventually drifts and starts making growing errors.

Thank you! Are you on the latest version?
We tested with OpenCode and it worked well.
You might want to see the video https://youtu.be/56DSREp9RgU?si=iOWkmvIDK8-bKdOE.

Yeah. I updated immediately: 0.9.42. I am on a linux system so there could be some divergence. I will keep on my integration of llguidance for the time being.

@jtmonroe I was able to reproduce the issue!

This appears to be another corner case where the model generates an invalid schema from its chat template 😅. In your case, there was an extra "{". Another issue I encountered is that the model sometimes uses ` instead of " to terminate the JSON arguments, which can also cause problems.

I’ve submitted a PR to improve the JSON sanity checks for these corner cases. You can try the artifacts here and see if they resolve the issue.

Feel free to report any other issues you encounter — your feedback is always appreciated!

Sounds good. Will do.

[jtmonroe]

Downloaded and built your changes. Got some more errors for you. The first is with the write tool in opencode. The rust curly braces do not seem to be escaped.

        {
            "content": "tool\nCould...e endings.",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": "{\"tool\":\"edit\",\"error\":\"Invalid input for tool edit: JSON parsing failed: Text: {\\\"filePath\\\":\\\"src/main.rs\\\",\\\"newString\\\":\\\"use crate::task_model::{Task, TaskStatus};\\\\n\\\\n// Centralized task management structure for state (to satisfy Task application: Implement state management)\\\\npub struct TaskManager {\\\\n    tasks: std::sync::Arc<std::sync::Mutex<Vec<Task>>>,\\\\n}\\\\n\\\\nimpl TaskManager {\\\\n    pub fn new() -> Self {\\\\n        TaskManager {\\\\\\\"tasks\\\\\\\": std::sync::Arc::new(std::sync::Mutex::new(Vec::new())) }\\\\n    }\\\\n\\\\npub impl TaskManager {\\\\n    // Method to add a task, called from the UI callback handler\\\\n    pub fn add_task(&self,\\\\\\\"description\\\\\\\": String) {\\\\n        let mut tasks = self.tasks.lock().unwrap();\\\\n        let new_id = tasks.len() + 1;\\\\n        tasks.push(Task {\\\\\\\"id\\\\\\\":\\\\\\\"new_id as u32\\\",\\\"description\\\":\\\"description\\\",\\\"status\\\": TaskStatus::\\\"Pending \\\"});\\\\n    }\\\\n\\\\n    // Method to retrieve tasks for UI binding (simplified for now)\\\\n    pub fn get_tasks(&self) -> Vec<Task> {\\\\n        self.tasks.lock().unwrap().clone()\\\\n    }\\\\n}\\\\n\\\",\\\\\\\"oldString\\\\\\\":\\\\\\\"use std::error::Error;\\\\n\\\\nslint::include_modules!();\\\\n\\\\nfn main() -> Result<(), Box<dyn Error>> {\\\\n    let ui = AppWindow::new()?;\\\\n    // Existing setup logic that we are replacing to incorporate state management.\\\\n    ui.on_request_increase_value({ ... }); \\\\n    ui.run()?; \\\\n    Ok(())\\\\n}\\\"}.\\nError message: JSON Parse error: Unexpected identifier \\\"TaskStatus\\\"\"}",
                        "name": "invalid"
                    },
                    "id": "call_1778715663_15",
                    "type": "function"
                }
            ]
        },

and the second is the question tool with the same "double curly" at the front:

        {
            "content": "",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": "{\"tool\":\"question\",\"error\":\"Invalid input for tool question: JSON parsing failed: Text: {{\\\"questions\\\":[{{\\\"header\\\":\\\"Data Persistence?\\\",\\\"multiple\\\":false,\\\"options\\\":[{\\\"label\\\":\\\"No, in-memory only (Recommended)\\\",\\\"description\\\":\\\"State will reset on restart.\\\"},{\\\"label\\\":\\\"Use file storage (e.g., JSON/DB)\\\",\\\"description\\\":\\\"Tasks should persist between sessions.\\\"}],\\\"question\\\":\\\"Do you require tasks to be saved and reloaded after the application closes?\\\"},{{\\\"header\\\":\\\"Task Model Detail?\\\",\\\"multiple\\\":false,\\\"options\\\":[{\\\"label\\\":\\\"Basic:\\\\\\\"ID\\\\\\\", Description, Status only (Recommended)\\\",\\\"description\\\":\\\"Minimal fields for MVP.\\\"},{\\\"label\\\":\\\"Advanced: Add creation timestamp/due date\\\",\\\"description\\\":\\\"More granular tracking needed from the start.\\\"}],\\\"question\\\":\\\"What level of detail should the initial Task model contain beyond description and status?\\\"}]}]}.\\nError message: JSON Parse error: Expected '}'\"}",
                        "name": "invalid"
                    },
                    "id": "call_1778717897_16",
                    "type": "function"
                }
            ]
        },

That said the Todo task is working more reliably now.

[jtmonroe]

If needs be, I can export my entire session where I got the errors. I have it as a markdown file on my device.