Skip to content

The behavior of gemma4-it:e4b running in FastFlowLM and llama.cpp is very different: the model running in FFLM does not use the appropriate tools #559

Description

@antrv

I'm running FLM serve command on laptop with Windows 11 x64 and Ryzen AI 7 360 Pro.

flm serve gemma4-it:e4b -p 1234 --ctx-len 65536 --q-len 20 --socket 20 --cors 0

I run a simple C# console application which uses OpenAI API and define tools like get_time, list_directory, read_file, write_file, edit_file as well as system prompt to use skills in .skills subdirectory. I have .skills/task-manager/SKILL.md file with the description of the skill to manage TODO list in tasks.md file.

I enter the following user prompt: I need to buy bread, eggs and milk. Add the task to the todo list.

The behavior with FLM:

You: I need to buy bread, eggs and milk. Add the task to the todo list

Model: I can certainly help you manage your tasks, but I do not have a built-in "todo list" tool or file system access to maintain one unless you specify where it should be saved.

If you would like me to add this task to a specific file (e.g., `todo.txt`), please let me know the path, and I can use the `write_file` tool for you!

The behavior with llama.cpp (the command is .\llama-server -hf ggml-org/gemma-4-E4B-it-GGUF --jinja -c 0 --host 127.0.0.1 --port 1234):

You: I need to buy bread, eggs and milk. Add the task to the todo list

  → list_directory({"path":".skills"})
  ← {
  "directory": ".skills",
  "subdirectories": [
    "task-manager"
  ],
  "files": [],
  "message": "If you don\u0027t see the file you need, check subdirectories."
}

  → read_file({"path":".skills/task-manager/SKILL.md"})
  ← **SKILL.md content**

  → read_file({"path":"tasks.md"})
  ← { "error": "File not found: tasks.md" }

  → write_file({"content":"# My Tasks\n- [ ] Buy bread, eggs and milk","path":"tasks.md"})
  ← { "status": "success", "message": "File saved to tasks.md." }

Model: I have added "Buy bread, eggs and milk" to your todo list in `tasks.md`.

.skills/task-manager/SKILL.md file:

---
name: task-manager
description: Manages a daily todo list, adding new tasks or marking existing ones as complete with a timestamp.
---

# Task Manager Skill

You are responsible for managing the user's task list located at `tasks.md`.

## Workflow: Marking a Task as Complete
If the user asks to complete, finish, or check off a task, follow these EXACT steps:

1. **Get Context**: Call `read_file(path="tasks.md")` to see the current tasks.
   * *Fallback*: If the file does not exist, tell the user there are no tasks to complete.
2. **Get Time**: Call `get_time()` so you know when this task was completed.
3. **Execute Update**: Call `edit_file` to update the task.
   * `search_text`: The exact line of the uncompleted task (e.g., `- [ ] Fix the C++ memory leak`)
   * `replacement_text`: The same line, but with an `x` and the current date/time appended (e.g., `- [x] Fix the C++ memory leak (Completed: Thursday, October 26)`)

## Workflow: Adding a New Task
If the user asks to add a new task, follow these EXACT steps:

1. **Check State**: Call `read_file(path="tasks.md")` to see the current file.
2. **Create if Missing**: If the file does not exist, use `write_file` to create `tasks.md` with the content:
   # My Tasks
   - [ ] New Task Name
3. **Append if Exists**: If the file *does* exist, use `edit_file` to add the task. 
   * *Hint for Appending*: Find the last task in the list to use as your `search_text`. 
   * `replacement_text`: The last task, followed by a newline `\n`, followed by the new task `- [ ] Your new task`.

## Constraints
- ALWAYS use `read_file` before `edit_file` so you know the exact `search_text` to use.
- Do not make up tasks; only use what the user provided.

C# source code (.NET 10):

using Microsoft.Extensions.Logging;
using System.ClientModel;
using System.ClientModel.Primitives;
using System.Reflection;
using System.Text.Json;
using OpenAI;
using OpenAI.Chat;
using OpenAI.Models;

#pragma warning disable OPENAI001

Init.LoggingOptions.EnableLogging = true;
Init.LoggingOptions.EnableMessageLogging = true;
Init.LoggingOptions.EnableMessageContentLogging = false;

OpenAIClient client = Init.CreateClient();
List<string> models = await Init.GetAvailableModels(client);
if (models.Count == 0)
{
    Console.WriteLine("No models found.");
    return;
}

Init.PrintModels(models);
string model = "gemma4-it:e4b"; //models[0];
Console.WriteLine($"Using model: {model}");

ChatClient chatClient = client.GetChatClient(model);
ChatCompletionOptions options = new()
{
    Temperature = 0.1f, // Reduced for tool use consistency; 0.8 is often too "creative" for JSON tool calling
    Tools =
    {
        ChatTool.CreateFunctionTool(
            functionName: "get_time",
            functionDescription: "Returns the current time. Use this when the user's request is time-sensitive or relates to scheduling."),

        ChatTool.CreateFunctionTool(
            functionName: "list_directory",
            functionDescription: "Lists files/folders. REQUIRED for 'Skill Discovery': check the '.skills' folder to see what specialized skills are available.",
            functionParameters: BinaryData.FromString("""
            {
                "type": "object",
                "properties": {
                    "path": { "type": "string", "description": "Path to list (e.g. '.skills' or './project')" }
                },
                "required": ["path"]
            }
            """)),

        ChatTool.CreateFunctionTool(
            functionName: "read_file",
            functionDescription: "Reads file content. REQUIRED for 'Skill Activation': read the '.skills/<name>/SKILL.md' file to learn how to perform a specific task.",
            functionParameters: BinaryData.FromString("""
            {
                "type": "object",
                "properties": {
                    "path": { "type": "string", "description": "Relative path to the file." }
                },
                "required": ["path"]
            }
            """)),

        ChatTool.CreateFunctionTool(
            functionName: "write_file",
            functionDescription: "Creates a new file or overwrites an existing one with new content.",
            functionParameters: BinaryData.FromString("""
            {
                "type": "object",
                "properties": {
                    "path": { "type": "string", "description": "Path where the file should be saved." },
                    "content": { "type": "string", "description": "The full text content to write." }
                },
                "required": ["path", "content"]
            }
            """)),

        // Search and Replace pattern is easier for 4B models than line numbers
        ChatTool.CreateFunctionTool(
            functionName: "edit_file",
            functionDescription: "Updates a specific section of a file using a search-and-replace pattern.",
            functionParameters: BinaryData.FromString("""
            {
                "type": "object",
                "properties": {
                    "path": { "type": "string", "description": "Path to the file to edit." },
                    "search_text": { "type": "string", "description": "The exact string currently in the file that needs to change." },
                    "replacement_text": { "type": "string", "description": "The new text to put in place of the search_text." }
                },
                "required": ["path", "search_text", "replacement_text"]
            }
            """))
    },
};

const string systemPrompt =
    """
    # Role and Environment
    You are an autonomous AI orchestration agent running in a local environment. 
    Your primary function is to fulfill user requests by effectively using your available tools and 
    discovering specialized workflows called "Skills."
    
    # Available Tools & Their Usage
    You have access to the following tools. Never hallucinate tools outside of this list.
    - `get_time`: Use this if the user asks for the current date/time, or if a task requires scheduling context.
    - `list_directory`: Use this to explore folders. **CRITICAL:** Use this on the `.skills` directory if you need to discover available specialized workflows.
    - `read_file`: Use this to read file contents. **CRITICAL:** Use this to read `.skills/<skill_name>/SKILL.md` files to learn how to execute a specific skill.
    - `write_file`: Use this to create entirely new files or completely overwrite existing ones.
    - `edit_file`: Use this to modify a specific section of an existing file using a search-and-replace text match. 
    
    # The "Skill Discovery" Workflow (Progressive Disclosure)
    You do not know everything upfront. If the user asks for a complex task, a specific company workflow, or a specialized process, you MUST follow this sequence:
    1. **Discover:** Call `list_directory` on the path `.skills`. 
    2. **Identify:** Look at the returned folder names. Choose the one that best matches the user's request.
    3. **Learn:** Call `read_file` on `.skills/<chosen_folder>/SKILL.md`.
    4. **Execute:** Read the instructions provided in the `SKILL.md` file and use your file t
    """;

List<ChatMessage> history =
[
    new SystemChatMessage(systemPrompt),
];

Console.WriteLine("Chat with the model.");
Console.WriteLine("Print 'exit' to exit.\n");
while (true)
{
    Console.ForegroundColor = ConsoleColor.DarkCyan;
    Console.Write("You: ");
    string? input = Console.ReadLine()?.Trim();
    Console.ResetColor();
    if (string.IsNullOrWhiteSpace(input))
        continue;

    if (input == "exit")
    {
        Console.WriteLine("Bye!");
        break;
    }

    history.Add(new UserChatMessage(input));

    while (true)
    {
        ClientResult<ChatCompletion> response = await chatClient.CompleteChatAsync(history, options);
        ChatCompletion message = response.Value;
        history.Add(new AssistantChatMessage(message));

        if (message.FinishReason == ChatFinishReason.ToolCalls)
        {
            foreach (ChatToolCall? call in message.ToolCalls)
            {
                Console.ForegroundColor = ConsoleColor.Yellow;
                Console.WriteLine($"  → {call.FunctionName}({call.FunctionArguments})");
                Console.ResetColor();

                string result = DispatchTool(call.FunctionName, call.FunctionArguments.ToString());

                Console.ForegroundColor = ConsoleColor.DarkGray;
                Console.WriteLine($"  ← {result[..Math.Min(300, result.Length)]}");
                Console.ResetColor();

                history.Add(new ToolChatMessage(call.Id, result));
            }

            continue;
        }

        Console.ForegroundColor = ConsoleColor.Green;

        foreach (ChatMessageContentPart contentPart in message.Content)
            Console.WriteLine($"Model: {contentPart.Text}\n");

        Console.ResetColor();

        Console.WriteLine($"Input token count: {message.Usage.InputTokenCount}, " +
            $"output token count: {message.Usage.OutputTokenCount}");

        break;
    }
}

static string DispatchTool(string name, string argsJson)
{
    try
    {
        // Some local models wrap JSON in extra braces or markdown blocks
        argsJson = argsJson.Trim().Trim('`').Replace("json", "");
        if (argsJson.StartsWith("{{") && argsJson.EndsWith("}}"))
        {
            argsJson = argsJson.Substring(1, argsJson.Length - 2);
        }

        using JsonDocument doc = JsonDocument.Parse(argsJson);
        JsonElement args = doc.RootElement;

        return name switch
        {
            "get_time" => GetTime(),
            "list_directory" => Listing(args.GetProperty("path").GetString()!),
            "read_file" => ReadFile(args.GetProperty("path").GetString()!),
            "write_file" => WriteFile(args.GetProperty("path").GetString()!, args.GetProperty("content").GetString()!),
            "edit_file" => EditFile(
                args.GetProperty("path").GetString()!,
                args.GetProperty("search_text").GetString()!,
                args.GetProperty("replacement_text").GetString()!),
            _ => $"{{ \"error\": \"Unknown tool: {name}\" }}"
        };
    }
    catch (Exception ex)
    {
        return $"{{ \"error\": \"Exception using tool {name}: {ex.Message}\" }}";
    }
}

// Ensures the requested path is inside the BasePath to prevent directory traversal.
static string GetValidatedPath(string inputPath)
{
    string basePath = Utils.GetWorkspacePath();

    string fullPath = Path.GetFullPath(Path.Combine(basePath, inputPath));

    if (!fullPath.StartsWith(basePath, StringComparison.OrdinalIgnoreCase))
    {
        throw new UnauthorizedAccessException(
            "Access Denied: You cannot access files outside of the workspace directory.");
    }

    return fullPath;
}

static string GetTime()
{
    DateTime now = DateTime.Now;
    return $"{{ \"current_date\": \"{now:D}\", \"current_time\": \"{now:T}\", \"timezone\": \"{TimeZoneInfo.Local.DisplayName}\" }}";
}

static string Listing(string path)
{
    try
    {
        string targetPath = GetValidatedPath(path);
        if (!Directory.Exists(targetPath))
            return $"{{ \"error\": \"Directory not found: {path}\" }}";

        // Get relative paths to keep the model's context clean
        IEnumerable<string> dirs = Directory.EnumerateDirectories(targetPath).
            Select(p => Path.GetRelativePath(targetPath, p));

        IEnumerable<string> files = Directory.EnumerateFiles(targetPath).
            Select(p => Path.GetRelativePath(targetPath, p));

        return JsonSerializer.Serialize(new
        {
            directory = path,
            subdirectories = dirs,
            files = files,
            message = "If you don't see the file you need, check subdirectories.",
        }, new JsonSerializerOptions { WriteIndented = true });
    }
    catch (Exception ex)
    {
        return $"{{ \"error\": \"{ex.Message}\" }}";
    }
}

static string ReadFile(string path)
{
    try
    {
        string targetPath = GetValidatedPath(path);
        return File.Exists(targetPath)
            ? File.ReadAllText(targetPath)
            : $"{{ \"error\": \"File not found: {path}\" }}";
    }
    catch (Exception ex)
    {
        return $"{{ \"error\": \"{ex.Message}\" }}";
    }
}

static string WriteFile(string path, string content)
{
    try
    {
        string targetPath = GetValidatedPath(path);

        string? directory = Path.GetDirectoryName(targetPath);
        if (directory != null && !Directory.Exists(directory))
            Directory.CreateDirectory(directory);

        File.WriteAllText(targetPath, content);
        return $"{{ \"status\": \"success\", \"message\": \"File saved to {path}.\" }}";
    }
    catch (Exception ex)
    {
        return $"{{ \"error\": \"{ex.Message}\" }}";
    }
}

static string EditFile(string path, string searchText, string replacementText)
{
    try
    {
        string targetPath = GetValidatedPath(path);
        if (!File.Exists(targetPath))
            return $"{{ \"error\": \"File not found: {path}\" }}";

        string content = File.ReadAllText(targetPath);

        if (!content.Contains(searchText))
        {
            return
                $"{{ \"error\": \"The exact search_text was not found in {path}. Use 'read_file' to confirm the exact content before editing.\" }}";
        }

        string newContent = content.Replace(searchText, replacementText);
        File.WriteAllText(targetPath, newContent);

        return $"{{ \"status\": \"success\", \"message\": \"Successfully replaced text in {path}.\" }}";
    }
    catch (Exception ex)
    {
        return $"{{ \"error\": \"{ex.Message}\" }}";
    }
}

public static class Init
{
    public static readonly ClientLoggingOptions LoggingOptions = new()
    {
        EnableLogging = true,
        LoggerFactory = new MyLoggerFactory(),
        EnableMessageContentLogging = true,
        EnableMessageLogging = true,
        MessageContentSizeLimit = int.MaxValue,
    };

    public static readonly OpenAIClientOptions ClientOptions = new()
    {
        Endpoint = new Uri("http://localhost:1234/v1"),
        NetworkTimeout = TimeSpan.FromMinutes(10),
        ClientLoggingOptions = LoggingOptions,
    };

    public static OpenAIClient CreateClient() => new(new ApiKeyCredential("lm-studio"), ClientOptions);

    public static async Task<List<string>> GetAvailableModels(OpenAIClient client)
    {
        OpenAIModelCollection? models = (await client.GetOpenAIModelClient().GetModelsAsync())?.Value;
        return models is null ? [] : models.Select(m => m.Id).ToList();
    }

    public static void PrintModels(List<string> models)
    {
        Console.WriteLine($"Available {models.Count} model(s):");
        foreach (string model in models)
            Console.WriteLine(model);
    }
}

public sealed class MyLoggerFactory: ILoggerFactory
{
    public void Dispose()
    {
    }

    public ILogger CreateLogger(string categoryName) => new MyLogger();

    public void AddProvider(ILoggerProvider provider)
    {
    }
}

public sealed class MyLogger: ILogger
{
    public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
        Func<TState, Exception?, string> formatter)
    {
        ConsoleColor color = logLevel switch
        {
            LogLevel.Trace => ConsoleColor.Gray,
            LogLevel.Debug => ConsoleColor.DarkGray,
            LogLevel.Information => ConsoleColor.White,
            LogLevel.Warning => ConsoleColor.Yellow,
            LogLevel.Error => ConsoleColor.Red,
            LogLevel.Critical => ConsoleColor.DarkRed,
            _ => ConsoleColor.White,
        };

        ConsoleColor oldColor = Console.ForegroundColor;
        Console.ForegroundColor = color;

        object? newState = state;
        if (newState is IReadOnlyList<KeyValuePair<string, object?>> stateProperties)
        {
            string json = string.Empty;
            int entryIndex = -1;
            for (int index = 0; index < stateProperties.Count; index++)
            {
                KeyValuePair<string, object?> entry = stateProperties[index];
                if (entry is { Key: "content", Value: string jsonString })
                {
                    json = GetFormattedJson(jsonString);
                    entryIndex = index;
                    break;
                }
            }

            if (entryIndex >= 0)
            {
                FieldInfo? field = typeof(TState).GetField($"_value{entryIndex}",
                    BindingFlags.GetField | BindingFlags.Instance | BindingFlags.NonPublic);

                field?.SetValue(newState, json);
            }
        }

        Console.WriteLine(formatter((TState)newState!, exception));

        Console.ForegroundColor = oldColor;
    }

    public bool IsEnabled(LogLevel logLevel) => true;

    public IDisposable BeginScope<TState>(TState state)
        where TState: notnull =>
        new Scope();

    private sealed class Scope: IDisposable
    {
        public void Dispose()
        {
        }
    }

    private static string GetFormattedJson(string jsonString)
    {
        try
        {
            using JsonDocument doc = JsonDocument.Parse(jsonString);
            string formatted = JsonSerializer.Serialize(
                doc.RootElement,
                new JsonSerializerOptions { WriteIndented = true }
            );

            return formatted;
        }
        catch (JsonException)
        {
            return jsonString;
        }
    }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions