LM Studio Chat Interface: A Mini Framework for Agentic Tool Calling

This project provides a terminal-based chat interface for interacting with a local language model (LLM) server running on LM Studio. More than just a chat interface, it serves as a mini framework for building agentic tool calling capabilities, enabling the LLM to execute predefined Python functions (tools) based on conversation context. This open-source project (MIT License) welcomes contributors to make it more robust and expand its potential.

About LM Studio

LM Studio is a powerful platform for running local language models, making AI accessible to everyone without the need for cloud dependency. Over the past year, LM Studio has made remarkable progress, evolving from a simple local LLM runner to a comprehensive environment for AI experimentation and application development. Their team, a dedicated group of innovators based in Silicon Valley, has been instrumental in pushing the boundaries of what's possible with local AI, fostering a community of developers and researchers who are building the future of agentic AI.

Project Structure

• llmchat.py: Main script for running the chat interface with tool calling capabilities.
• create_sales_database.py: Script to create a sample SQLite database for sales data (used by database tools).
• tools/: Directory for tool modules that the LLM can call.
  • math_operations.py: Basic math operations like multiplication.
  • web_requests.py: Tools for making HTTP requests.
  • database_operations.py: Tools for querying a sample sales database.
  • file_operations.py: Tools for reading file content from the local system.
  • json_operations.py: Tools for reading and processing JSON files, particularly for structured data like accounting data.
• templates/: Directory with example scripts for guidance on creating new tools.
  • multiply_tool_example.py: Example script for creating a new tool.
• examples/: Directory with sample files for testing.
  • sample_text.txt: Sample text file for testing file operations.
  • sample.json: Sample JSON file with structured accounting data for testing JSON operations.
• install.sh: Installation script to set up the environment and dependencies.
• requirements.txt: Lists the Python dependencies.
• product_sales.db: SQLite database file for storing sample sales data (generated by running create_sales_database.py).
• .gitignore: Specifies files and directories to be ignored by Git, such as virtual environments and temporary files.

Author

Name: Magnus Froste
Contact: GitHub Profile

Prerequisites

• Python 3.6 or higher: Ensure Python is installed on your system.
• LM Studio: Installed and running a server on localhost:1234 with a compatible model loaded (e.g., Qwen2.5-7B-Instruct-GGUF).

Installation

Automated Installation (Recommended)

Run the interactive installation script to set up the environment and dependencies:

bash install.sh

This script will:

1. Check for Python 3.6 or higher.
2. Create and activate a virtual environment if not already done.
3. Verify if the LM Studio server is running on localhost:1234.
4. Install required dependencies from requirements.txt.
5. Create a sample sales database if it doesn't exist.

Manual Installation

If you prefer manual setup:

1. Clone the Repository (if applicable):

   git clone <repository-url>
   cd <repository-directory>

2. Set Up Virtual Environment:

   python3 -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install Dependencies:

   pip install -r requirements.txt

4. Create Sample Sales Database (optional):

   python create_sales_database.py

5. Ensure LM Studio Server is Running:
   • Start LM Studio and load a model (e.g., Qwen2.5-7B-Instruct-GGUF).
   • Confirm the server is accessible at http://localhost:1234.

Usage

Run the chat interface with:

venv/bin/python3 llmchat.py

• Chat: Type your message and press Enter to send it to the model.
• Exit: Type exit and press Enter to stop the chat session.
• Tool Calling: The model may call predefined functions based on your input. For example, asking "What's the total of 5 and 10?" might trigger a call to add_numbers from math_operations.py, and the model will respond with the result (e.g., "The total of 5 and 10 is 15.").
• File Operations Test: To test the File System Operations tool, ask the model to read a file, for example, "Can you read the content of the file at 'examples/sample_text.txt'?". Use relative paths from the project directory (do not start with a '/'). The model should use the read_file_content function from file_operations.py to retrieve and display the content. You can then ask follow-up questions about the text. If no file path is specified in follow-up queries, the model will assume the last-used file or the default (examples/sample_text.txt). Ensure the file is read first if starting a new session.
• JSON Operations Test: To test the JSON Operations tool, ask the model to read a JSON file, for example, "Can you read the content of the JSON file at 'examples/sample.json'?". Use relative paths from the project directory. The model should use the read_json_file function from json_operations.py to retrieve and display the structured data, such as accounting information. You can then ask for analysis or specific details from the data. If no file path is specified in follow-up queries, the model will assume the last-used file or the default (examples/sample.json).

Tool Interaction Examples

This section provides examples and guidance on how to interact with the available tools in the LM Studio Chat Interface. Each tool has specific use cases, and crafting effective queries can enhance the accuracy and relevance of the responses. Below are descriptions, example questions, and tips for each tool.

Math Operations (math_operations.py)

• Description: Handles basic mathematical operations like multiplication. Useful for quick calculations.
• Example Queries:
  • "What is 7 multiplied by 8?"
  • "Can you calculate 15 times 3 for me?"
• Tips: Be clear about the numbers and operation you want (e.g., multiplication). The tool currently supports multiplying two numbers, so structure your question accordingly.

Web Requests (web_requests.py)

• Description: Makes HTTP GET requests to specified URLs and returns the response. Useful for fetching online data or content.
• Example Queries:
  • "Can you fetch the content from 'https://example.com'?"
  • "What is the response from 'https://api.example.com/data'?"
• Tips: Provide the full URL including the protocol (http:// or https://). If the response is lengthy, it may be truncated, so ask for specific parts if needed (though the tool currently returns a snippet).

Database Operations (database_operations.py)

• Description: Queries a sample sales database to retrieve information like monthly sales, product lists, or top expensive sales. Useful for business data analysis.
• Example Queries:
  • "What were the total sales for January 2025?"
  • "Can you list all sold products from the database?"
  • "Show me the top 3 most expensive product sales."
• Tips: Specify the time frame (e.g., month in 'YYYY-MM' format) or limit (e.g., top 3) when relevant. For monthly sales, ensure the format is correct or let the model convert named months (like 'January').

File Operations (file_operations.py)

• Description: Reads content from local text files and returns it as a string. Useful for accessing and discussing file content.
• Example Queries:
  • "Can you read the content of the file at 'examples/sample_text.txt'?"
  • "What does the text say about the LM Studio Team?" (assuming the last-used or default file)
• Tips: Always specify the file path using relative paths from the project directory (e.g., 'examples/sample_text.txt') for the first query. For follow-up questions, you can omit the path, and the model will assume the last-used file or default (examples/sample_text.txt). Ensure the file is read first if starting a new session.

JSON Operations (json_operations.py)

• Description: Reads and processes structured data from JSON files, such as accounting data. Useful for detailed analysis of formatted information.
• Example Queries:
  • "Can you read the content of the JSON file at 'examples/sample.json'?"
  • "What is the closing balance for account 1930?" (assuming the last-used or default file)
  • "Can you summarize the liability accounts?" (assuming the last-used or default file)
  • "Compare the opening and closing balances for assets in 'examples/sample.json'." (specifying file if different from last-used)
• Tips: Start with the file path to anchor the query for the first request. For follow-up queries, you can omit the path, and the model will assume the last-used file or default (examples/sample.json). For general overviews, a broad question like summarizing data is fine. For targeted analysis, mention specific accounts, sections (e.g., 'liabilities'), or transactions. Be precise with account numbers or categories for detailed insights.

These examples should help you interact effectively with the tools. Adjust the specificity of your questions based on whether you need a broad summary or detailed analysis. If a tool doesn’t cover your specific need, the model can suggest alternative approaches.

System Prompt

The system prompt used to initialize the chat with the LLM is as follows:

You are a helpful assistant that can use tools to answer questions. If you don't know the answer, you can search for information. Always use the provided tools to assist with queries. When you decide to use a tool, clearly inform the user by stating which tool you are calling and for what purpose before invoking it. For example, say 'I will call the [tool name] tool to [purpose].' If a tool is not available for a specific task, inform the user and suggest an alternative approach.

This prompt ensures that the model leverages the available tools to provide accurate and helpful responses.

Tool Calling: Building Agentic AI with LM Studio

Tool calling is a transformative approach to making LLMs more interactive and functional, turning them into agentic systems capable of performing tasks beyond mere text generation. However, it relies on several moving parts that must work in harmony:

1. The LLM Model: The language model itself is the core component. It must be capable of understanding user intent and recognizing when a tool should be called. Models like Qwen2.5-7B-Instruct-GGUF, which are fine-tuned for instruction-following, are particularly suited for tool calling, but their effectiveness depends on their training data and inherent capabilities.

2. The System Prompt: A well-crafted system prompt is crucial. It sets the context for the LLM, instructing it on how to behave and when to use tools. The prompt must clearly define the LLM's role as an assistant that can leverage tools, ensuring it prioritizes tool usage over speculative answers when appropriate.

3. Defined Functions (Tools): These are the Python functions that the LLM can call. Each tool must be meticulously declared with a clear name, description, and parameter schema. For example, in this framework, tools like multiply_numbers or get_sales_data_by_month are defined with JSON schemas that specify their purpose and required inputs. This structured declaration helps the LLM understand what each tool does and how to invoke it correctly.

Context-Augmented Generation (CAG) for Precise Data Retrieval

In this framework, we emphasize the retrieval of external data through a method we call Context-Augmented Generation (CAG), rather than the more common Retrieval-Augmented Generation (RAG). While RAG often focuses on enriching responses with retrieved information that may be subject to interpretation or summarization, CAG prioritizes delivering precise, unaltered data directly from the source. This approach is crucial when handling sensitive or critical information, such as accounting data, where accuracy is paramount. With CAG, the LLM acts as a conduit to fetch and present external content—whether from files, databases, or web sources—ensuring that the information remains lossless and true to the original. For example, when analyzing financial records from a JSON file like examples/sample.json, the model retrieves and displays the exact figures and context without paraphrasing or risking data distortion, allowing users to trust the integrity of the response.

Understanding Tool Calling Decisions and Metadata

One of the most intricate aspects of tool calling is how the LLM decides whether to invoke a tool, which tool to select, or when to respond without using a tool. This decision-making process relies heavily on metadata—structured information provided to the model about available tools, user intent, and contextual cues. Below, we break down this process to help users understand and learn from this proof of concept (PoC):

• Tool Calling Decision Process:

  • User Query Analysis: When a user inputs a query, the LLM first analyzes the intent and content of the request. For example, a query like "Multiply 5 by 3" contains clear numerical and operational keywords that suggest a mathematical task, prompting the model to consider a relevant tool.
  • Tool Metadata Matching: The LLM refers to the metadata defined for each tool in llmchat.py (see the tools list). This metadata includes the tool's name, description, and parameters. A well-crafted description, such as "Multiply two numbers and return the result" for multiply_numbers, helps the model match the user’s intent to the appropriate tool. If no tool matches the query's intent (e.g., a casual greeting like "Hello"), the LLM may decide not to call a tool and respond directly.
  • System Prompt Guidance: The system prompt in llmchat.py (found in the main() function) provides overarching instructions to the LLM. It explicitly guides the model to use tools for specific tasks and to inform the user when a tool is called. For instance, our prompt states, "Always use the provided tools to assist with queries," encouraging tool usage when relevant metadata aligns with the query.
  • Contextual Memory: The LLM maintains a conversation history (the messages list in llmchat.py). This context can influence tool calling decisions. For example, if a user previously asked to read a file and follows up with "Summarize it," the model can infer from the conversation context and metadata (like the last-used file tracked in last_file_paths) that the read_file_content tool or a related analysis is still relevant.
  • Decision Outcome: Ultimately, the LLM decides to call a tool if the query aligns with a tool's metadata and system instructions. If the query is ambiguous or outside the scope of defined tools, the model may respond without invoking a tool, often explaining why (e.g., "I don’t have a tool for that, but I can suggest an alternative approach").

• Why Metadata is Crucial:

  • Precision in Tool Selection: Metadata acts as a blueprint for the LLM. Without clear tool names, descriptions, and parameter definitions, the model might misinterpret user intent or fail to call the correct tool. In llmchat.py, each tool’s metadata is meticulously structured to map directly to potential user queries.
  • User Intent Clarification: Metadata bridges the gap between human language and programmatic functions. For instance, the description field for read_json_file ("Read and return content from a JSON file for analysis, such as accounting data") tells the LLM that queries involving JSON or accounting data should trigger this tool.
  • Reducing Decision Errors: Poor or vague metadata can lead to incorrect tool calls or missed opportunities. Enhancing metadata with specific keywords and use cases (as seen in our tools list) minimizes these errors, making the LLM’s decision-making more reliable.

• Learning from this PoC:

  • To make this project a learning space, we’ve annotated llmchat.py with detailed comments where tool metadata is defined (look for // TOOL_METADATA markers). These markers highlight how metadata influences the LLM’s decisions, allowing users to see the direct link between a tool’s description and the model’s actions.
  • Users are encouraged to experiment with queries of varying clarity and observe the LLM’s decisions. For example, compare the outcomes of asking "What’s in my JSON file?" versus "Read the JSON at 'examples/sample.json'." Note whether a tool was called, which one, and how the metadata or context might have influenced that choice.

• Challenges in Decision-Making:

  • Ambiguity: If a user query is vague (e.g., "Help with data"), the LLM might not find a clear match in the metadata and opt not to call a tool. Improving query specificity or metadata descriptions can mitigate this.
  • Overlapping Tools: If multiple tools have similar metadata, the LLM might struggle to pick the most relevant one. We address this by ensuring distinct descriptions and parameter requirements.
  • Training Limitations: The LLM’s training data affects its ability to interpret intent. Even with perfect metadata, some models may inconsistently decide to use tools, which is why we use instruction-tuned models like Qwen2.5-7B-Instruct-GGUF.

This section aims to demystify tool calling, showing users how metadata, user input, and system design interplay to drive the LLM’s decisions. By studying the code comments and experimenting with queries, you can gain clarity on achieving desired outcomes through effective tool interaction.

Challenges in Tool Calling

Despite the potential of tool calling, LLMs face significant challenges in selecting the right function based on user queries. These challenges include:

• Knowledge Limitations: LLMs may not always have up-to-date or comprehensive knowledge about the tools available or the specific context of a user's request. This can lead to incorrect tool selection or failure to recognize when a tool should be used.
• Awareness of Capabilities: An LLM must be aware of what it can assist with through tool calling. If the system prompt or tool descriptions are unclear, the model might not realize a tool is relevant to a query, resulting in suboptimal responses.
• Ambiguity in User Intent: User queries can be vague or ambiguous, making it difficult for the LLM to decide which tool, if any, to call. Improving tool descriptions and training models to ask clarifying questions can mitigate this issue.

This mini framework aims to address these challenges by providing a clear structure for tool definition and system prompts, but it requires ongoing refinement. Contributors are encouraged to enhance tool discovery mechanisms, improve prompt engineering, and integrate more advanced models to make agentic tool calling more robust.

Example Tools

This framework already includes several tools for demonstration:

• Math Operations: Simple calculations like multiplication through multiply_numbers.
• Sales Data Queries: Tools to interact with a sample SQLite database, including:
  • get_sales_data_by_month: Retrieve sales data for a specific month in 'YYYY-MM' format.
  • list_all_sold_products: Returns a list of all unique products sold, showing units sold and total revenue.
  • get_top_expensive_products: Retrieves the top most expensive individual sales.
  • list_available_tools: Allows users to see all supported functions by asking, 'What functions do you support?' or 'List available tools'.

Extending with More Functions (Tool Calling)

You can extend this project by adding more functions that the LLM can call as tools. Here's how to add a new tool for making an HTTP request:

1. Define the New Function: Add a new function in web_requests.py to handle HTTP requests. For example:

   import requests
   
   def make_http_request(url: str) -> str:
       """Make an HTTP GET request to the specified URL and return the response text."""
       try:
           response = requests.get(url)
           response.raise_for_status()
           print(f"HTTP request to {url} successful")
           return response.text[:500] + "..." if len(response.text) > 500 else response.text
       except Exception as e:
           print(f"Error making HTTP request to {url}: {str(e)}")
           return f"Error: {str(e)}"

2. Add the Tool to the Tools List: Update the tools list to include the new function:

   tools = [
       {
           "type": "function",
           "function": {
               "name": "multiply_numbers",
               "description": "Multiply two numbers and return the result.",
               "parameters": {
                   "type": "object",
                   "properties": {
                       "a": {
                           "type": "number",
                           "description": "The first number"
                       },
                       "b": {
                           "type": "number",
                           "description": "The second number"
                       }
                   },
                   "required": ["a", "b"]
               }
           }
       },
       # Add your new tool here
   ]

3. Register the Function for Tool Calling: Ensure the function is registered so the LLM can call it during the chat:

   def execute_tool(tool_call):
       tool_name = tool_call.function.name
       arguments = json.loads(tool_call.function.arguments)
       
       if tool_name == "multiply_numbers":
           return multiply_numbers(arguments["a"], arguments["b"])
       elif tool_name == "make_http_request":
           return make_http_request(arguments["url"])
       # Add more elif conditions for additional tools
       else:
           return f"Tool {tool_name} not found"
   
   # Get the final response from the model after tool execution
   final_message = chat_with_model(messages)
   messages.append(final_message)
   print(f"Assistant (after HTTP request): {final_message.content}")

4. Install Additional Dependencies: If your new function requires additional libraries (like requests for HTTP requests), install them in your virtual environment:

   source venv/bin/activate  # On macOS/Linux
   # OR
   venv\Scripts\activate  # On Windows
   pip install requests

5. Update System Message: Optionally, update the system message to inform the model about the new capability:

   messages = [
       {"role": "system", "content": "You are a helpful assistant that can use tools to answer questions. If you don't know the answer, you can search for information. Always use the provided tools to assist with the queries. If a tool is not available for a specific task, inform the user and suggest an alternative approach."}
   ]

Now, when interacting with the model, you can ask it to fetch content from a URL, for example: "Can you get information from https://api.example.com/data?" and it will call the make_http_request function to retrieve the data.

Remember to handle errors appropriately in your functions and limit response sizes if necessary to avoid overwhelming the model with too much data.

Troubleshooting

• LM Studio Server Not Detected:
  • Ensure LM Studio is installed and running.
  • Check if the server is accessible at http://localhost:1234 using a browser or curl http://localhost:1234.
  • If using a different port, modify the base_url in llmchat.py.
• Installation Issues:
  • Verify Python version with python3 --version.
  • Ensure virtual environment is activated before running pip install.
• EOF Error:
  • Run the script in an interactive terminal session to provide input.

Contributing

Feel free to submit issues or pull requests for improvements to the chat interface or additional tools. As an open-source project under the MIT License, we welcome contributions to enhance this mini framework for agentic tool calling with LM Studio.

AI Coding Tool Instructions

This section provides structured guidance for AI coding tools (e.g., Windsurf, Cursor, Copilot, Cline, Roo) to assist developers in contributing to this project. The goal is to streamline the process of adding new tools, updating the tool calling mechanism, and improving LLM selection and usage of tools. Follow these instructions to ensure accurate and efficient code modifications.

General Guidelines

• Purpose: This project is a mini framework for agentic tool calling with LM Studio, enabling LLMs to execute predefined Python functions based on user input.
• Target Audience: Instructions are for AI coding tools assisting human developers forking this repository.
• Compatibility: Ensure compatibility with Python 3.6+, LM Studio running on localhost:1234, and models like Qwen2.5-7B-Instruct-GGUF.
• Feedback: If instructions are unclear, AI tools should prompt the user for clarification or suggest improvements to this section.

Adding a New Tool

• Directive: ADD_TOOL
• Objective: Add a new callable function that the LLM can use via the tool calling mechanism.
• Steps:
  1. Create Tool File:
     • Location: Place the new tool file in the tools/ directory.
     • Naming: Use a descriptive name (e.g., web_search.py for a web search tool).
     • Structure: Follow the format of existing tools like file_operations.py. Define functions with clear docstrings, type hints, and error handling.
     • Example: Refer to templates/multiply_tool_example.py for a basic template.
  2. Update llmchat.py:
     • Location: /Users/mafr/Code/lmstudio/llmchat.py
     • Import: Add an import statement for the new tool function near the top of the file, e.g., from tools.web_search import search_web.
     • Tool Definition: Add a new entry to the tools list with the function's name, description, and parameters. Follow the format of existing tool definitions like read_file_content.
     • Tool Handling: Update the handle_tool_call function to process calls to the new tool, returning formatted results. Follow the pattern of existing elif blocks.
  3. Update Documentation:
     • Location: /Users/mafr/Code/lmstudio/README.md
     • Project Structure: Add the new tool under the tools/ section with a brief description.
     • Usage Example: Optionally, add a usage example under the Usage section showing how to invoke the new tool.
• Validation: After adding the tool, test it by running venv/bin/python3 llmchat.py and asking the LLM to use the new function.

Updating Tool Calling Mechanism

• Directive: UPDATE_MECHANISM
• Objective: Improve how tools are selected and called by the LLM, enhancing accuracy and context awareness.
• Steps:
  1. Modify System Prompt:
     • Location: /Users/mafr/Code/lmstudio/llmchat.py
     • Section: Update the system message in the main() function to provide clearer instructions to the LLM on when and how to call tools. Enhance guidance on tool selection based on user intent.
  2. Enhance Tool Descriptions:
     • Location: /Users/mafr/Code/lmstudio/llmchat.py
     • Section: Refine descriptions in the tools list to be more precise and descriptive, helping the LLM match tools to user queries effectively.
  3. Improve Tool Handling Logic:
     • Location: /Users/mafr/Code/lmstudio/llmchat.py
     • Section: Update the handle_tool_call function or add pre-processing logic before tool calls to validate inputs or provide fallback responses.
• Validation: Test changes by running venv/bin/python3 llmchat.py and verifying that the LLM selects and uses tools more accurately for varied user inputs.

Project Structure for Reference

• Core Files:
  • llmchat.py: Main script where tools are defined and called. Primary file for tool integration.
  • tools/: Directory for tool modules. Add new tool files here.
  • templates/: Directory with example scripts for tool creation guidance.
  • examples/: Directory with sample files for testing.
  • install.sh: Installation script to set up the environment and dependencies.
  • requirements.txt: Lists the Python dependencies.
  • product_sales.db: SQLite database file for storing sample sales data (generated by running create_sales_database.py).
  • .gitignore: Specifies files and directories to be ignored by Git, such as virtual environments and temporary files.

Notes for AI Tools

• Precision: Always specify exact file paths and function names when suggesting code changes to avoid ambiguity.
• Context Awareness: Understand the user's intent and the project's goal of being a mini framework for agentic tool calling when proposing additions or modifications.
• Iterative Development: If a task is complex, break it into smaller steps and confirm with the user before proceeding to the next step.

We encourage AI coding tools to use these instructions to assist developers in forking and enhancing this project. If you have suggestions for improving this section, please prompt the user to update it.

License

This project is licensed under the MIT License - see the LICENSE file for details.