LLM-Proxy

A robust HTTP proxy server built with FastAPI designed specifically for collecting fine-tuning data at the LLM level, especially for CLINE and Roo coder. This proxy provides request/response logging, error handling, and supports both streaming and non-streaming responses.

Purpose

This project facilitates the collection of high-quality data for fine-tuning large language models. It works by capturing the raw API requests and responses, which can then be processed into training data formats.

Features

• Full Request/Response Logging: All requests and responses are logged to files for debugging and analysis
• Streaming Support: Handles streaming responses (like SSE) efficiently
• Robust Error Handling: Comprehensive error handling with detailed logging
• JSON Pretty Printing: Logs JSON responses in a readable format
• Configurable: Simple configuration options for host, port and logging directory

Requirements

• Python 3.7+
• FastAPI
• Uvicorn
• httpx

Installation

1. Clone the repository:

   git clone https://github.com/JensCaasen/llm-proxy.git
   cd llm-proxy

2. Install dependencies:

   pip install -r requirements.txt

Usage

Setting Up

To use this proxy for data collection:

1. Go to your application's Settings
2. Use Custom Base URLs
3. Add http://localhost:5002?p=(existing URL here) to the URL
4. Keep all other credentials and configurations the same

For example, if your current API endpoint is https://api.openai.com, you would change it to:

http://localhost:5002?p=https://api.openai.com

Starting the Server

Run the proxy server:

./run_llm_proxy.sh

or

python proxy_server.py

By default, the server runs on localhost:5002 and logs to the logs/ directory.

Data Collection Process

The process consists of two phases:

1. Data Collection Phase: The proxy intercepts and logs all API requests and responses when you use your application.
2. Post-Processing Phase: The logged data is processed into a usable format for fine-tuning.

Processing Collected Data

Use the parse_log.py script to process the raw logged data:

python parse_log.py logs/

This script parses the raw data into a usable format, supporting:

• OpenAI and Anthropic API formats
• Both streaming and non-streaming responses

Each log file is processed and a corresponding -parsed.txt file is created containing the extracted text content.

How It Works

1. The proxy receives an incoming request with the p parameter specifying the upstream URL
2. It logs the request details to a uniquely numbered request file
3. The request is forwarded to the upstream URL
4. The response is streamed back to the client and simultaneously logged
5. All logs are stored in the configured logs directory

Logging

The proxy logs each request/response pair to the logs/ directory with the following naming convention:

• {request_num}-request.json: The request body
• {request_num}-response.json: The response body (if it's valid JSON)
• {request_num}-response.txt: The response body (if it's not valid JSON)
• {request_num}-response-error-*.txt: Error information if the request fails

Configuration

You can modify the following variables in proxy_server.py:

LOGS_DIR = "logs"  # Directory for logs
HOST = "localhost"  # Host to bind to
PORT = 5002  # Port to listen on