Build/Implement Unified LLM API v1

[avirajsingh7]

Currently, our system is tightly coupled with OpenAI Assistants, meaning all LLM interactions rely on the assistants table and OpenAI’s assistant structure.
This makes it difficult to evolve or integrate other LLM providers and limits flexibility in configuration and control.

Goal

Create a new endpoint that decouples the LLM request flow from the assistants table and performs OpenAI model calls dynamically based on request configuration.

V1 Design Overview

New Endpoint: POST /v1/llm/call

Sample Request Body:

{
  "llm": {
    "model": "gpt-4.1",
    "temperature": 0.3
  },
  "vector_store_id": "<whatever>",
  "prompt": "<prompt_string>"
}

{
  "llm": {
    "model": "gpt-5",
    "reasoning": {
      "effort": "medium"
    },
    "text": {
      "verbosity": "medium"
    }
  },
  "vector_store_id": "<whatever>",
  "prompt": "<prompt_string>"
}

Key Points

• Focus only on OpenAI models for V1
• Remove dependency on the assistants table
• Accept a config JSON in the request to define provider/model settings
• Prepare architecture for multi-provider extensibility in future versions