Custom LLM Template addition support for the AI Workspace

[Tharanidk]

Problem

Currently, AI Workspace supports a fixed set of built-in LLM provider templates (OpenAI, Mistral, Gemini, Azure OpenAI, Azure AI Foundry, AWS Bedrock, Anthropic). Any LLM that is not on this list such as DeepSeek, Kimi, Groq or Novita, cannot be onboarded through the UI.

Proposed Solution

Template creation in the UI:

1. Add a "Create" entry under the custom templates section in the sidebar settings

2. A form collects the template definition:

   • Basic: name, description
   • Endpoint & Auth: endpoint URL (required), auth type, auth header, value prefix
   • Token & Model Mapping (optional): prompt/completion/total/remaining token identifiers + locations, request/response model identifiers
   • OpenAPI Spec URL (optional): for surfacing resources in the UI

3. On submit, the UI calls POST /llm-provider-templates.

4. Deploy the template to the gateway, the template's configuration must reach the data plane so a provider referencing it can deploy

5. The new template becomes selectable on the Add LLM Provider screen; a provider is created from it via POST /llm-providers with template:

Example: Adding DeepSeek Template

Template (POST /llm-provider-templates):

{
  "id": "deepseek",
  "name": "DeepSeek",
  "metadata": {
    "endpointUrl": "https://api.deepseek.com",
    "auth": { "type": "apiKey", "header": "Authorization", "valuePrefix": "Bearer " }
  },
  "promptTokens": { "identifier": "$.usage.prompt_tokens", "location": "payload" },
  "completionTokens": { "identifier": "$.usage.completion_tokens", "location": "payload" },
  "totalTokens": { "identifier": "$.usage.total_tokens", "location": "payload" },
  "requestModel": { "identifier": "$.model", "location": "payload" },
  "responseModel": { "identifier": "$.model", "location": "payload" }
}

API Endpoints

All endpoints are administrator-only and JWT-authenticated; organization scope is derived from the JWT — the same pattern as the gateway and LLM provider endpoints. These already exist; this feature consumes them from the UI.

Method	Path	Description
POST	/llm-provider-templates	Create a custom template
GET	/llm-provider-templates	List templates (built-in + custom)
GET	/llm-provider-templates/:id	Get a template by id
PUT	/llm-provider-templates/:id	Update a custom template
DELETE	/llm-provider-templates/:id	Delete — blocked if referenced by a provider

Discussion points

• What happens when a template is edited or deleted while providers depend on it?
  (block delete while in use / snapshot into provider / versioning?)

[Tharanidk]

Meeting Notes 17/06/2026

Attendees: Krishan Wijesena ,Pubudu Gunatilaka,Nuwan Dias, Anusha Jayasundara, Arshardh Ifthikar, Tharani Karunathilaka

• Default (built-in) provider templates are read-only but we need to show them in the template listing.
• Template versioning should be implemented. A new version can be created from an existing template and appears as a new card.
• The control plane does not push templates to the gateway. For deployment, the system generates a YAML from the given details and then user can download the YAML file to deploy that to gateway running a CURL.
• Pagination in Add LLM Provider: after 9 templates, show a "2+" style card that expands/scales on click.
• Resource-wise token & model override (per-endpoint) should be implemented..