feat: add custom OpenAI-compatible provider for self-hosted models by shaikh-amer · Pull Request #166 · TinyAGI/tinyagi

shaikh-amer · 2026-03-08T06:20:56Z

Summary:

Adds custom provider support for any OpenAI-compatible API endpoint (SGLang, Ollama, vLLM, LM Studio, DeepSeek, etc.), enabling self-hosted models without needing Codex CLI or Claude CLI. Tested with Qwen3-32B on AMD MI300X GPU via SGLang.

Changes:

src/lib/invoke.ts — new custom provider block using native fetch() with base_url validation, trailing-slash normalisation, conditional Authorization header, configurable AbortController timeout (timeout_ms), HTTP/JSON error handling, and model-gated <think> stripping (Qwen + DeepSeek-R1)

src/lib/types.ts — added base_url?, api_key?, and timeout_ms? fields to AgentConfig; clarified custom is per-agent only in Settings.models.provider comment

lib/agents.sh — added custom option to agent_add wizard with validated prompts for model, base_url, and optional API key; added custom case to agent_provider subcommand; cleans up base_url, api_key, and timeout_ms when switching away from custom

Testing:

Agents correctly route messages to self-hosted endpoint via Telegram

<think> blocks stripped from Qwen3-32B responses

Switching providers via agent_provider correctly cleans up custom fields

greptile-apps · 2026-03-08T06:22:31Z

Greptile Summary

This PR introduces a custom provider for any OpenAI-compatible API endpoint (SGLang, Ollama, vLLM, LM Studio, etc.), allowing self-hosted models to be used as agents without requiring Codex CLI or Claude CLI. The implementation spans invoke.ts (fetch-based API call with timeout, error handling, and model-gated <think> stripping), types.ts (new base_url, api_key, timeout_ms fields on AgentConfig), and agents.sh (wizard and agent_provider subcommand support).

The majority of issues flagged in earlier review rounds have been addressed:

HTTP non-2xx responses now throw an explicit error
response.json() is wrapped in a try/catch for malformed proxy responses
The Authorization header is now conditional (omitted when no key is configured)
base_url trailing slashes are normalised at runtime
Empty base_url and empty model name are validated at wizard time and throw errors
<think> stripping is now gated behind the model name (Qwen / DeepSeek-R only)
CUSTOM_API_KEY env-var fallback added so secrets can stay out of settings.json
The stateless / no-history limitation is documented in a comment
The Settings.models.provider comment now clearly marks custom as per-agent only

Two remaining items worth attention:

agent_provider custom silently clears a stored api_key when the user presses Enter without re-entering it during a config update (e.g. updating only base_url). A user could inadvertently lose their stored credential with no warning.
opencode is absent from the agent_provider switch statement — a pre-existing gap that this PR touches around without closing, meaning agent_provider <id> opencode still hits the error case.

Confidence Score: 3/5

Mostly safe to merge for the TypeScript layer; the bash wizard has a silent credential-deletion edge case worth fixing before wide use.
The invoke.ts changes are well-guarded (timeout, HTTP error check, JSON parse guard, conditional auth header, model-gated think-stripping). The types.ts changes are clean. The main concern is in lib/agents.sh: agent_provider custom will silently delete a previously stored api_key whenever the user doesn't re-enter it during an update, which could cause silent authentication failures that are hard to trace. The missing opencode case in agent_provider is a pre-existing gap but continues to block the round-trip workflow. Neither issue causes data loss, but both can produce confusing runtime failures.
lib/agents.sh — the agent_provider custom case and the missing opencode provider switch branch.

Important Files Changed

Filename	Overview
src/lib/invoke.ts	Adds the `custom` provider block with `fetch`-based OpenAI-compatible API invocation, `AbortController` timeout, conditional `Authorization` header, HTTP/JSON error handling, and model-gated `<think>` stripping. Previously flagged issues (HTTP error check, JSON parse error boundary, double-slash URL, Authorization header leakage, think-stripping guard) are all addressed. One minor style issue: optional chaining on `agent.model` contradicts its non-optional type declaration.
lib/agents.sh	Adds `custom` to the `agent_add` wizard (with model, base_url, and API key prompts + empty-value validation) and a new `custom` case in `agent_provider` (with interactive base_url, model, and API key capture). The anthropic/openai `agent_provider` cases now correctly clean up `base_url`, `api_key`, and `timeout_ms` on switch. Two issues: `agent_provider custom` silently deletes stored `api_key` when the user presses Enter without re-entering it, and `opencode` is still absent from the `agent_provider` switch statement.
src/lib/types.ts	Adds `base_url?`, `api_key?`, and `timeout_ms?` to `AgentConfig`; updates provider comments to list `custom` (per-agent only) and clarifies the global `models.provider` field does not support `custom`. Duplicate comment removed. Clean change with no issues.

Sequence Diagram

sequenceDiagram
    participant User as Telegram/Discord User
    participant Invoke as invokeAgent()
    participant Cfg as AgentConfig
    participant API as Self-Hosted API<br/>(SGLang / Ollama / vLLM)

    User->>Invoke: message
    Invoke->>Cfg: read provider, base_url, api_key, model, timeout_ms
    alt provider === 'custom'
        Invoke->>Invoke: validate base_url (throw if missing)
        Invoke->>Invoke: strip trailing slash from base_url
        Invoke->>Invoke: resolve apiKey (agent.api_key || CUSTOM_API_KEY || '')
        Invoke->>Invoke: start AbortController (timeout_ms, default 120s)
        Invoke->>API: POST {base_url}/chat/completions<br/>{ model, messages: [{role:'user', content: message}] }<br/>Authorization: Bearer {apiKey} (if set)
        alt HTTP error (non-2xx)
            API-->>Invoke: 4xx / 5xx
            Invoke-->>User: throw "Custom provider HTTP error: {status}"
        else non-JSON body
            API-->>Invoke: 200 OK with HTML/text
            Invoke-->>User: throw "Custom provider returned non-JSON response"
        else timeout
            Invoke-->>User: throw "Custom provider timed out after Xs"
        else success
            API-->>Invoke: { choices: [{ message: { content: "..." } }] }
            Invoke->>Invoke: extract choices[0].message.content
            alt model includes 'qwen' or 'deepseek-r'
                Invoke->>Invoke: strip <think>…</think> blocks
            end
            Invoke-->>User: cleaned response text
        end
    end

Comments Outside Diff (3)

src/lib/invoke.ts, line 212-213 (link)

Optional chaining on a required field

agent.model is declared as a required string (not string | undefined) in AgentConfig. The ?. optional chaining operator on line 212 is therefore redundant — TypeScript will never treat this field as undefined based on the declared type. While harmless at runtime (it just adds a no-op guard), it's inconsistent with the type contract and could create confusion about whether model can actually be absent.

If there are real-world situations where model may be undefined (e.g., a partially-initialised config loaded from a user-edited JSON file), the field should be typed string | undefined or string with a default value rather than papering over it with optional chaining.

Or update the type:
```
// types.ts
model?: string;  // e.g. 'sonnet', 'opus', 'gpt-5.3-codex'
```
lib/agents.sh, line 559-562 (link)

Stored api_key silently deleted when updating base_url

When a user calls agent_provider <id> custom to update only the base_url of an already-configured custom agent, the final read prompt asks for the API key with no indication that leaving it blank will delete the previously-stored key. If the user presses Enter without typing anything, new_api_key is "", and the jq filter runs del(.agents[$id].api_key) — silently removing the stored credential.

This is particularly surprising because the analogous prompt in agent_add (line 225) explicitly says "or press Enter for 'none'" and behaves symmetrically. The agent_provider prompt just says "leave blank if not required", which a user updating only their base URL might reasonably interpret as "skip this field".

Consider adding a note to the prompt, or preserving the existing key when the user leaves the field blank:
```
# Show a hint if a key is already stored
if jq -e --arg id "$agent_id" '.agents[$id].api_key // empty' "$SETTINGS_FILE" > /dev/null 2>&1; then
    read -rp "Enter API key (leave blank to keep existing key): " new_api_key
else
    read -rp "Enter API key (leave blank if not required): " new_api_key
fi
```

lib/agents.sh, line 566-574 (link)

opencode missing from agent_provider switch and usage string

The agent_provider subcommand's case statement now handles anthropic, openai, and custom — but opencode is still absent. A user who created an opencode agent (which is supported in agent_add) and tries to switch it to custom via agent_provider will succeed, but the reverse (agent_provider <id> opencode) hits the * catch-all and exits with an error referencing a usage string that doesn't mention opencode.

This gap existed before this PR, but since the PR now adds custom as a new supported case and touches this code, it would be a good opportunity to add the opencode case for consistency:

opencode)
    if [ -n "$model_arg" ]; then
        jq --arg id "$agent_id" --arg model "$model_arg" \
            'del(.agents[$id].base_url) | del(.agents[$id].api_key) | del(.agents[$id].timeout_ms) | .agents[$id].provider = "opencode" | .agents[$id].model = $model' \
            "$SETTINGS_FILE" > "$tmp_file" && mv "$tmp_file" "$SETTINGS_FILE"
        echo -e "${GREEN}✓ Agent '${agent_id}' switched to OpenCode with model: ${model_arg}${NC}"
    else
        jq --arg id "$agent_id" \
            'del(.agents[$id].base_url) | del(.agents[$id].api_key) | del(.agents[$id].timeout_ms) | .agents[$id].provider = "opencode"' \
            "$SETTINGS_FILE" > "$tmp_file" && mv "$tmp_file" "$SETTINGS_FILE"
        echo -e "${GREEN}✓ Agent '${agent_id}' switched to OpenCode${NC}"
    fi
    ;;

_{Last reviewed commit: f05aca5}

src/lib/invoke.ts

shaikh-amer · 2026-03-08T14:30:26Z

@greptile review

src/lib/invoke.ts

src/lib/types.ts

src/lib/invoke.ts

shaikh-amer · 2026-03-08T14:39:39Z

@greptile review

src/lib/invoke.ts

…mment

shaikh-amer · 2026-03-08T14:44:58Z

@greptile review

lib/agents.sh

shaikh-amer · 2026-03-08T14:52:01Z

@greptile review

lib/agents.sh

shaikh-amer · 2026-03-08T15:01:03Z

@greptile review

lib/agents.sh

fix: replace redundant AGENT_MODEL init with comment for custom provider Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

shaikh-amer · 2026-03-08T17:00:31Z

@greptile review

lib/agents.sh

src/lib/invoke.ts

shaikh-amer · 2026-03-08T17:21:11Z

@greptile review

src/lib/invoke.ts

…er when no key

shaikh-amer · 2026-03-08T17:29:12Z

@greptile review

src/lib/types.ts

shaikh-amer · 2026-03-08T19:11:21Z

@greptile review

…nt_provider custom

shaikh-amer · 2026-03-08T19:54:08Z

@greptile review

…n blank in agent_provider

shaikh-amer · 2026-03-08T20:01:43Z

@greptile review

shaikh-amer · 2026-03-08T20:08:22Z

@greptile review

…n --model omitted

shaikh-amer · 2026-03-08T20:16:53Z

@greptile review

…deepseek-r

shaikh-amer · 2026-03-08T20:24:57Z

@greptile review

…_provider custom

shaikh-amer · 2026-03-08T20:34:49Z

@greptile review

… responses

shaikh-amer · 2026-03-08T20:43:15Z

@greptile review

…sentinel in agent_provider

shaikh-amer · 2026-03-08T21:00:34Z

@greptile review

shaikh-amer · 2026-03-08T21:03:50Z

@jlia0 This PR has gone through 20+ commits addressing all issues flagged by greptile - error handling, auth header, trailing slash, timeout, JSON guard, think-strip, wizard validation, credential cleanup. Tested live with Qwen3-32B on AMD MI300X via SGLang. Ready for review!

jlia0 · 2026-03-09T08:31:31Z

thank you for the contribution, but this won't work because it's only making an LLM call instead of executing an agent, no?

shaikh-amer · 2026-03-09T14:03:37Z

thank you for the contribution, but this won't work because it's only making an LLM call instead of executing an agent, no?

Hi @jlia0, good catch! I'll rework it to follow the same CLI execution pattern as the anthropic/openai providers. Would ollama CLI be the right executor for self-hosted models, or do you have a preferred approach?

jlia0 · 2026-03-09T14:15:53Z

thank you for the contribution, but this won't work because it's only making an LLM call instead of executing an agent, no?

Hi @jlia0, good catch! I'll rework it to follow the same CLI execution pattern as the anthropic/openai providers. Would ollama CLI be the right executor for self-hosted models, or do you have a preferred approach?

you can just set the base url and model envs for codex cli and you can use any model with any provider

jlia0 · 2026-03-09T17:19:11Z

closing as I have implemented the custom provider #178

shaikh-amer · 2026-03-09T19:58:24Z

@jlia0 Thanks for implementing this — the final design with tinyclaw provider list/add/remove and custom: syntax is much cleaner. Happy to have contributed to the idea! 🙌

feat: add custom OpenAI-compatible provider for self-hosted models

2b87778