DROID + LiteLLM Gateway

This guide describes how the Factory AI DROID CLI uses the local LiteLLM gateway to reach both Ollama (via the litellm proxy) and Ollama Cloud fallback models.

Endpoint & Authentication

• Base URL: http://127.0.0.1:4000/v1/
• LiteLLM requires a bearer token. Direnv materialises the master key into ~/.config/lab-secrets/litellm.env (LITELLM_MASTER_KEY).
• Ensure LITELLM_MASTER_KEY and LITELLM_SALT_KEY live inside secrets/global.enc.env and run source .envrc before starting DROID or the LiteLLM systemd unit.

The gateway exposes the catalogue described in ai/backend/ai-backend-unified/config/litellm-unified.yaml, including fallback chains across local, cloud, llama.cpp, and vLLM backends.

DROID Configuration Example

Add a custom model entry in ~/.factory/config.json (or a project override):

{
  "custom_models": [
    {
      "model_display_name": "qwen3-coder (LiteLLM)",
      "model": "qwen3-coder:480b-cloud",
      "base_url": "http://127.0.0.1:4000/v1/",
      "api_key": "env:LITELLM_MASTER_KEY",
      "provider": "generic-chat-completion-api",
      "max_tokens": 128000
    }
  ]
}

Use additional entries for other aliases (e.g. qwen2.5-coder:7b, llama3.1:latest) pointing at the same base URL. The env: prefix instructs DROID to pull the key from the environment and send it as Authorization: Bearer <value>.

Health & Guardrails

• Start the service with ai/services/openwebui/scripts/run_litellm.sh; it writes a systemd unit that targets ai/backend/ai-backend-unified/runtime.

• Validate redis/backends using lab/bin/check_ai_health.sh and ai/services/openwebui/scripts/validate_backends.sh before invoking DROID.

• Unauthenticated requests to /v1/* should respond with HTTP 401. Example check:

  curl -s -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
       http://127.0.0.1:4000/v1/models | jq '.data[].id'

Helper Scripts

• scripts/bootstrap.sh: copies the sample config into ~/.factory/config.d and checks for the master key.
• scripts/healthcheck.sh: validates LiteLLM auth (401/200) and optionally redis.
• scripts/exec.sh: wrapper around droid exec that ensures the master key is available.
• After a master-key rotation, rerun scripts/bootstrap.sh so ~/.factory/config.d/lab-litellm.json and ~/.config/lab-secrets/litellm.env pick up the new token.

Secrets & Rotation

• Secrets remain in the encrypted secrets/global.enc.env; never commit the plaintext master key.
• To rotate: update LITELLM_MASTER_KEY with sops --set, run source .envrc, and restart the service (systemctl --user restart litellm.service). Notify clients (including DROID configs) to reload the key.
• CI jobs invoking droid exec should inject the master key through the pipeline secret store prior to execution.

Follow-on Tasks

• Consider enabling LiteLLM encrypted persistence (requires the salt key) for logs/metrics once automation expands.
• Keep the integration roadmap aligned with the hybrid litellm-hybrid.yaml.applied profile used by OpenWebUI.