feat: LiteLLM gateway for unified local + cloud model serving#2
Merged
Conversation
Add optional LiteLLM gateway that runs as a Docker container on the DGX, sitting in front of vLLM and cloud providers (OpenRouter, Ollama, Zen/OpenCode, Together AI) under a single OpenAI-compatible API on port 4000. - Setup wizard: optional step after vLLM to install and configure gateway - `spark gateway start|stop|status|logs`: thin Docker wrapper for lifecycle - `spark gateway add|remove`: manage providers post-setup with API keys - Auto-wires vLLM as default backend when gateway is enabled - Config stored in ~/.config/spark/gateway.json, generates litellm_config.yaml - Doctor check validates gateway status when configured - Setup descriptions: each step now shows a brief explanation of what it does
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spark gatewaysubcommand:start|stop|status|logs|add|removefor full lifecycle and provider management post-setupDetails
During
spark setup, after vLLM is configured, users are asked if they want to install the LiteLLM gateway. If yes:ghcr.io/berriai/litellm:main-lateston the DGX~/.config/spark/gateway.jsonlitellm_config.yamland starts the containerAfter setup, providers can be added/removed with
spark gateway add openrouter/spark gateway remove openrouter. Thedoctorcommand also checks gateway health when configured.Test plan
bash -n spark— syntax check passesbash tests/run.sh— 6/6 tests passspark help— gateway command visiblespark gateway— shows usage with all subcommandsspark setupon real DGX — LiteLLM prompt appears, image pulls, container starts🤖 Generated with Claude Code