feat: streaming AI upgrade with cloud-first provider failover - #1
Merged
Conversation
…page - Swap model from Qwen3.5-4B (2.7GB, slow) to Qwen2.5-Coder-1.5B (1.6GB, 3x faster) - Add streaming SSE endpoint (/api/chat/stream) for real-time token display - Add Groq as primary cloud AI backend (sub-1s responses, free tier) - Keep Ollama as local fallback for privacy/offline use - Expand intent system to cover more patterns instantly (no LLM needed) - Reduce context overhead (30 cells max, shorter system prompt, 4 exchanges) - Add landing page for smartsht.com (static HTML, self-contained) - Add nginx config for production deployment - Add PM2 ecosystem config for server process management - Update homepage to smartsht.com
Improve production chat reliability by adding configurable provider order with OpenRouter, Hugging Face, Groq, and Ollama fallbacks, and document new environment variables for deployment.
Add VS Code Snyk auto-organization setting and persist the package license field in the lockfile for consistent local tooling behavior.
Reviewer's GuideImplements streaming AI responses over SSE, adds a cloud-first multi-provider LLM routing layer with failover to local Ollama, significantly expands intent fast-path handling, and introduces a marketing landing page plus deployment/config docs for smartsht.com. Sequence diagram for streaming chat with cloud-first LLM failoversequenceDiagram
actor User
participant ChatPanel
participant useStore
participant AgentClient as chatWithAgentServerStream
participant Api as Express_index_ts
participant Router as callProviderStream
participant OpenRouter as chatWithOpenAiCompatibleStream
participant Groq as chatWithGroqStream
participant Ollama as chatWithOllamaStream
User->>ChatPanel: submit message
ChatPanel->>useStore: sendAIMessage
useStore->>AgentClient: chatWithAgentServerStream(message, context, history, onToken)
AgentClient->>Api: POST /api/chat/stream
Api->>Api: resolveIntent
alt template fast path
Api-->>AgentClient: SSE data:{type:'complete', source:'template'}
AgentClient->>useStore: on complete
useStore->>ChatPanel: update messages
else LLM path
Api->>Api: providerOrder, providerIsConfigured
loop providers in order
Api->>Router: callProviderStream(provider, messages)
alt OpenRouter/HuggingFace
Router->>OpenRouter: chatWithOpenAiCompatibleStream
OpenRouter-->>Router: tokens
else Groq
Router->>Groq: chatWithGroqStream
Groq-->>Router: tokens
else Ollama
Router->>Ollama: chatWithOllamaStream
Ollama-->>Router: tokens
end
Router-->>Api: fullText
Api-->>AgentClient: SSE data:{type:'token', content}
AgentClient->>useStore: onToken(token)
useStore->>ChatPanel: append streaming assistant content
end
Api-->>AgentClient: SSE data:{type:'complete', source:'llm'}
AgentClient->>useStore: final ServerChatResponse
useStore->>ChatPanel: replace streaming message with final
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (21)
📝 WalkthroughWalkthroughThis PR adds multi-provider LLM support (OpenRouter, Hugging Face, Groq, Ollama) with ordered failover and SSE streaming across server and client, switches the default local model to Qwen2.5-Coder-1.5B, expands intent detection, and adds a new landing page with nginx deployment config. ChangesMulti-provider LLM Streaming Chat
Estimated code review effort: 4 (Complex) | ~60 minutes Landing Page and Deployment Config
Estimated code review effort: 2 (Simple) | ~15 minutes Sequence Diagram(s)sequenceDiagram
participant Client as useStore (sendMessage)
participant AgentClient as agentClient.ts
participant Server as /api/chat/stream
participant Provider as LLM Provider (Groq/OpenRouter/HF/Ollama)
Client->>AgentClient: chatWithAgentServerStream(history, onToken)
AgentClient->>Server: POST /api/chat/stream (SSE)
Server->>Server: check fast-path intent template
alt no fast-path match
Server->>Provider: callProviderStream(messages)
loop streamed tokens
Provider-->>Server: token chunk
Server-->>AgentClient: SSE "token" event
AgentClient-->>Client: onToken(chunk) appends to placeholder
end
Server->>Server: parse accumulated response
end
Server-->>AgentClient: SSE "complete" event (message, actions)
AgentClient-->>Client: final ServerChatResponse
Client->>Client: replace placeholder message with final response
Poem
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The SSE client parsing in
chatWithAgentServerStreamassumes eachdata:line arrives as a complete chunk split by\n; consider buffering partial lines across reads or using a small SSE parser to avoid issues when events are split across chunks. - The provider selection logic (
providerOrder,providerIsConfigured,callProvider,callProviderStream) is duplicated between streaming and non‑streaming endpoints; pulling this into a shared helper would reduce drift and make it easier to extend or adjust provider behavior in one place.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The SSE client parsing in `chatWithAgentServerStream` assumes each `data:` line arrives as a complete chunk split by `\n`; consider buffering partial lines across reads or using a small SSE parser to avoid issues when events are split across chunks.
- The provider selection logic (`providerOrder`, `providerIsConfigured`, `callProvider`, `callProviderStream`) is duplicated between streaming and non‑streaming endpoints; pulling this into a shared helper would reduce drift and make it easier to extend or adjust provider behavior in one place.
## Individual Comments
### Comment 1
<location path="models/README.md" line_range="28" />
<code_context>
+
+## Why this model?
+
+- **Speed**: 1.5B params generates tokens 3–4× faster than 4B on CPU
+- **Quality**: Qwen2.5-Coder-Instruct is specifically trained for structured output / JSON — perfect for tool calling
+- **Size**: Q8_0 quantization preserves quality while keeping the file under 1.6GB
</code_context>
<issue_to_address>
**issue (typo):** Use plural verb with "params" ("params generate" instead of "params generates").
This keeps subject–verb agreement with the plural noun "params."
```suggestion
- **Speed**: 1.5B params generate tokens 3–4× faster than 4B on CPU
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
|
||
| ## Why this model? | ||
|
|
||
| - **Speed**: 1.5B params generates tokens 3–4× faster than 4B on CPU |
There was a problem hiding this comment.
issue (typo): Use plural verb with "params" ("params generate" instead of "params generates").
This keeps subject–verb agreement with the plural noun "params."
Suggested change
| - **Speed**: 1.5B params generates tokens 3–4× faster than 4B on CPU | |
| - **Speed**: 1.5B params generate tokens 3–4× faster than 4B on CPU |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
Summary by Sourcery
Introduce streaming AI chat with multi-provider routing and cloud-first failover, plus a new public landing page and updated model configuration.
New Features:
Enhancements:
Build:
Deployment:
Documentation:
Summary by CodeRabbit
New Features
Bug Fixes
Documentation