🔧 [Stream Router] Extend the model health & latency tracking functionality to account for streaming API #163
Assignees
Milestone
Comments
roma-glushko
added a commit
that referenced
this issue
Mar 5, 2024
…quests & removed latency from the model interface as it turned out to be a property of a specific model action
roma-glushko
added a commit
that referenced
this issue
Mar 6, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 6, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 6, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 6, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 6, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
…sponseChunk channels. The channel can contain both success and error messages/chunks. Separated stream and sync chat schemas
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 11, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 13, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 13, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 17, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 17, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 17, 2024
roma-glushko
added a commit
that referenced
this issue
Mar 17, 2024
…tStream (#166) - Separated sync and streaming chat schemas - Extracted assumptions on where to find latency from routing strategies to a separate `LatencyGetters` that can be different for different models/workflows - Elaborated the client provider `chatStream()` interface. Clients now expose a response channel instead of being provided with by caller - Connected the stream chat workflow to latency & health tracking - Refined the `chatStream()` method of clients to return a stream struct - Separated latency tracking of the streaming workflow from the sync chat workflow - defined a new `HealthTracker` to incorporate all health tracking logic
Merged
roma-glushko
added a commit
that referenced
this issue
Apr 7, 2024
A new release candidate that includes the new streaming chat functionality. ### Added - ✨Streaming Chat Workflow #149 #163 #161 (@roma-glushko) - ✨Streaming Support for Azure OpenAI #173 (@mkrueger12) - ✨Cohere Streaming Chat Support #171 (@mkrueger12) - ✨Start counting token usage in Anthropic Chat #183 (@roma-glushko) - ✨Handle unauthorized error in health tracker #170 (@roma-glushko) ### Fixed - 🐛 Fix Anthropic API key header #183 (@roma-glushko) ### Security - 🔓 Update crypto lib, golang, fiber #148 (@roma-glushko) ### Miscellaneous - 🐛 Update README.md to fix helm chart location #167 (@arjunnair22) - 🔧 Updated .go-version (@roma-glushko) - ✅ Covered the telemetry by tests #146 (@roma-glushko) - 📝 Separate and list all supported capabilities per provider #190 (@roma-glushko)
Merged
roma-glushko
added a commit
that referenced
this issue
Apr 16, 2024
## Summary ✨ Bringing support for streaming chat in Glide (integrated with OpenAI, Azure OpenAI and Cohere) ✨ Started handling 401 errors to mark models as premaritally unavailable (e.g. when API key was not correct) 🐛 Fixing the panic related to swagger.yaml file 🐛 Fixing Anthropic chat workflow by passing API key correctly 🔧 Improved Cohere param config and validation ## Changelog ### Added - ✨Streaming Chat Workflow #149 #163 #161 (@roma-glushko) - ✨Streaming Support for Azure OpenAI #173 (@mkrueger12) - ✨Cohere Streaming Chat Support #171 (@mkrueger12) - ✨Start counting token usage in Anthropic Chat #183 (@roma-glushko) - ✨Handle unauthorized error in health tracker #170 (@roma-glushko) - 🔧 #195 #196: Set router ctx in stream chunks & handle end of stream in case of some errors (@roma-glushko) - 🐛🔧 #197: Handle max_tokens & content_filtered finish reasons across OpenAI, Azure and Cohere (@roma-glushko) ## Changed - 🔧 💥 #198: Expose more Cohere params & fixing validation of provider params in config (breaking change) (@roma-glushko) - 🔧 #186: Rendering Durations in a human-friendly way (@roma-glushko) ### Fixed - 🐛 Fix Anthropic API key header #183 (@roma-glushko) - 🐛 #209: Embed Swagger specs into binary to fix panics caused by missing swagger.yaml file (@roma-glushko) - 🐛 #200: Implemented a custom json per line stream reader to read Cohere chat streams correctly (@roma-glushko) ### Security - 🔓 Update crypto lib, golang, fiber #148 (@roma-glushko) ### Miscellaneous - 🐛 Update README.md to fix helm chart location #167 (@arjunnair22) - 🔧 Updated .go-version (@roma-glushko) - ✅ Covered the telemetry by tests #146 (@roma-glushko) - 📝 Separate and list all supported capabilities per provider #190 (@roma-glushko)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: