Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Lang Chat] Implement a PoC of a async router with streaming lang API #149

Closed
roma-glushko opened this issue Feb 18, 2024 · 0 comments · Fixed by #150
Closed

[Lang Chat] Implement a PoC of a async router with streaming lang API #149

roma-glushko opened this issue Feb 18, 2024 · 0 comments · Fixed by #150
Assignees
@roma-glushko
Labels
api:http area:api model:language

Comments

@roma-glushko
Copy link
Member

In this task, I'm trying to implement the async routing workflow proposed in GEP0008. The workflow is going to be used to implement several APIs but in this PoC I will focus on the streaming chat API to unblock @mkrueger12 with actual work there.
@roma-glushko roma-glushko self-assigned this Feb 18, 2024
@roma-glushko roma-glushko changed the title [Async API] Implement a PoC with streaming lang API [Lang Chat] Implement a PoC of a async router with streaming lang API Feb 18, 2024
roma-glushko added a commit that referenced this issue Feb 18, 2024
@roma-glushko
#149: Removed OpenAPI docs for websocket endpoint
1fa271a
roma-glushko added a commit that referenced this issue Feb 18, 2024
@roma-glushko
#149: Updated the openAPI docs
746747f
@roma-glushko roma-glushko linked a pull request Feb 18, 2024 that will close this issue
🔧#149: Streaming Chat Workflow #150
Merged
roma-glushko added a commit that referenced this issue Feb 18, 2024
@roma-glushko
#149: linting
71606f3
roma-glushko added a commit that referenced this issue Feb 24, 2024
@roma-glushko
#149: Added two new methods to LangModelProvider interface: ChatStrea…
d81303d 
…m & SupportChartStream
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Inited SSE reader and processing OpenAI stream chat events
724196d
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Added logging to the streaming chat workflow
6172e99
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Fixed the payload content type to get the chat stream working
3c5e367
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Handle a case of ending the chat stream with a special marker
ced94be
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Renamed ProviderResponse into ModelResponse
02db9a0
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Reject nonJSON payloads
407a904
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Fixed the issue with chatReqTemplate mutation & fixed linting e…
6960732 
…rrors
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Moved the error handling to a sep method
c6f8ae4
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149 Covering the chatStream() method
09fe128
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Covered the support method
c17d72a
roma-glushko added a commit that referenced this issue Feb 25, 2024
@roma-glushko
#149: Covered a case of rate limiting
480158d
roma-glushko added a commit that referenced this issue Feb 28, 2024
@roma-glushko
#149: Fixed the test
4417a9e
roma-glushko added a commit that referenced this issue Feb 28, 2024
@roma-glushko
#149: Updated the OpenAI schema
c00b488
roma-glushko added a commit that referenced this issue Feb 28, 2024
@roma-glushko
#149: Allowed to mock stream support
2d970b0
roma-glushko added a commit that referenced this issue Feb 28, 2024
@roma-glushko
#149: Fixed linting error
f2482ae
roma-glushko added a commit that referenced this issue Feb 28, 2024
@roma-glushko
#149: Covered the support method by tests
20c48b8
roma-glushko added a commit that referenced this issue Mar 2, 2024
@roma-glushko
#149: Removed the broken test
1cb2ccf
roma-glushko added a commit that referenced this issue Mar 2, 2024
@roma-glushko
#149: Covered SSE by tests
f50ef15
roma-glushko added a commit that referenced this issue Mar 2, 2024
@roma-glushko
✨ #149: Streaming Chat Workflow (#150)
4edb644 
- Initing a new type of workflow, a streaming (async) routing workflow using the Streaming Chat API as an example
- Updated the Bruno collection
- Updated the LanguageModel API to include `ChatStream()` and `SupportChatStream()` methods
- Get the streaming router working
- Implemented SSE event parsing to be able to work with OpenAI streaming chat API
- Integrated OpenAI chat streaming into the Glide's streaming chat API
- Covered the happy workflow by tests
@roma-glushko roma-glushko mentioned this issue Apr 7, 2024
Merged
roma-glushko added a commit that referenced this issue Apr 7, 2024
@roma-glushko
📦 Release: v0.0.3-rc1 (#192)
5e0dd0f 
A new release candidate that includes the new streaming chat functionality.

### Added

- ✨Streaming Chat Workflow #149 #163 #161 (@roma-glushko)
- ✨Streaming Support for Azure OpenAI #173 (@mkrueger12)
- ✨Cohere Streaming Chat Support #171 (@mkrueger12)
- ✨Start counting token usage in Anthropic Chat #183 (@roma-glushko)
- ✨Handle unauthorized error in health tracker #170 (@roma-glushko)

### Fixed

- 🐛 Fix Anthropic API key header #183 (@roma-glushko)

### Security

-  🔓 Update crypto lib, golang, fiber #148 (@roma-glushko)

### Miscellaneous

-  🐛 Update README.md to fix helm chart location #167 (@arjunnair22)
- 🔧 Updated .go-version (@roma-glushko)
-  ✅ Covered the telemetry by tests #146 (@roma-glushko)
- 📝 Separate and list all supported capabilities per provider #190 (@roma-glushko)
@roma-glushko roma-glushko mentioned this issue Apr 16, 2024
Merged
roma-glushko added a commit that referenced this issue Apr 16, 2024
@roma-glushko
📦 v0.0.3 (#215)
754af34 
## Summary

✨ Bringing support for streaming chat in Glide (integrated with OpenAI, Azure OpenAI and Cohere)
✨ Started handling 401 errors to mark models as premaritally unavailable (e.g. when API key was not correct)
🐛 Fixing the panic related to swagger.yaml file
🐛 Fixing Anthropic chat workflow by passing API key correctly
🔧 Improved Cohere param config and validation

## Changelog

### Added

- ✨Streaming Chat Workflow #149 #163 #161 (@roma-glushko)
- ✨Streaming Support for Azure OpenAI #173 (@mkrueger12)
- ✨Cohere Streaming Chat Support #171 (@mkrueger12)
- ✨Start counting token usage in Anthropic Chat #183 (@roma-glushko)
- ✨Handle unauthorized error in health tracker #170 (@roma-glushko)
- 🔧 #195 #196: Set router ctx in stream chunks & handle end of stream in case of some errors (@roma-glushko)
- 🐛🔧 #197: Handle max_tokens & content_filtered finish reasons across OpenAI, Azure and Cohere (@roma-glushko)

## Changed

- 🔧 💥 #198: Expose more Cohere params & fixing validation of provider params in config (breaking change) (@roma-glushko)
- 🔧 #186: Rendering Durations in a human-friendly way (@roma-glushko)

### Fixed

- 🐛 Fix Anthropic API key header #183 (@roma-glushko)
- 🐛 #209: Embed Swagger specs into binary to fix panics caused by missing swagger.yaml file (@roma-glushko)
- 🐛 #200: Implemented a custom json per line stream reader to read Cohere chat streams correctly (@roma-glushko)

### Security

-  🔓 Update crypto lib, golang, fiber #148 (@roma-glushko)

### Miscellaneous

-  🐛 Update README.md to fix helm chart location #167 (@arjunnair22)
- 🔧 Updated .go-version (@roma-glushko)
-  ✅ Covered the telemetry by tests #146 (@roma-glushko)
- 📝 Separate and list all supported capabilities per provider #190 (@roma-glushko)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@roma-glushko roma-glushko

Labels
api:http area:api model:language
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

🔧#149: Streaming Chat Workflow EinStack/glide
1 participant
@roma-glushko