Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🔧 [Lang API] Track what if the chat chunk is the final one #161

Closed
roma-glushko opened this issue Feb 25, 2024 · 0 comments · Fixed by #179
Closed

🔧 [Lang API] Track what if the chat chunk is the final one #161

roma-glushko opened this issue Feb 25, 2024 · 0 comments · Fixed by #179
Assignees
Labels
api:http good first issue Good for newcomers help wanted Extra attention is needed model:language size:small type:enhancement New feature or request

Comments

@roma-glushko
Copy link
Member

roma-glushko commented Feb 25, 2024

Expose the status of the chunk in the stream: if it's the final chunk that closed the whole stream or not:

// TODO: use objectpool here
chatResponse := schemas.ChatResponse{
ID: completionChunk.ID,
Created: completionChunk.Created,
Provider: providerName,
Cached: false,
ModelName: completionChunk.ModelName,
ModelResponse: schemas.ModelResponse{
SystemID: map[string]string{
"system_fingerprint": completionChunk.SystemFingerprint,
},
Message: schemas.ChatMessage{
Role: completionChunk.Choices[0].Delta.Role,
Content: completionChunk.Choices[0].Delta.Content,
},
},
// TODO: Pass info if this is the final message
}
responseC <- chatResponse

@roma-glushko roma-glushko self-assigned this Feb 25, 2024
@roma-glushko roma-glushko changed the title 🔧 [Lang] Track what if the chat chunk is the final one 🔧 [Lang API] Track what if the chat chunk is the final one Feb 25, 2024
@roma-glushko roma-glushko added good first issue Good for newcomers help wanted Extra attention is needed labels Feb 25, 2024
@roma-glushko roma-glushko removed their assignment Feb 25, 2024
@roma-glushko roma-glushko self-assigned this Mar 13, 2024
@roma-glushko roma-glushko added this to the Glide: Public Preview milestone Mar 13, 2024
roma-glushko added a commit that referenced this issue Mar 17, 2024
… finish reason field & added metadata to stream chat response
@roma-glushko roma-glushko linked a pull request Mar 17, 2024 that will close this issue
roma-glushko added a commit that referenced this issue Mar 19, 2024
- Separated chat & chat stream request schemas 
- introduced a new finish reason field 
- added metadata to stream chat response
- allow to attach some metadata to a chat stream request and then attach it to each chat stream chunk
- adjusted error message schema to include request ID and metadata
roma-glushko added a commit that referenced this issue Apr 7, 2024
A new release candidate that includes the new streaming chat functionality.

### Added

- ✨Streaming Chat Workflow #149 #163 #161 (@roma-glushko)
- ✨Streaming Support for Azure OpenAI #173 (@mkrueger12)
- ✨Cohere Streaming Chat Support #171 (@mkrueger12)
- ✨Start counting token usage in Anthropic Chat #183 (@roma-glushko)
- ✨Handle unauthorized error in health tracker #170 (@roma-glushko)

### Fixed

- 🐛 Fix Anthropic API key header #183 (@roma-glushko)

### Security

-  🔓 Update crypto lib, golang, fiber #148 (@roma-glushko)

### Miscellaneous

-  🐛 Update README.md to fix helm chart location #167 (@arjunnair22)
- 🔧 Updated .go-version (@roma-glushko)
-  ✅ Covered the telemetry by tests #146 (@roma-glushko)
- 📝 Separate and list all supported capabilities per provider #190 (@roma-glushko)
roma-glushko added a commit that referenced this issue Apr 16, 2024
## Summary

✨ Bringing support for streaming chat in Glide (integrated with OpenAI, Azure OpenAI and Cohere)
✨ Started handling 401 errors to mark models as premaritally unavailable (e.g. when API key was not correct)
🐛 Fixing the panic related to swagger.yaml file
🐛 Fixing Anthropic chat workflow by passing API key correctly
🔧 Improved Cohere param config and validation

## Changelog

### Added

- ✨Streaming Chat Workflow #149 #163 #161 (@roma-glushko)
- ✨Streaming Support for Azure OpenAI #173 (@mkrueger12)
- ✨Cohere Streaming Chat Support #171 (@mkrueger12)
- ✨Start counting token usage in Anthropic Chat #183 (@roma-glushko)
- ✨Handle unauthorized error in health tracker #170 (@roma-glushko)
- 🔧 #195 #196: Set router ctx in stream chunks & handle end of stream in case of some errors (@roma-glushko)
- 🐛🔧 #197: Handle max_tokens & content_filtered finish reasons across OpenAI, Azure and Cohere (@roma-glushko)

## Changed

- 🔧 💥 #198: Expose more Cohere params & fixing validation of provider params in config (breaking change) (@roma-glushko)
- 🔧 #186: Rendering Durations in a human-friendly way (@roma-glushko)

### Fixed

- 🐛 Fix Anthropic API key header #183 (@roma-glushko)
- 🐛 #209: Embed Swagger specs into binary to fix panics caused by missing swagger.yaml file (@roma-glushko)
- 🐛 #200: Implemented a custom json per line stream reader to read Cohere chat streams correctly (@roma-glushko)

### Security

-  🔓 Update crypto lib, golang, fiber #148 (@roma-glushko)

### Miscellaneous

-  🐛 Update README.md to fix helm chart location #167 (@arjunnair22)
- 🔧 Updated .go-version (@roma-glushko)
-  ✅ Covered the telemetry by tests #146 (@roma-glushko)
- 📝 Separate and list all supported capabilities per provider #190 (@roma-glushko)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api:http good first issue Good for newcomers help wanted Extra attention is needed model:language size:small type:enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant