Streaming Support for Ayushma Responses #7
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Required backend change: coronasafe/ayushma#101
This pull request significantly enhances the user experience by adding real-time streaming support for Ayushma responses. Instead of waiting for the AI to generate a complete response, users will now be able to see each token as it arrives from OpenAI. This provides a more interactive and engaging experience, as users can follow the AI's thought process in real time.
To achieve this functionality, The
@microsoft/fetch-event-sourcepackage has been integrated, which efficiently processes the incoming event source response stream. This package is designed specifically to handle streaming data, ensuring smooth and reliable delivery of tokens to the user interface.The core changes have been made in the
requestmethod within theapi.tsfile. This method has been updated to support a newstream=trueoption, which enables the streaming functionality. Additionally, anonMessagecallback has been introduced that receives anEventSourceMessage. This callback is responsible for handling individual tokens as they arrive from the AI, ensuring that they are displayed to the user in a timely manner.Overall, this enhancement greatly improves the responsiveness and interactivity of the system, providing users with a more engaging and dynamic experience when interacting with Ayushma (AI) responses.
ayushma_streaming.mp4