Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add streaming support to text converse endpoint #101

Merged
merged 3 commits into from
May 8, 2023
Merged

Conversation

Ashesh3
Copy link
Member

@Ashesh3 Ashesh3 commented May 8, 2023

Frontend change: ohcnetwork/ayushma_fe#7

This pull request introduces crucial updates to the backend that enable real-time streaming support for Ayushma (AI) responses. The changes ensure that users can interact with the AI in a more engaging and dynamic manner, as tokens are displayed as soon as they are received from the AI.

  • ChatOpenAI is initialized with streaming=True and with a callback_manager which receives each new token as it is received from OpenAI.
  • Django's StreamingHttpResponse is initialized with content_type="text/event-stream" and the LLM's output is streamed to the client in the below format.
  • Employed start_blocking_portal to initiate the lang_chain_helper.get_response coroutine, which retrieves the AI responses in real time and populates the token_queue accordingly.
  • Each token is added to a queue in the StreamingQueueCallbackHandler which is then processed to create a streaming json output.
  • Implemented a loop that continuously reads tokens from the token_queue, appending them to the chat_response string and yielding an updated JSON response for each token. The loop terminates when the RESPONSE_END token is encountered.
  • Format of message: {"chat": chat_id, "delta": delta, "message": message, "stop": stop} where delta is the new token received and message is the entire message streamed so far.
  • Minor fixes such as fixing a logical issue with split_text and refactoring the code to use ChatMessageType instead of hardcoded numbers are also made.

These backend enhancements work in tandem with the frontend updates to provide a seamless, real-time streaming experience for Ayushma (AI) responses, significantly improving user engagement and interactivity.

@shivankacker shivankacker merged commit 7cae173 into master May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants