PredictiveChat

PredictiveChat is a quick Flask prototyped designed to showcase the usage of LLMs in predicting user behavior to decrease latency.

Here's a X/Twitter thread with a flowchart of how the code works.

Goal

The primary goal of PredictiveChat is to demonstrate an example of leveraging LLMs to predict the end of user messages. This prediction capability is used to preemptively generate responses, thereby decreasing the perceived latency in chat applications. It's an exploration into enhancing chat interfaces with AI-driven anticipatory features to make conversations feel instantaneous.

Features

• Real-time text prediction to anticipate the end of user sentences.
• Dynamic response generation based on predicted text completion.
• Similarity checking to avoid redundant response generation for highly similar messages.
• User interface that updates based on prediction confidence levels.

Setup

Prerequisites

• Python 3.x
• Flask
• OpenAI API Key

Installation

1. Clone the repository:

   git clone https://github.com/yoheinakajima/predictivechat.git
   

2. Navigate to the project directory:

   cd predictivechat
   

3. Install the required Python packages:

   pip install flask openai scipy numpy
   

4. Set your OpenAI API key as an environment variable:

   export OPENAI_API_KEY='your_api_key_here'
   

Running the Application

1. Start the Flask application:

   python main.py
   

2. Open a web browser and navigate to http://localhost:80 to view the application.

Usage

1. Type a message in the input box.
2. The application predicts the completion of your message and generates a response based on the predicted text.
3. If the message completion likelihood is high, the app directly displays the AI-generated response to decrease response latency.

Code Overview

main.py

main.py contains the Flask application logic, including routes for rendering the front-end and processing POST requests with message text. It utilizes OpenAI's API to predict message completions and generate responses.

templates/index.html

templates/index.html is the front-end of the application, designed to capture user input and display predictions and responses dynamically. It employs JavaScript to interact with the Flask back-end and update the UI based on prediction outcomes.

License

This project is open-sourced under the MIT License. See the LICENSE file for more details.