What is this project?

Streaming Chat Bot made with fast-api and Websocket.

⚠️ Under Active development

What are the fundamentals?

• You can choose a model based on your preference.
• the LLM has been called in streaming mode
• Websocket has been used for this chat-bot; so you can send events
• each chat session is stored inside postgresql, including messages.
• temperature, max_token, stop_sequence has been set.
• Token Usage per conversation is tracked.
• Websocket manager Handles Multiple users streaming at once.
• Built-in logic to handle max Token hits.
• a responsive Frontend.

Later on:

• Redis will be added for using chats and postgres for older chats.
• handle production concerns.
• use fixed-sized seed to debug

What are the Productivity Concerns?

• What happens if the connection drops mid-stream?

Want to read Architecture of the app?

• Install Excalidraw extension via your IDE
• Open excalidraw folder and read the files.

How to install dependencies?

• clone the repo
• pip install -e .

How to run the frontend UI?

• Start the FastAPI backend: uvicorn app.main:app --reload
• In a second terminal, launch the Flet app: python -m app.ui.flet_app

Why not SSE instead of Websocket?

SSE is made for one line communication, user can't send events when Ai is mid response; which means no Stop Generating which is bad.