Issue with Managing Conversation Context in AnythingLLM Using threadSlug

[G3kPLAY]

I’ve been working on a project where I’m integrating AnythingLLM with a custom WebUI that interacts with a specialized SQL assistant model (LLaMA 3.1). The setup is primarily used for generating and optimizing SQL queries in Teradata. We’re using AnythingLLM to manage different conversation threads with the model.

Current Setup:
WebUI: A custom front-end that allows users to interact with AnythingLLM.
Backend: Flask application that connects the WebUI to AnythingLLM, handling the API requests.
Model: LLaMA 3.1 model, integrated with AnythingLLM for natural language query optimization in Teradata.
The Problem:
Every time a new conversation is started, a new threadSlug is generated and passed from the WebUI to AnythingLLM. Despite correctly generating and using new threadSlugs, the responses from the model appear to be carrying over context from previous conversations.

Code Snippets:
Here’s a summary of how we’re handling conversations and threadSlugs in our routes.py:

import requests
from flask import request, jsonify
import logging
import uuid

API_KEY = "YOUR_API_KEY_HERE"  # Placeholder for the actual API key
THREAD_URL = "http://localhost:3001/api/v1/workspace/taisa/thread/new"
DELETE_THREAD_URL = "http://localhost:3001/api/v1/workspace/taisa/thread/"
CHAT_URL = "http://localhost:3001/api/v1/workspace/taisa/chat"

conversation_id = None  # Variable to store the current threadSlug

def start_new_conversation():
    global conversation_id
    previous_conversation_id = conversation_id
    conversation_id = str(uuid.uuid4())

    try:
        response = requests.post(THREAD_URL, headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        }, json={"slug": conversation_id})

        response.raise_for_status()
        data = response.json()

        if "thread" in data and "slug" in data["thread"]:
            conversation_id = data["thread"]["slug"]
            logging.debug(f'New conversation automatically started with ID: {conversation_id}')

            # Delete the previous thread if it exists
            if previous_conversation_id:
                delete_thread(previous_conversation_id)

        else:
            logging.error('Failed to start a new conversation: No conversation ID returned from API.')
            conversation_id = None

    except requests.exceptions.RequestException as e:
        logging.error(f'Error starting a new conversation with the API: {e}')
        conversation_id = None

def delete_thread(thread_slug):
    try:
        delete_url = f"{DELETE_THREAD_URL}{thread_slug}"
        response = requests.delete(delete_url, headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        })

        if response.status_code == 200:
            logging.debug(f'Thread {thread_slug} successfully deleted.')
        else:
            logging.error(f'Failed to delete thread {thread_slug}. Status code: {response.status_code}')

    except requests.exceptions.RequestException as e:
        logging.error(f'Error deleting thread {thread_slug}: {e}')

@app.route('/ask', methods=['POST'])
def ask():
    global conversation_id
    user_message = request.json.get('message')
    received_thread_slug = request.json.get('thread_slug')

    logging.debug(f'Received message: {user_message}')
    logging.debug(f'Received thread_slug: {received_thread_slug}')

    payload = {
        "message": user_message,
        "mode": "chat",
        "thread_slug": received_thread_slug
    }

    try:
        response = requests.post(CHAT_URL, headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        }, json=payload)

        response.raise_for_status()
        data = response.json()

        text_response = data.get('textResponse')
        if text_response is None:
            logging.error('No response received from the model or there was an error with the API.')
            text_response = 'Error: Could not retrieve a response from the model.'

    except requests.exceptions.RequestException as e:
        logging.error(f'Error communicating with the API: {e}')
        text_response = 'Error: Could not connect to the API.'

    logging.debug(f'Response from API: {text_response}')

    formatted_response = text_response.replace('\\n', '<br>')
    return {"response": formatted_response}

@app.route('/new_conversation', methods=['POST'])
def new_conversation():
    start_new_conversation()
    if conversation_id:
        return jsonify({"status": "New conversation started.", "conversation_id": conversation_id})
    else:
        return jsonify({"status": "Failed to start new conversation."}), 500

Observations:
threadSlug Generation: The threadSlugs are correctly generated and appear to be unique.
Context Carryover: Despite deleting old threads and ensuring new threadSlugs are used, the model seems to retain context from previous conversations, affecting the responses.

What We Need:
Guidance on Ensuring Separation of Context: How can we ensure that each new threadSlug completely isolates the conversation from any previous context?
Proper API Usage: Are there any additional steps or endpoints we should be utilizing to ensure that old contexts do not persist when starting a new conversation?
Best Practices: Any insights or best practices on managing conversation context in AnythingLLM, particularly when using threadSlugs.

Thank you in advance for your help!

[shatfield4]

@G3kPLAY I see your issue here. The context is being carried over because you are chatting with just the default thread.

You are using CHAT_URL = "http://localhost:3001/api/v1/workspace/{slug}/chat" as the endpoint for chatting. Using this endpoint will chat with the default thread on that workspace.

You need to use the http://localhost:3001/v1/workspace/{slug}/thread/{threadSlug}/chat endpoint to chat directly with a thread.