[Bug]: Running the chatbot customization tutorial doesn't load context data

[robgon-art]

Bug Description

Hello, great project! I was able to run the Starter Tutorial just fine on a Google Colab running llama_index v0.8.23.post1 using my OpenAI API token.

But when I tried to run the Customization Tutorial called "I want a chatbot instead of Q&A," it didn't seem to load any context from the index, and the response was wrong.

Am I missing something? Or is there a bug with loading a context using the chat engine?

Version

0.8.23.post1

Steps to Reproduce

First, here's my code that works using query_engine.query()

!mkdir data
!wget https://raw.githubusercontent.com/jerryjliu/llama_index/main/examples/paul_graham_essay/data/paul_graham_essay.txt -P data

import os
os.environ["OPENAI_API_KEY"] = 'my-openai-key'

from llama_index import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents)

import textwrap
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(textwrap.fill(response.response, width=150, replace_whitespace=False))

# Output:
# The author worked on writing and programming outside of school before college. They wrote short stories and tried writing programs on an IBM 1401
# computer. They also built a microcomputer kit and started programming on it, writing simple games and a word processor.

The above worked fine. But this doesn't seem to work using query_engine.chat()

query_engine = index.as_chat_engine()
response = query_engine.chat("What did the author do growing up?")
print(textwrap.fill(response.response, width=150, replace_whitespace=False))
print()
response = query_engine.chat("Oh interesting, tell me more.")
print(textwrap.fill(response.response, width=150, replace_whitespace=False))

# Output:
# I'm sorry, but I don't have access to personal information about the author.
# 
# I apologize for the confusion. As an AI language model, I don't have real-time access to personal information about individuals unless it has been
# shared with me in the course of our conversation. I can provide general information and answer questions based on my training, but I don't have
# specific details about the author's personal life or experiences. Is there anything else I can help you with?

Relevant Logs/Tracbacks

For reference, here are the versions of packages that were installed from running `!pip install llama-index`


Collecting llama-index
  Downloading llama_index-0.8.23.post1-py3-none-any.whl (786 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 786.5/786.5 kB 5.3 MB/s eta 0:00:00
Collecting tiktoken (from llama-index)
  Downloading tiktoken-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 9.9 MB/s eta 0:00:00
Collecting dataclasses-json (from llama-index)
  Downloading dataclasses_json-0.6.0-py3-none-any.whl (27 kB)
Collecting langchain>=0.0.262 (from llama-index)
  Downloading langchain-0.0.285-py3-none-any.whl (1.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 15.6 MB/s eta 0:00:00
Requirement already satisfied: sqlalchemy>=2.0.15 in /usr/local/lib/python3.10/dist-packages (from llama-index) (2.0.20)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from llama-index) (1.23.5)
Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (8.2.3)
Collecting openai>=0.26.4 (from llama-index)
  Downloading openai-0.28.0-py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.5/76.5 kB 7.6 MB/s eta 0:00:00
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from llama-index) (1.5.3)
Collecting urllib3<2 (from llama-index)
  Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 10.6 MB/s eta 0:00:00
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (2023.6.0)
Collecting typing-inspect>=0.8.0 (from llama-index)
  Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-index) (4.5.0)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from llama-index) (4.11.2)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from llama-index) (1.5.7)
Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from llama-index) (3.8.1)
Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain>=0.0.262->llama-index) (6.0.1)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain>=0.0.262->llama-index) (3.8.5)
Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain>=0.0.262->llama-index) (4.0.3)
Collecting dataclasses-json (from llama-index)
  Downloading dataclasses_json-0.5.14-py3-none-any.whl (26 kB)
Collecting langsmith<0.1.0,>=0.0.21 (from langchain>=0.0.262->llama-index)
  Downloading langsmith-0.0.35-py3-none-any.whl (37 kB)
Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in /usr/local/lib/python3.10/dist-packages (from langchain>=0.0.262->llama-index) (2.8.5)
Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain>=0.0.262->llama-index) (1.10.12)
Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain>=0.0.262->llama-index) (2.31.0)
Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json->llama-index)
  Downloading marshmallow-3.20.1-py3-none-any.whl (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.4/49.4 kB 5.8 MB/s eta 0:00:00
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from openai>=0.26.4->llama-index) (4.66.1)
Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from sqlalchemy>=2.0.15->llama-index) (2.0.2)
Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.8.0->llama-index)
  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->llama-index) (2.5)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->llama-index) (8.1.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk->llama-index) (1.3.2)
Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk->llama-index) (2023.6.3)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-index) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->llama-index) (2023.3.post1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.262->llama-index) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.262->llama-index) (3.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.262->llama-index) (6.0.4)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.262->llama-index) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.262->llama-index) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain>=0.0.262->llama-index) (1.3.1)
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.10/dist-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index) (23.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->llama-index) (1.16.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain>=0.0.262->llama-index) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain>=0.0.262->llama-index) (2023.7.22)
Installing collected packages: urllib3, mypy-extensions, marshmallow, typing-inspect, tiktoken, openai, langsmith, dataclasses-json, langchain, llama-index
  Attempting uninstall: urllib3
    Found existing installation: urllib3 2.0.4
    Uninstalling urllib3-2.0.4:
      Successfully uninstalled urllib3-2.0.4
Successfully installed dataclasses-json-0.5.14 langchain-0.0.285 langsmith-0.0.35 llama-index-0.8.23.post1 marshmallow-3.20.1 mypy-extensions-1.0.0 openai-0.28.0 tiktoken-0.4.0 typing-inspect-0.9.0 urllib3-1.26.16

[dosubot]

🤖

Hello,

Thank you for reaching out and providing detailed information about your issue. Based on the context provided, it seems like the issue might be related to the way the query_engine.chat() function handles streaming responses.

In the LlamaIndex codebase, there is a temporary hack to toggle streaming on and off for the query engine. This is done by checking if the query engine is an instance of RetrieverQueryEngine and then manually setting the _streaming attribute of the _response_synthesizer of the query engine.

Here is the relevant code snippet:

from llama_index.query_engine.retriever_query_engine import RetrieverQueryEngine

if isinstance(self._query_engine, RetrieverQueryEngine):
    is_streaming = self._query_engine._response_synthesizer._streaming
    self._query_engine._response_synthesizer._streaming = False

This hack is used because the query engine currently uses a class attribute to configure streaming, and the developers are in the process of moving towards separate streaming and non-streaming methods. If the query engine you are using is not an instance of RetrieverQueryEngine or if it does not have a _response_synthesizer with a _streaming attribute, this could be causing the issue.

Additionally, the chat() function generates a standalone question from the conversation context and the last message, and then queries the query engine for a response. If there is an issue with the generation of the standalone question or the querying process, this could also be causing the function to not work as expected.

Without more specific information about the error or issue you are experiencing, it's difficult to provide a more precise answer. Could you please provide more details about the error message or the incorrect response you are getting? This will help us to better understand and address the issue.

Thank you for your patience and cooperation.

Sources

• llama_index/chat_engine/condense_question.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@robgon-art The default chat engine is an agent -- and an agent decides whether to query the index or not based on the user message and description of the index.

Tbh, this default mode could be improved. But you can try passing setting the index to be called at least once on every user message with index.as_chat_engine(..., function_call="query_engine_tool")

A system_prompt="..." kwarg may also help the bot better understand it's purpose.

Or, you can setup the agent from scratch with a better name/description
https://gpt-index.readthedocs.io/en/stable/examples/agent/openai_agent_with_query_engine.html

Or, you can try index.as_chat_engine(chat_mode="context")
https://gpt-index.readthedocs.io/en/stable/examples/chat_engine/chat_engine_context.html

[robgon-art]

Hi Logan, OK, thanks for the info. Setting chat_mode="context" worked just fine. This also led me to the page below, which basically said the same thing:

https://gpt-index.readthedocs.io/en/v0.7.18/examples/chat_engine/chat_engine_context.html

Maybe add this to the sample code here?
https://gpt-index.readthedocs.io/en/v0.7.18/getting_started/customization.html

from llama_index import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_chat_engine(chat_mode="context")
response = query_engine.chat("What did the author do growing up?")
print(response)

response = query_engine.chat("Oh interesting, tell me more.")
print(response)