-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
context_selection doesn't work correctly when chat_mode = "context" #1175
Comments
To resolve the issue with context selection in a Here is an example of how you can use the import numpy as np
from llama_index.core import VectorStoreIndex
from llama_index.readers.web import SimpleWebPageReader
from trulens_eval import TruLlama, Feedback, Tru, feedback
from trulens_eval.feedback import Groundedness
tru = Tru()
tru.reset_database()
def evaluate(app_id, chat_engine, tru):
openai = feedback.OpenAI()
# Correct context selection using select_context
context_selection = TruLlama.select_context(chat_engine)
# Question/answer relevance between overall question and answer.
f_qa_relevance = Feedback(openai.relevance, name="QA Relevance").on_input_output()
grounded = Groundedness(groundedness_provider=openai)
f_groundedness = (
Feedback(grounded.groundedness_measure_with_cot_reasons, name="Groundedness")
.on(context_selection)
.on_output()
)
f_context_relevance = (
Feedback(
openai.qs_relevance_with_cot_reasons,
np.max,
name="Context relevance",
)
.on_input()
.on(context_selection)
)
tru_recorder = TruLlama(
chat_engine,
app_id=app_id,
tru=tru,
feedbacks=[f_context_relevance, f_groundedness, f_qa_relevance],
)
question = "What did the author do growing up?"
with tru_recorder as recording:
response = chat_engine.chat(question)
print(response)
app_id = "test_chat_engine"
documents = SimpleWebPageReader(html_to_text=True).load_data(
["http://paulgraham.com/worked.html"]
)
index = VectorStoreIndex.from_documents(documents)
chat_engine = index.as_chat_engine(chat_mode="context")
evaluate(app_id, chat_engine, tru)
tru.run_dashboard() This code snippet demonstrates how to select the context using the |
Hi; can you print out the calls that got recorded in your example with this snippet (add it after you print response): for call in recording.get().calls:
print(call.stack[-1].path) |
The snippet you shared prints the following:
If it's helpful, here's the entire output with the full stack trace:
|
Hi, Checking in on this. Is there anything I should be doing differently to get this to work? |
Apologies I forgot an important bit to print: for call in recording.get().calls:
print(call.stack[-1].path, call.method()) |
That returns the following:
|
Checking in on this again to make sure it's not lost in the shuffle. I have upgraded to llama-index==0.10.44, trulens_eval==0.31.0. Now I'm getting a different error. Here is the code:
I get this error: With this output:
You can reproduce this by creating a new conda environment, running |
Thanks for the printouts. It looks like the app you configured has a component under the Can you try replacing the selector # Add to imports: from trulens_eval import Select
Select.RecordCalls._retriever.retrieve.rets.source_nodes[:].node.text.collect() You might have the adjust the parts after |
@sfc-gh-pmardziel Thanks. That got me further, but still not quite there. I changed the functions to be the following:
Now it runs, but I get the following warnings:
In the dashboard the context is empty and context relevance and groundedness are both set to 0.0. Do I have to get |
This message suggests that retriever was called more than once. I'm unsure which call is the one you might be looking for. You can adjust the selector for a each call, for example here are the selectors to pick the first and second call respectively. Select.RecordCalls._retriever.retrieve[0].rets.source_nodes[:].node.text.collect()
Select.RecordCalls._retriever.retrieve[1].rets.source_nodes[:].node.text.collect() You can also try looking at the contexts from all of the calls at once using this selector: Select.RecordCalls._retriever.retrieve[:].rets.source_nodes[:].node.text.collect() |
Bug Description
When evaluating a llama_index rag application, context selection does not work properly when chat_mode="context"
To Reproduce
Expected behavior
This should run and provide context relevance, answer relevance, and groundedness. This is what happens if you remove
chat_mode="context"
Relevant Logs/Tracebacks
Environment:
Additional context
Should I be setting
context_selection
to something else?The text was updated successfully, but these errors were encountered: