New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ConversationalRetrievalChain doesn't return score with sources #5067
Comments
Hey @zigax1 if you see the return line of the return [doc for doc, _ in docs_and_scores] Try using the |
@AvikantSrivastava Thank you for the answer, I really appreciate it. I am using ConversationalRetrievalChain which has it integrated. The code snippet is in above. Thank you |
I checked the implementation of The chain internally calls |
@hwchase17 Should we add a flag to |
That would be awesome |
Thanks for raising this issue! @zigax1 what do you wish to do with the scores once they're available? |
@vowelparrot, I am using ConversationalRetrievalChain to chat over multiple files (some of them are PDF, some docx, txt,..). So for me is very important to also return the source, so I easily know from which file the answer came. Here score comes in to make wonders. I could easily than filter the sources before displaying them, if the score is low. I hope this will be a good enough answer for you to raise the PR for @AvikantSrivastava to integrate this funcionality. Wish all of you a nice day. |
@vowelparrot Something similar to the issue here.
def _similarity_search_with_relevance_scores(
self,
query: str,
k: int = 4,
**kwargs: Any,
) -> List[Tuple[Document, float]]:
"""Return docs and relevance scores, normalized on a scale from 0 to 1.
0 is dissimilar, 1 is most similar.
"""
raise NotImplementedError
``` |
@KeshavSingh29 great question - the short answer is we're working with the maintainers of the vector stores towards that goal. The longer answer is that each of the vector stores use different distance or similarity functions to compute scores (that also frequently are sensitive to the embeddings you're using). Sometimes it's a distance metric (lower number means more similar), while (less often) it's a similarity metric (higher number is more similar), and the results are usually unbounded. We could add a default implementation that tries to capture the most common use cases but we have refrained from trying to put a 'one size fits all' normalization function in the base class to avoid providing erroneous results |
@vowelparrot What are your thoughts about my answer? Will there be incoming change for Thank you for the answer. |
@AvikantSrivastava do you have any new information about opening a PR? Is solving this in process or not yet? Thanks |
Fixes #5067 Verified the following code now works correctly: ``` db = Chroma(persist_directory=index_directory(index_name), embedding_function=embeddings) retriever = db.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold": 0.4}) docs = retriever.get_relevant_documents(query) ```
…i#5655) Fixes langchain-ai#5067 Verified the following code now works correctly: ``` db = Chroma(persist_directory=index_directory(index_name), embedding_function=embeddings) retriever = db.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold": 0.4}) docs = retriever.get_relevant_documents(query) ```
Probably a dumb question, but what version of which package do I need to have for this to work? I've just done a pip install yesterday and it's still not returning scores. |
Also doesn't seem to work for me still |
This issue should be reopened, the issue is still not solved, the PR #5655 is not related to the behavior of ConversationalRetrievalChain, it's only specific for Chroma, but here it's the problem, scores are neglected with the _ |
No chance we can return source documents with scores now, with ConversationalRetrievalChain.from_llm? |
is this resolved? I am trying to get a score when using ConversationalRetrivalChain with the FAISS vector store. |
looks this still only return the doc but not the score; langchain/langchain/vectorstores/faiss.py Line 206 in 0c3de0a
|
Here's my hacky solution. It's a retriever that saves the score as part of the documents metadata. I just took the original class MyVectorStoreRetriever(VectorStoreRetriever):
# See https://github.com/langchain-ai/langchain/blob/61dd92f8215daef3d9cf1734b0d1f8c70c1571c3/libs/langchain/langchain/vectorstores/base.py#L500
def _get_relevant_documents(
self, query: str, *, run_manager: CallbackManagerForRetrieverRun
) -> List[Document]:
docs_and_similarities = (
self.vectorstore.similarity_search_with_relevance_scores(
query, **self.search_kwargs
)
)
# Make the score part of the document metadata
for doc, similarity in docs_and_similarities:
doc.metadata["score"] = similarity
docs = [doc for doc, _ in docs_and_similarities]
return docs Instead of doing retriever = MyVectorStoreRetriever(
vectorstore=vectordb,
search_type="similarity_score_threshold",
search_kwargs={"score_threshold": similarity_threshold, "k": 3},
) Good luck :) |
Thanks, this works totally fine! |
Took me a bit to find the correct imports :
|
Also worked for me. This could be considered as a final solution to this PR. |
System Info
LangChain v0.0.171
ChromaDB v0.3.22
Python v3.10.11
Who can help?
No response
Information
Related Components
Reproduction
This is my code:
Expected behavior
When I print the result directly after
result = self.chain({"question": question})
, I get displayed sources, metadata, kwargs, question, chat_history.I see here: https://github.com/hwchase17/langchain/blob/0c3de0a0b32fadb8caf3e6d803287229409f9da9/langchain/vectorstores/chroma.py#L165 and in line 182 in the official source code, that the similarity_search_with_score() is being called by default.
How can I also display the score than?
The text was updated successfully, but these errors were encountered: