Skip to content

Redis scores seems to be sorted from lowest to highest (with cosine) #8177

Closed
Closed
Redis scores seems to be sorted from lowest to highest (with cosine)#8177
Labels
bugRelated to a bug, vulnerability, unexpected error with an existing feature

Description

@FrancescoSaverioZuppichini
Francesco Saverio Zuppichini (FrancescoSaverioZuppichini)
opened on Jul 24, 2023

System Info

python = 3.10
langchain = 0.0.222

Who can help?

No response

Information

  • The official example notebooks/scripts
  • My own modified scripts

Related Components

  • LLMs/Chat Models
  • Embedding Models
  • Prompts / Prompt Templates / Prompt Selectors
  • Output Parsers
  • Document Loaders
  • Vector Stores / Retrievers
  • Memory
  • Agents / Agent Executors
  • Tools / Toolkits
  • Chains
  • Callbacks/Tracing
  • Async

Reproduction

Not sure how to give you all the steps but here attached a screenshot of the debugger run at line vectorstores.redis 338 and as you can see it goes from the lowest to the highest. You can see the vector_score on the right

Screenshot 2023-07-24 at 13 40 28

Expected behavior

I think it should return from highest to lowest

Activity

  1. dosubot
    added
    area: vector store
    bugRelated to a bug, vulnerability, unexpected error with an existing feature
    on Jul 24, 2023

  2. dosubot commented on Jul 24, 2023

    @dosubot
    dosubotbot
    on Jul 24, 2023

    Answer generated by a 🤖

    Answer

    I understand that you're experiencing an issue with the Redis vector store in LangChain, where the scores seem to be sorted from lowest to highest when using cosine similarity. You're expecting the scores to be sorted from highest to lowest instead.

    In LangChain, the Redis vector store sorts the results from lowest to highest because the cosine similarity, which is used as the scoring metric, ranges from -1 to 1. A score of 1 means the vectors are identical, and -1 means they are diametrically opposed. However, Redis interprets these scores as distances, so a lower score means the vectors are closer together, hence more similar.

    The sorting is done in the _prepare_query method, where .sort_by("vector_score") sorts the results by the vector_score field, which holds the cosine similarity scores. Since Redis treats these scores as distances, it sorts them from lowest to highest.

    If you want the results to be sorted from highest to lowest, you can modify the .sort_by method to sort in descending order by adding DESC:

    def _prepare_query(self, k: int) -> Query:
    ...
    return (
        Query(base_query)
        .return_fields(*return_fields)
        .sort_by("vector_score", "DESC")
        .paging(0, k)
        .dialect(2)
    )

    Please note that this change will make the results return from least similar to most similar, which is the opposite of the current behavior.

    This is one potential solution to your issue. However, please be aware that as an AI, I can only see text and cannot see images or links. If you believe this is a bug that could impact other users, you might want to consider making a pull request with your proposed changes.

    If this solution doesn't align with your experience, I would recommend upgrading to the latest version of LangChain to see if the issue persists.

    Sources

    This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

    Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

  3. FrancescoSaverioZuppichini commented on Jul 25, 2023

    @FrancescoSaverioZuppichini
    Francesco Saverio Zuppichini (FrancescoSaverioZuppichini)
    on Jul 25, 2023
    Author

    damn this bot is awesome!

  4. Spartee commented on Sep 15, 2023

    @Spartee
    Sam Partee (Spartee)
    on Sep 15, 2023
    Contributor

    I've tried to make the score vs distance appearent in the new API for Redis. distance should be used for vector distance and score should be used for vector similarity. Is this still causing you issues or can this be closed Francesco Saverio Zuppichini (@FrancescoSaverioZuppichini) ?

  5. FrancescoSaverioZuppichini commented on Sep 15, 2023

    @FrancescoSaverioZuppichini
    Francesco Saverio Zuppichini (FrancescoSaverioZuppichini)
    on Sep 15, 2023
    Author

    I've tried to make the score vs distance appearent in the new API for Redis. distance should be used for vector distance and score should be used for vector similarity. Is this still causing you issues or can this be closed Francesco Saverio Zuppichini (@FrancescoSaverioZuppichini) ?

    not sure I understood - I am not using langchain anymore tbh but out of curiosity, is it a distance or real cosine similarity?

  6. dosubot commented on Dec 18, 2023

    @dosubot
    dosubotbot
    on Dec 18, 2023 – with DosuBot

    Hi, Francesco Saverio Zuppichini (@FrancescoSaverioZuppichini)

    I'm helping the LangChain team manage our backlog and am marking this issue as stale. From what I understand, you reported an issue with Redis scores appearing to be sorted from lowest to highest when using cosine similarity, and I provided an explanation of how Redis interprets and sorts the scores, suggesting a potential solution to modify the sorting method. Spartee also mentioned an update to the API for Redis, asking if the issue can be closed. You expressed uncertainty about the update and whether the scores represent distance or real cosine similarity, indicating that you are no longer using LangChain.

    Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

  7. dosubot
    added
    stale
    on Dec 18, 2023

  8. FrancescoSaverioZuppichini commented on Dec 19, 2023

    @FrancescoSaverioZuppichini
    Francesco Saverio Zuppichini (FrancescoSaverioZuppichini)
    on Dec 19, 2023
    Author

    Hi, Francesco Saverio Zuppichini (@FrancescoSaverioZuppichini)

    I'm helping the LangChain team manage our backlog and am marking this issue as stale. From what I understand, you reported an issue with Redis scores appearing to be sorted from lowest to highest when using cosine similarity, and I provided an explanation of how Redis interprets and sorts the scores, suggesting a potential solution to modify the sorting method. Spartee also mentioned an update to the API for Redis, asking if the issue can be closed. You expressed uncertainty about the update and whether the scores represent distance or real cosine similarity, indicating that you are no longer using LangChain.

    Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

    you got it my friend, let's close it - it is not cosine similarity but a distance metric which is very confusing btw

  9. Francesco Saverio Zuppichini (FrancescoSaverioZuppichini)
    closed this as completedon Dec 19, 2023

  10. dosubot commented on Dec 19, 2023

    @dosubot
    dosubotbot
    on Dec 19, 2023 – with DosuBot

    Thank you Francesco Saverio Zuppichini (@FrancescoSaverioZuppichini) for closing the issue! Your contribution is much appreciated.

  11. dosubot
    removed
    stale
    on Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugRelated to a bug, vulnerability, unexpected error with an existing feature

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions