DOC: How to use or and operator with filtering metadata in a elastic vector store.

[djaffer]

Checklist

• I added a very descriptive title to this issue.
• I included a link to the documentation page I am referring to (if applicable).

Issue with current documentation:

The idea is to filter in as retriever function to get only subset of documents using operator or and. The self query in langchain does not seem supported with elastic vector.

Idea or request for content:

The as retriever should allow filter to take a function which just gets called over all documents.

The documentation is like a treasure hunt right now to find. Please consider improving.

[dosubot]

To use the "or" and "and" operators for filtering metadata in an Elastic vector store within a retriever function, you can leverage the buildMetadataTerms method in the ElasticVectorSearch class to construct the necessary filter queries. Here is an example of how you can configure this:

import { ElasticVectorSearch } from "your-elastic-vector-store-package"; // Replace with actual import
import { Client } from "@elastic/elasticsearch";
import { Document } from "@langchain/core/documents";
import { ChatOpenAI } from "your-llm-package"; // Replace with actual import

const client = new Client({ node: 'http://localhost:9200' });
const indexName = 'your-index-name';
const embeddings = new YourEmbeddingsClass(); // Replace with your embeddings class

const elasticVectorSearch = new ElasticVectorSearch(embeddings, {
  client,
  indexName,
});

const filter = [
  { field: "director", operator: "term", value: "Greta Gerwig" },
  { field: "genre", operator: "term", value: "Drama" }
];

const query = {
  bool: {
    must: elasticVectorSearch.buildMetadataTerms(filter).must,
    must_not: elasticVectorSearch.buildMetadataTerms(filter).must_not,
  }
};

const relevantDocuments = await elasticVectorSearch.similaritySearchVectorWithScore(
  [/* your query vector */],
  10,
  query
);

console.log(relevantDocuments);

In this example, the buildMetadataTerms method is used to construct the must and must_not clauses for the Elasticsearch query. You can adjust the filter array to include your desired filtering criteria and logical operators. This configuration will ensure that the filters are combined using the specified logical operator when retrieving documents from the Elastic vector store ^[1][2].

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[djaffer]

@dosu How to use the query filter in retriever?

[dosubot]

To use the filter in the retriever function for an Elastic vector store, you can set the mergeFiltersOperator parameter in the searchParams of the SelfQueryRetriever class. Here is an example of how you can configure this:

import { SelfQueryRetriever, FunctionalTranslator } from 'path-to-langchain';
import { ElasticVectorStore } from 'path-to-elastic-vector-store'; // Replace with actual import

const selfQueryRetriever = new SelfQueryRetriever({
  vectorStore: new ElasticVectorStore(), // Initialize your Elastic vector store
  structuredQueryTranslator: new FunctionalTranslator(),
  queryConstructor: yourQueryConstructor, // Replace with your actual query constructor
  searchParams: {
    k: 10, // Number of documents to retrieve
    filter: yourFilter, // Replace with your actual filter
    mergeFiltersOperator: "or", // Use "or" or "and" as needed
    forceDefaultFilter: false,
  },
  useOriginalQuery: true,
  verbose: true,
});

// Example usage
const relevantDocuments = await selfQueryRetriever.getRelevantDocuments("Your query here");

In this example, replace yourQueryConstructor and yourFilter with your actual query constructor and filter. The mergeFiltersOperator can be set to either "or" or "and" depending on your requirement ^[1][2].

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[djaffer]

Solved it by creating patch and passing in the filter query.
#5568

const filter = [
{ field: "genre", operator: "or", value: "Drama" },
{ field: "genre", operator: "or", value: "Action" },
];

const reteriver = elasticVectorStore.asRetriever({
filter: filter
)
or
ScoreThresholdRetriever.fromVectorStore(vectorStore, {
minSimilarityScore: 0.5, // Finds results with at least this similarity score
maxK: 1, // The maximum K value to use. Use it based to your chunk size to make sure you don't run out of tokens
kIncrement: 1, // How much to increase K by each time. It'll fetch N results, then N + kIncrement, then N + kIncrement * 2, etc.
filter: filter,
});

After this change, the filter will be passed to the elastic vector similaritySearchVectorWithScore function.

const relevantDocuments = await elasticVectorSearch.similaritySearchVectorWithScore(
[/* your query vector */],
10,
filter
);