Feature Request: Custom Fields for Vector Database Filtering in Spring AI

[M-Razavi]

Expected Behavior

The VectorStore interface and related classes in Spring AI should support custom fields for vector database filtering. This would allow users to:

1. Add custom fields to documents
2. Perform similarity searches with filtering based on these custom fields

Example code for how this might look:

// Adding a document with custom fields
Document doc = Document.builder()
    .withContent("Sample content")
    .withCustomField("category", "technology")
    .withCustomField("publishDate", LocalDate.now())
    .build();

vectorStore.add(List.of(doc));

// Performing a search with custom field filtering
SearchRequest request = SearchRequest.builder()
    .withQuery("AI advancements")
    .withCustomFieldFilter("category", "technology")
    .withCustomFieldFilter("publishDate", LocalDate.now().minusDays(7))
    .build();

List<Document> results = vectorStore.similaritySearch(request);

Current Behavior

Currently, the VectorStore interface does not provide a way to add custom fields and limited to the Document class. The Document class has a fixed set of fields. This limits the ability to perform more granular and domain-specific searches.

Context

This feature is needed because:

1. Many applications require domain-specific metadata for documents (e.g., publication date, author, category).
2. Users often need to filter search results based on these custom attributes in addition to similarity matching.

[dafriz]

Passing a Filter.Expression to the SearchRequest covers this using metadata. For example:

// Adding a document with metadata fields
Document doc = Document.builder()
    .withContent("Sample content")
    .withMetadata("category", "technology")
    .build();

vectorStore.add(List.of(doc));

// Performing a search with filter expression
SearchRequest request = SearchRequest
    .query("AI advancements")
    .withFilterExpression(
        new FilterExpressionBuilder()
             .eq("category", "technology")
             .build()   
    );

List<Document> results = vectorStore.similaritySearch(request);