[vector-databases] Add support for Apache Solr

[eolivelli]

Summary:

• This patch introduces support for Apache Solr as Vector Database
• This is a brief guide about how to enable Vector Search on Solr: https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html
• LangStream provides support for creating collections, data-types and adding fields, so you can control you Solr collection fully from LangStream (if you want)

Solr resource configuration:

  - type: "vector-database"
    name: "SolrDataSource"
    configuration:
      service: "solr"
      user: "${secrets.solr.username}"
      password: "${secrets.solr.password}"
      host: "${secrets.solr.host}"
      port: "${secrets.solr.port}"
      collection-name: "documents"

Solr assets:
We have now the "solr-collection" asset that allows you to handle a Solr collection.

This is an example:

assets:
  - name: "documents-table"
    asset-type: "solr-collection"
    creation-mode: create-if-not-exists
    deletion-mode: delete
    config:
      collection-name: "documents"
      datasource: "SolrDataSource"
      create-statements:
        - api: "/api/collections"
          method: "POST"
          body: |
            {
              "name": "documents",
              "numShards": 1,
              "replicationFactor": 1
             }
        - "api": "/schema"
          "body": |
            {
             "add-field-type" : {
                   "name": "knn_vector",
                   "class": "solr.DenseVectorField",
                   "vectorDimension": "1536",
                   "similarityFunction": "cosine"
              }
             }
        - "api": "/schema"
          "body": |
            {
              "add-field":{
                "name":"embeddings",
                "type":"knn_vector",
                "stored":true,
                "indexed":true
                }
            }
        - "api": "/schema"
          "body": |
            {
               "add-field":{
                   "name":"text",
                   "type":"string",
                   "stored":true,
                   "indexed":false,
                   "multiValued": false
               }
            }

Querying Solr

In order to query Solr you use the "query-vector-db" agent as usual, and you can pass the parameters this way:

  - name: "lookup-related-documents"
    type: "query-vector-db"
    configuration:
      datasource: "SolrDataSource"
      query: |
        {
          "q": "{!knn f=embeddings topK=10}?"
        }
      fields:
        - "fn:toListOfFloat(value.question_embeddings)"
      output-field: "value.related_documents"

Writing to Solr

You can use the "vector-db-sink" to write to Solr.
This is an example:

  - name: "Write to Solr"
    type: "vector-db-sink"
    input: chunks-topic
    configuration:
      datasource: "SolrDataSource"
      collection-name: "documents"
      fields:
        - name: "id"
          expression: "fn:concat(value.filename, value.chunk_id)"
        - name: "embeddings"
          expression: "fn:toListOfFloat(value.embeddings_vector)"
        - name: "text"
          expression: "value.text"

[eolivelli]

@alessandrobenedetti @dsmiley you may be interested in taking a look to/review how we are using the Solr client and APIs.