Auto embedding

[richard-epsilla]

Automatically embed the attributes

1. When start DB, need provide embedding service base url:

./build/vectordb -e http://localhost:8889

2. Define embedding indices:

{
    "name": "MyTable124",
    "returnTableId": true,
    "fields": [
        {
        "name": "ID2",
        "dataType": "FLOAT"
        },
        {
        "name": "Document",
        "dataType": "STRING"
        },
        {
        "name": "Document2",
        "dataType": "STRING"
        },
        {
        "name": "ID1",
        "dataType": "BIGINT"
        },
        {
            "name": "Embedding",
            "dataType": "VECTOR_FLOAT",
            "dimensions": 4,
            "metricType": "COSINE"
        }
    ],
    "indices": [
        {
            "name": "MyIndex",
            "field": "Document",
            "model": "BAAI/bge-base-en-v1.5"
        },
        {
            "name": "MyIndex2",
            "field": "Document2",
            "model": "BAAI/bge-small-en-v1.5"
        }
    ]
}

3. When insert the attributes, the indices will be automatically embedded
   Supported embedding models:
   BAAI/bge-small-en
   BAAI/bge-small-en-v1.5
   BAAI/bge-small-zh-v1.5
   BAAI/bge-base-en
   BAAI/bge-base-en-v1.5
   sentence-transformers/all-MiniLM-L6-v2
   openai/text-embedding-ada-002

When using openai embedding for a table, need pass X-OpenAI-API-Key in request header to insert, query, and loaddb API

4. When query, can query by index. Can provide either a query vector, or a query string. If provide query string, it will automatically embed using the model defined in the index

POST http://localhost:8888/api/<DB>/data/query
Option 1: with query
{
    "table": "VideoData",
    "query": "What's the best way to code xxx?",
    "queryIndex": "Embedding",
    "limit": 5
}
If there is just 1 index, the queryIndex can be omitted:
{
    "table": "VideoData",
    "query": "What's the best way to code xxx?",
    "limit": 5
}

Option 2: with query vector
{
    "table": "VideoData",
    "queryVector": [ 0.06929776072502136,
                0.49731335043907166,
                0.6196035146713257,
                0.6032981276512146 ...],
    "queryIndex": "Embedding",
    "limit": 5
}