Implement vectorDB & embeddings model

[lukehinds]

We require a method of performing similarity search.

There are several approaches here we could take.

One is the current approach of combing an embeddings model with a vectorDB

Choices here are Qdrant, Milvus and Pinecone, weaviate

The second would be a memory / file based approach

Possible candiates here would be Usearch an FAISS,

Embeddings model

Either way an embeddings model will be needed

We will need a means of calling an embeddings API , luckily this is pretty simple with LiteLLM, which provides quite wide provider coverage: https://docs.litellm.ai/docs/embedding/supported_embedding

With this approach we can support a small local embeddings model, or if someone prefers they can use a cloud service.

[lukehinds]

Performed a POC with weaviate , created a schema and import JSON.

Working fine for package names, general words search, may not be suitable for semantic search, so looking an embeddings for vectorization.

Hitting some issues with API, as experimental so may be not complete feature wise.