VectorDB agnostic implementation

[porimol]

Why?

Developing a VectorDB agnostic implementation refers to creating PoRAG library that can work with any VectorDB without being dependent on a specific one (E.g.: ChromaDB).

Here are several reasons why VectorDB agnostic implementation approach might add more values:

1. Freedom of Choice: Being tied to a specific VectorDB can limit the options and make it difficult to switch to a better or more cost-effective VectorDB in the future
2. Future-Proofing: VectorDB technology evolve rapidly. A VectorDB agnostic approach ensures that the library can integrate with newer databases that may offer better performance, features, or cost benefits in the future.
3. Performance Optimization: Different VectorDBs might have different strengths and weaknesses. An agnostic implementation allows you to benchmark and optimize performance by choosing the best DB for specific workloads or scenarios.
4. Interoperability and Integration: An agnostic approach ensures that the implementation can work across different platforms and environments, enhancing interoperability.
5. Better Developer Experience: Developers can work with a standardized interface or API, reducing the learning curve and making it easier to onboard new team members.

How it might be looks like?

Example: 1

python main.py --vectordb qdrant --text_path test.txt

Example: 2

# config file
VECTOR_DB=chromadb
URI: ........
AUTH_TOKEN:.....

python main.py --text_path test.txt

Of course those two examples are not limited there are several others way to improve it.

Conclusion
A VectorDB agnostic implementation provides significant advantages in terms of flexibility, future-proofing, performance optimization, scalability, cost efficiency, interoperability, and developer experience. It allows developers to adapt to changing needs, leverage the best available technologies.

[asiff00]

@porimol Makes sense! It gives us more flexibility and control. We'll add that to the to-do list. Thanks for you input!