VectorDB agnostic implementation

[porimol]

Why?

Developing a VectorDB agnostic implementation refers to creating PoRAG library that can work with any VectorDB without being dependent on a specific one (E.g.: ChromaDB).

Here are several reasons why VectorDB agnostic implementation approach might add more values:

1. Freedom of Choice: Being tied to a specific VectorDB can limit the options and make it difficult to switch to a better or more cost-effective VectorDB in the future
2. Future-Proofing: VectorDB technology evolve rapidly. A VectorDB agnostic approach ensures that the library can integrate with newer databases that may offer better performance, features, or cost benefits in the future.
3. Performance Optimization: Different VectorDBs might have different strengths and weaknesses. An agnostic implementation allows you to benchmark and optimize performance by choosing the best DB for specific workloads or scenarios.
4. Interoperability and Integration: An agnostic approach ensures that the implementation can work across different platforms and environments, enhancing interoperability.
5. Better Developer Experience: Developers can work with a standardized interface or API, reducing the learning curve and making it easier to onboard new team members.

How it might be looks like?

Example: 1

python main.py --vectordb qdrant --text_path test.txt

Example: 2

# config file
VECTOR_DB=chromadb
URI: ........
AUTH_TOKEN:.....

python main.py --text_path test.txt

Of course those two examples are not limited there are several others way to improve it.

Conclusion
A VectorDB agnostic implementation provides significant advantages in terms of flexibility, future-proofing, performance optimization, scalability, cost efficiency, interoperability, and developer experience. It allows developers to adapt to changing needs, leverage the best available technologies.

[asiff00]

@porimol Makes sense! It gives us more flexibility and control. We'll add that to the to-do list. Thanks for you input!

[hassanaliemon]

Hello @porimol,
I appreciate the points you've made about the value of a VectorDB agnostic implementation for the PoRAG library. It is true that if we make our own VectorDB we will have more flexibility but it has it's own cost:

Complexity and Corner Cases: While the idea of a VectorDB agnostic implementation is appealing, fully decoupling from specific databases like ChromaDB requires considerable effort. We need to account for various corner cases and unique features, which can be a complex and time-consuming process.

Developers' Practicality: From a developer's perspective, it's often more rational to leverage existing packages and libraries. This approach is widely adopted in the industry because it helps avoid reinventing the wheel and allows developers to focus on higher-level functionalities rather than underlying integrations. Just as we routinely use packages like NumPy for numerical computations, relying on a robust VectorDB can streamline development and maintenance.

Convenience and Efficiency: Utilizing a well-established VectorDB like ChromaDB can offer significant convenience. It provides a standardized interface and ready-made solutions that have been tested and optimized. This can save considerable development time and reduce the risk of bugs or performance issues that might arise from building and maintaining a custom agnostic layer.

As a result, agnostic approach has its own merits, it is essential to weigh the trade-offs in terms of complexity, practicality, and developer efficiency. Leveraging existing VectorDBs can often be the more efficient and developer-friendly route.

But you can contribute to our cause by developing your own VectorDB which we will be gladly using in our PoRAG module. We have more limitations mentioned as well like parsing bangla PDF. You are most welcome to contribute to parsing as well, as we face a tremendous problem because available packages can not parse Bangla from PDFs. We have to extend those packages to make it Bangla Support.

[porimol]

@hassanaliemon I think somehow the interpretation of this concept somehow you ended with a different meaning. For this library (PoRAG) VectorDB agnostic implementation doesn't mean to develop YET-ANOTHER-VECTORDB! It's mean that develop this library as a tool that will not be tightly coupled with any VectorDB rather it will give the flexiblity to developers (precisely users) to decide which VectoDB they would like to use!

I hope you understand the concept, if need more clarification how this concept works, I can help.

[hassanaliemon]

We mainly used ChromaDB as it is fully open-sourced and convenient with langchain. Other options can be explored but we would like to explore them after Bangla PDF parsing.

[hassanaliemon]

@porimol I think you can contribute to PoRAG by adding other VectorDB, we will be glad to collaborate