Integrate weaviate as another DocumentStore

[lalitpagaria]

Is your feature request related to a problem? Please describe.
Haystack already support Vector search via FAISS and Milvus. In this both solution document/data reside in SQL store.
So main idea is what if we have data and embedding close to each other which Weaviate do (Yes Elasticsearch as well have this capability but not performant). Hence reduction in less network calls.

Describe the solution you'd like
What about integrating Weaviate as another document store.

Describe alternatives you've considered
I thought about having FAISS as embedding store and RocksDB as document store (only keeping vectorId to text mapping). I am sure this would beat many system but it would not be as customisable as other solutions :)
Also making it distributed would be challenge along with adding filter queries.

Additional context
I feel it would be easier to integrate via Python binding. All would be done via GraphQL api interface as done in case of Milvus.

[venuraja79]

Found this notebook that uses python client to connect to Weaviate. https://github.com/semi-technologies/Getting-Started-With-Weaviate-Python-Client/blob/main/Getting-Started-With-Weaviate-Python-Client.ipynb

[lalitpagaria]

@venuraja79 Would you like to contribute and create PR?
You can check ElasticSearchDocumentStore which use elasticsearch client or MilvusDocumentStore which use milvus client to sample integration with Haystack.

Obviously Haystack community can support you in this journey.

[venuraja79]

sure @lalitpagaria. Just started reviewing the Weaviate docs.

[venuraja79]

Few design decisions -

1. Haystack Index == Weaviate class
2. Haystack Document meta (dict) - to be stored as a property in weaviate
3. text2vec-transformers to create the vectors, it will be configurable though

Just a quick update -
have made some progress in creating schema, writing docs and querying the system. With this, I can start implementing the document store and will post further progress here.

[lalitpagaria]

Awesome! @venuraja79
Can you please create WIP PR so people can review and provide early feedback on design.

[venuraja79]

All - raised a WIP PR. Write, get and query methods have been tested offline. I'll create automated tests and update during the next iteration.
A few design questions and the dev status (pending items etc.,) are in the PR itself. Please feel free to review.

#1064

[LarsAC]

Great idea. Weaviate / haystack looks like a good fit. Happy to support / test, if help is needed ?

[LarsAC]

Confirmed working for a simple scenario, very nice. If there is something specific you would like me to test, please let me know.

[venuraja79]

Hi @LarsAC, thanks for your help earlier. We have made a few design changes from the last version and have updated the code & tests. Except for query and update embeddings methods, others have been validated. Any review / tests from your side will be great when you get a chance.

[tholor]

Implemented in #1064