Persist Documents in a Database

[myntoralex]

Currently, only the HNSWPostgresIndexer supports persistence of Documents. Could we add the PQLiteIndexer some database persistence, not necessarily Postgres, but any database provider?

[JoanFM]

I believe the easiest way could be to support pointing to an external SQLite?

[numb3r3]

@myntoralex Currently, persistence is already supported via local-disk kv-store lmdb (the data is located at workspace folder defaultly). Is this local-disk persistence is what you expected? Or do you want to have external/remote data persistence supporting?

[numb3r3]

I see, I think you want external persistence supports actually. And there maybe two branches we can proceed:

1. supporting external database e.g., ElasticeSearch, Postgres etc.,
2. supporting object storage e.g., AWS S3, Google GCS

It would be great to get your preference, and that will help us to determine the priority of new features.

[myntoralex]

I would definitely prefer option 1, since change data capture from one database table or collection ought to be reflected in a similar data view. Thanks!

[cristianmtr]

Hey @myntoralex

Could you elaborate on your use case? What are you trying to do exactly with the external storage? It might help clarify the direction to go