Persistent Indexes so that they don't have to be all in memory at once

[fazo96]

I was wondering, in case of Stores where the use case requires running complex queries the updateIndex function could be used to keep an external database in sync (A SQL DB but also a kvstore like level) so that all of the Index data doesn't have to be in memory and complex queries can be ran.

I'd call this "Persistent" or "External" Index.

This can be done by writing a custom Index, obviously, but I have a couple of questions about how well would it work:

1. I don't really undestand with which entries is updateIndex called: in the replication and write cases, it only goes through new entries? Or all entries in order but starting from the oldest new one? Or maybe something else. Knowing this is necessary to properly write custom stores, maybe it should be documented somewhere or it already is
2. When the DB is opened, if I understand correctly the Load process starts from scratch so it will get the heads, travel the links all the way back to the bottom and then work its way up with the updateIndex function. Is there any way to, for example, avoid the whole process by checking that the Persistent Index was already fully synced with the current Log heads? Of course this would only be useful with persistent indexes and not with the in memory ones

[haydenyoung]

Closing as I believe this is now addressed in the latest version of OrbitDB (e.g. KeyValueIndexed).