Determine how to use a staged sync approach

[sethvincent]

We'll be syncing data in a few stages:

• authstore
• datastores (data like observations)
• blobs
  • only syncing original images with desktop, and smaller versions between mobile devices.
  • downloading specific folders with hyperdrive-next will help with this

questions to answer:

• do corestore namespaces sync separately?
• or do we need separate corestores for each category? (authstore, datastore, blobstore)
• or is there another approach for being selective about which cores are downloaded when that doesn't rely on using corestore a particular way?

[tomasciccola]

Do authstore, records and blobs have different replication mechanisms?
i.e. maybe authstore gives some capabilities that dictates if a record or blob should be replicated by the peer.
or i.e. blobs are replicated lazyly to save space (download an image only when opening it?)
I'm gonna do some test to see how namespaces replicate. I think given the api namespaces sync separately (like when you do namespace.get(key)) when I'm gonna check to make sure

[sethvincent]

@tomasciccola so far replication in authstore and datastore has been the same: basically just exposing the corestore.replicate method.

[tomasciccola]

So, corestore namespaces sync separately. In fact cores inside a store do to. Yo need to explicitly ask for a core public key to get it. I thought that maybe passing the master key of a corestore to another peer would replicate every core in that store but doesn't seem to be the case.
I've published this gist as an example.
I think having every store on the same corestore could work; can't see any shortcomings for now...
About sync strategies, I'm reading authstore code here and I know you're (maybe?) writing a design document.
I think (just spitballing here...) the first step to sync should be auth since maybe there are capabilities/permissions derived from it. So after syncing authstore, further decisions could be taken.
Need to read more authstore code to get a better idea though...

[sethvincent]

Cool. The tricky thing I was just looking at is that it seems there's not a way to cleanly get a list of keys that are in a corestore namespace, so to be able to replicate only the keys that are in authstore for example we'll need to keep a list of those somewhere. Maybe a sqlite table that has three columns: storeType, coreId, and identityId. The last one would be for use cases like removing them if a device is removed from the project.

[sethvincent]

@tomasciccola If it sounds good to you maybe you could make a new branch (branched from the authstore branch rather than main) and put together something like:

• a class called something like CoreIdCache that:
  • takes an instance of this sqlite client as a constructor arg
  • creates a table if it doesn't exist with storeType, coreId, and identityId columns
  • has two methods that wrap the query method to get records by storeType and by identityId
  • and a method for adding new records

Then we'll be able to use that as the source of keys that we exchange with peers before syncing in each stage.

[tomasciccola]

so, here's some progress on this. The only thing is that the sqlite.query wrapper is expected to return a Doc which doesn't match the proposed table of storeType, coreId, identityId. I'm thinking that maybe the query that I make in the class should return the actual table matching Doc?

[tomasciccola]

I've created this issue to track this specific task