How does orbitdb sync?

[subhasisbanik]

I have prepared a cluster of 2 nodes in IPFS with IPFS Daemon and swarm key.
I am running orbitdb and connecting to the IPFS node with "ipfs-api" in HTTP protocol.
I am able to post data through orbitdb (feed, keyvalue) from one node and am able to fetch it back in another orbitdb instance connected to the other node.
The issue is that when I am calling a "put" in one instance and trying to "get" it from another instance, its not able to "get" it the first time. But if the same "get" is being called from the same node the second time, its showing the data.

I am not sure how orbitdb sync works but below is the code that I am running.

Node 1-----------
const OrbitDB = require('orbit-db');
const IpfsApi = require('ipfs-api');
const ipfs = IpfsApi('localhost', '5002', {protocol: 'http'})
const orbitdb = new OrbitDB(ipfs);
const writePermissions = [orbitdb.key.getPublic('hex'), ];
const kv = await orbitdb.kvstore('new-db',{write:writePermissions});
await kv.load();
await kv.put('test', 80);

Node 2------------
const OrbitDB = require('orbit-db');
const IpfsApi = require('ipfs-api');
const ipfs = IpfsApi('localhost', '5002', {protocol: 'http'})
const orbitdb = new OrbitDB(ipfs);
const db = await orbitdb.open('/orbitdb/QmPJ4QXJDfBjPKB6JKZcqmrDfDxbubC85BMQvxDcR2Gunb/new-db');
await db.load();
db.get('test');

Please help!!!

[tyleryasaka]

@subhasisbanik I know what the cause of this issue is (I ran into the same thing myself). db.load() resolves only after the database has finished loading from storage into memory. It does not wait for the database to be replicated from peers. The replication happens in the background, and is likely completing a split second after db.get('test') is run. That's why when you run the code a second time, you get the expected value.

If you want to wait for the replication to finish, you can listen for the replicated event. E.g.

await Promise.resolve(resolve => {
  db.events.on('replicated', resolve)
})

Beware, though; if no replication occurs, the above code will hang indefinitely. (For example, if the peers are already fully in sync.) This is a major problem; I don't know of a good solution off the top of my head, though I am looking into options. I'll report back if I find anything.

[subhasisbanik]

Thanks for the explanation. I would surely wait for a reply from you..

…

On Tue, Nov 20, 2018, 10:06 AM Tyler Yasaka ***@***.*** wrote: @subhasisbanik <https://github.com/subhasisbanik> I know what the cause of this issue is (I ran into the same thing myself). db.load() resolves only after the database has finished loading *from storage into memory*. It does not wait for the database to be replicated from peers. The replication happens in the background, and is likely completing a split second after db.get('test') is run. That's why when you run the code a second time, you get the expected value. If you want to wait for the replication to finish, you can listen for the replicated event. E.g. await Promise.resolve(resolve => { db.events.on('replicated', resolve) }) Beware, though; if no replication occurs, the above code will hang indefinitely. (For example, if the peers are already fully in sync.) This is a major problem; I don't know of good solution off the top of my head, though I am looking into options. I'll report back if I find anything. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/orbitdb/orbit-db/issues/474#issuecomment-440135153>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPkaAqZ7xiYgFJYQ7_nQtulBRrtfNWuks5uw4bHgaJpZM4XR1S9> .

[tyleryasaka]

Hi @subhasisbanik I did find a solution, which requires a couple modifications to orbit-db.

First, I made this change to orbit-db: projectaspen/orbit-db@f91220d

Then I made this minor adjustment to orbit-db-store: projectaspen/orbit-db-store@a31e1f7

This allows me to wait for the new synced event which is fired once for each peer + database, after all of the data has been copied over. If you wait for this event (once after opening each database) for each node you want to sync with, then when you retrieve data for orbit-db, it will be up-to-date.

To make this process smoother, I created 2 packages: 1 for the orbit-db server node, one for the client. They're experimental right now, but feel free to check them out! They use the orbit-db repo changes I linked to above.

• orbit-connect-client
• orbit-connect-server

[subhasisbanik]

Awesome. I will revert once I have a look into it.

…

On Wed, Nov 21, 2018, 3:07 PM Tyler Yasaka ***@***.*** wrote: Hi @subhasisbanik <https://github.com/subhasisbanik> I did find a solution, which requires a couple modifications to orbit-db. First, I made this change to orbit-db: ***@***.*** <projectaspen/orbit-db@f91220d> Then I made this minor adjustment to orbit-db-store: ***@***.*** <projectaspen/orbit-db-store@a31e1f7> This allows me to wait for the new synced event which is fired once for each peer + database, after all of the data has been copied over. If you wait for this event (once after opening each database) for each node you want to sync with, then when you retrieve data for orbit-db, it will be up-to-date. To make this process smoother, I created 2 packages: 1 for the orbit-db server node, one for the client. They're experimental right now, but feel free to check them out! They use the orbit-db repo changes I linked to above. - orbit-connect-client <https://github.com/projectaspen/aspen/tree/master/packages/orbit-connect-client> - orbit-connect-server <https://github.com/projectaspen/aspen/tree/master/packages/orbit-connect-server> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/orbitdb/orbit-db/issues/474#issuecomment-440597921>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPkaClMoTPUZRD71BejXmN3D8QioSaAks5uxR7cgaJpZM4XR1S9> .

[aphelionz]

Moving this to the Field Manual repo so we can detail this in the book