Can't replicate different, multiple databases (event concurrency bug?)

[cristiano-belloni]

This is an issue I have while trying to replicate two different dbs (in my case a keyvalue and a feed):

async function replicateFeed(address) {
    const feedDb = await orbitdb.feed(address)
    feedDb.events.on('replicated', () => console.log('feed replicated)', address)
    feedDb.events.on('replicate', () => console.log('feed replicate)', address)
    feedDb.events.on('replicate.progress', console.log('feed progress)', address)
}

async function replicateKvalue(address) {
    const feedDb = await orbitdb.keyvalue(address)
    feedDb.events.on('replicated', () => console.log('keyvalue replicated)', address)
    feedDb.events.on('replicate', () => console.log('keyvalue replicate)', address)
    feedDb.events.on('replicate.progress', console.log('keyvalue progress)', address)
}

await replicateKvalue(addressOfKvalue)
await replicateFeed(addressOfFeed)

If I run the above code, I get only the feed events. If I invert the bottom two lines, I get only the keyvalue events I get only the feed callback. The events, however, are sometimes the feed events and sometimes the keyvalue ones (ie, 50% of the time I get all the feeds and 50% all the keyvalues, despite the callbacks being called being only the feed one). If I invert the lines, vice versa.

Note that replicateKvalue and replicateFeed are not running concurrently: I await for each before running the other. What runs concurrently is the replication in the background. For some reason, only the last replication events seem to "win", but the updates are random. Looks like event rewriting?

In my case, waiting for replication serially (ie calling replicateFeed in the feed's replicated event handler) is not an option: I'm trying to write the "Twitter" demo and I'm doing these two replications for each followed user: If user are many, waiting serially for each of their databases and then waiting serially for each one of the users will take a lot of time.

[haadcode]

This is definitely a bug. I'll take a look asap. Thanks for reporting!

[haadcode]

@cristiano-belloni I'm having hard time to reproduce this. Can you share the full code for this? What are the exact addressed passed to each function? I've been looking into this but it doesn't happen here, I always get events from databases.

To clarify: it should be possible to open and use multiple databases concurrently.

[cristiano-belloni]

I'll try to simplify my code and reproduce this easily in the next days. Or I can open my Gitlab PoC repo, if you like, but it's much harder to read. I'm getting these results in Chrome, btw. I didn't try in Node.

…

On 30 Nov 2017 15:14, "Haad" ***@***.***> wrote: @cristiano-belloni <https://github.com/cristiano-belloni> I'm having hard time to reproduce this. Can you share the full code for this? What are the exact addressed passed to each function? I've been looking into this but it doesn't happen here, I always get events from databases. To clarify: *it should be possible to open and use multiple databases concurrently*. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#258 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATRuuGLPWXGdDuN2TUzEVRE5hhKsH_iks5s7sZRgaJpZM4QvkoE> .

[cristiano-belloni]

Ok, the mystery is solved: I was using Firefox as the browser for the followed user, and it seems that Firefox dies in the middle of replication with:

TypeError: address is undefined orbit-db-address.js:17

I guess that's the reason why the replication is not finished on the other side (where I was monitoring the console). If I use Chrome / incognito for the followed user, the replication seems to work. LMK if you want me to open a separate bug for replication not working in FF.

I'm getting a lot of events, though. Like 12 events for replicate on the keyvalue db and 4 events for replicated (which in my mind should be only one, since I replicate only one keyvalue). Same for the feed. Is that ok? As usual, LMK if I need to open a separate bug for too many events.

[cristiano-belloni]

Sorry, it seems I wrote too soon. Now with the same code I'm in the situation of the first post, even with incognito mode. The behaviour seems erratic, I'm kinda confused. I'll try to tear everything apart in the evening and see what's happening.

[haadcode]

I think I figured it out. This is indeed a bug!

What's happening is, I believe, that in the example above, OrbitDB is using the same local data directory for both peers, ie. they write to the same directory and the data gets messed up in a way that the feedDb gets the userDb events. I didn't find yet what exactly causes this and where it happens, but definitely something we want to fix!

Meanwhile, can you try giving the databases a unique local directory with: new OrbitDB(ipfs, <unique directory>) and see if that fixes the problem on your end? Here's an example I used to debug this: https://gist.github.com/haadcode/4bba3ede25e4f2bca191f886e7564c3e

@cristiano-belloni let me know if this fixes it. I'll keep digging as to how to fix it.

[haadcode]

Spoke too soon as to what's causing it 😄 Found the cause: ipfs-shipyard/ipfs-pubsub-room#25.

What's causing your problems @cristiano-belloni is that in our pubsub implementation, sometimes messages get handled in one topic and sometimes in the other and that's why you're seeing 50/50 chance for them to appear in one or the other.

I'll try to make a workaround fix for this today/tomorrow, but we'll need to fix it properly as discussed in the issue above.

[cristiano-belloni]

Thanks for your deep analysis @haadcode! If you need me to test something or run some branches, I'm happy to help :)

[haadcode]

Fix is coming soon! :)

[haadcode]

@cristiano-belloni this is now fixed in master (soon in npm as 0.18.3). Let me know if you still have the problem!

[haadcode]

Now published to npm as 0.18.3

[haadcode]

@cristiano-belloni we found out that the problem was actually in the message propagation layer in ipfs-pubsub-room. This was fixed by @pgte and the fixed version is now in orbit-db v0.18.4.

Make sure to pull in the latest! 😄

[cristiano-belloni]

@haadcode I will, thanks (currently using your 18.3 workaround and it works,btw!)