distributed zulip store: restore zulip from on-chain mirror

[dckc]

@leithaus asked if the on-chain mirror is sufficient to stand up another zulip server with the same history, ready to continue.

This restore feature depends on getting the mirror / backup part to be reliable as well:

• on-chain mirror zulip chat messages to rchain in near-real-time rvote#19
• delivery, order mirror doesn't ensure order nor delivery #3
• enough phlo chain_replica.js doesn't supply enough phlo rvote#43
• validAfterBlockNumber chain_replica.js doesn't update validAfterBlockNumber after start #6
• no 1 validator should be a SPOF chain_replica.js only uses one validator #5
• mirror all tables only 2 zulip tables are created on chain #4

[dckc]

@leithaus w/ @steverosstalbot are taking the ball

critical for the goal of decentralizing the chat

[dckc]

When asked what he might like to see, @leithaus said taking one zulip service with its store on rchain, and then standing up a new zulip instance and backing it up from rchain would be a cool end-to-end demo of distributed storage for a chat service.

[dckc]

GSJ writes:

Next time we meet, talk about what needs to be done to recover Zulip from rchain?

As we discussed, chain_replica.js adds a postgres trigger to each table for op=INSERT, UPDATE, and DELETE and when the trigger fires, it deploys something of the form db!(${lit(op)}, ${lit(table_name)}, ${lit(OLD)}, ${lit(NEW)}, Nil)

where OLD and NEW are the contents of the record before and after the operation. There is no OLD in the op=INSERT case (nor NEW in the DELETE case), so we use Nil in that case. An example from the README is:

new deployerId(`rho:rchain:deployerId`) in {
    for(db <<- @{[*deployerId, "zulip_iddb3"]}) {
        db!("INSERT", "zerver_reaction", Nil,
        {"id":11,"user_profile_id":8,"message_id":59,
         "emoji_name":"heart","emoji_code":"2764",
         "reaction_type":"unicode_emoji"}, Nil)
    }
  }

To restore Zulip from rchain:

1. look up the db
2. get the set of tables
3. for each table
   1. get the set of keys
   2. for each key
      1. get the contents of the record at this key in this table
      2. Insert it into postgres
         • perhaps write it to a file in JSON or CSV format and then load into postgres in batch in a separate step

Details / limitations:

1. myzulipdb.rho should share a read-only facet in the registry (#32).
   • work-around: if you have the deployer private key, you can find the db at @{[*deployerId, "zulip_iddb3"]}
   • "zulip_iddb3" should probably be parameterized
2. iddb.rho seems to be missing a "get tables" method. Oops (also part of iddb read-only facet #9).
3. we do have contract self(@"keys", @table_name /\ String, return).
   • The read-only facet should have a "get record" method too (#32).
   • work-around: if you have the closely-held db, the contents of each record is stored at @{[*db, table_name, key]}, so you can get it (using peek).

[dckc]

Why did you assign this to me? What do you expect me to do?

…

On Sat, Jul 25, 2020, 12:47 PM Jim Whitescarver ***@***.***> wrote: Assigned rchain-community/rv2020#24 <https://github.com/rchain-community/rv2020/issues/24> to @dckc <https://github.com/dckc>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/rchain-community/rv2020/issues/24#event-3586447522>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABE3SRX535WAK3HH4ZTRUTR5MLBRANCNFSM4OA3V5LQ> .

[dckc]

This is now nice-to-have