Improve/switch underlying protocol implementation

[grrowl]

Currently, we use scuttlebutt which is still maintained but the maintainer has mostly moved to interest in secure-scuttlebutt and we've outgrown it.

decisions

• scuttlebutt (current superclass)
  • Doesn't provide unforgeability (peers can lie about other peers)
  • ✅ Low-level access to ops
• secure-scuttlebutt
  • Lots of heavy dependencies
  • ✅ Uses secret-handshake
  • Generally "scuttlebutt done right" and ALL the trimmings.
  • ❌ Doesn't run in the browser
• Swarm.js
  • ✅ We could extend Syncable, which provides an Op interface
  • Also defaults logical timestamps
  • Secure, with plenty of work and thought put into it
• simple-scuttle
  • ✅ Light and excellent
  • Multiple peers re-use the same stream — so actions are not replayed to individual peers as needed. Replays are broadcast to all peers when any peer connects.
  • In my testing, did not always send all actions to all peers. Required manual re-syncing sometimes.

Objectives

• Lightweight
• Stream actions directly
• (less so: state snapshots, "rooms" subscription)
• Secure from malicious peer we can implement signatures/crypto in userland or higher-order reducer
• Subscriptions or channels would be nice (how far to propagate? TTL?)

[moimikey]

is simple-scuttle an option? what are your thoughts on it in comparison to dominic's implementation?

[moimikey]

it would be great to also have the option to hook into dispatcher.js#wrapDispatch to filter explicit action types. for example, my state tree has a browserInfo piece of state that needs to be updated locally, but not dispatched to peers like actions that are prepended with @. my workaround has been to open a PR with some third-party store enhancers and middlewares that dispatch actions that don't follow the format of @@vendor/action-type, and update them to include it, so i don't have to worry about propagating unnecessary messages.

[grrowl]

simple-scuttle looks really nice, I failed to flesh out this public list as it ended up pretty critical of these excellent projects and didn't want to appear to be sledging. Initially I hesitated on using simple-scuttle as it exposes a state-based, versioned-keys datatype, whereas I wanted to implement a more pure op-based stream as it maps perfectly to redux actions, leaving state wholly to the reducers. It exposes .history but I'd rather remove/reimplement the state itself.

That said, redux-scuttlebutt priorities have changed somewhat. I have a userland crypto/action-signing solution and I think userland implementation is the way to go for this. Subscriptions can be multiple, parallel Scuttlebutt streams (and should be treated as distinct data streams anyway). I'll take a deeper look today.

Also, I pushed the latest branch memory here to github, which is published on npm as redux-scuttlebutt@0.0.4-alpha.3 which I'm using for my own projects. It doesn't enable overriding isGossipType but I would certainly accept a PR or look at adding this as an option in future.

[moimikey]

it makes sense that you went with the approach that you ended up going. regarding .history, it's a good to have. I wouldn't remove it. the idea of reimplementing would be interesting perhaps as a separate module; redux-scuttlebutt-history mayhaps.

I have a userland crypto/action-signing solution and I think userland implementation is the way to go for this

yessss! 👍

—i'm currently using this library for a personal react/redux project and I must say, it's incredible how simple it was to implement. I was previously aware of scuttlebutt and gossip protocol. Also a big fan of Dominic and his mad science. It was only natural that this library provided me the best of both worlds.

i'd be more than happy to open a PR :) i'll take care of that this weekend.

[grrowl]

Attempted to move to simple-scuttle over the weekend. The set up was three peers, (two browsers and a server) —

• updates were not automatically forwarded along to secondary peers (when A->B, B didnt always forward updates to C)
• Since there's not multiple streams for a single Gossip instance, it seems like any broadcasted updates will go to all connected peers. Newly connected peers' updates (i.e. all of them on cold start) will be sent to all peers.
• Had to call .gossip() to sometimes get updates from peers. Seems like the vector clocks weren't updating correctly for all/current peers? many version numbers were in the past and therefore not re-sent.

Anyway, I couldn't get it to work reliably in my tests. I've saved my progress but in meantime have forked dominictarr/scuttlebutt, have implemented vector-clock-class, and will implement a few improvements (such as always replaying updates in timestamp order) and start stripping out irrelevant code.

[grrowl]

Preparing for a relatively major 0.2.0 release

• Added support for options.isGossipType
• Uses logical timestamps, not wall-clock timestamps (finally)
• We're currently depending on github:grrowl/scuttlebutt#logical-timestamps.
  • Haven't made any PRs for dominictarr/scuttlebutt as we're making a few breaking changes to the project and I'd like to have a more solid case for doing so before we do.
  • I suppose we'll stick with this forked implementation for now. No other library is as small and quick and reliable with such direct access to low-level Ops.
• The user-land signing I mentioned before can be found here
  • Will likely release properly in a more consumable form in the future. This works well for my use-case for now. Let me know if you need example usage code.