-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Follow up: EBT improvments #75
Comments
@arj03 found a bug ( ssbc/epidemic-broadcast-trees#42 ) where a peer received a message, confirmed it but then looses it somehow. It's a great test case (dealing with misbehaving remote peers correctly). Reproducing this right now would require stopping the go-ssb peer, truncating the log, re-indexing and then connecting again. "reboot" tests are on the table anyhow to check the state matrix persistence worked as intended. |
updates #75 also: * add noop to fill if called with zero feeds * make sure to remove feeds from ebt if someone stops following them
* open ebt.replicate for classic and just use replicateFormat for new formats * rename getClock to clock * rig up JS ebt tests * cleanup test setup * upgrade muxrpc to v2.0.11 * update ebt state after replicate hops walks (updates #75) * add noop to fill if called with zero feeds * make sure to remove feeds from ebt if someone stops following them * update js metafeeds * update to latest changes in JS ebt branch
* open ebt.replicate for classic and just use replicateFormat for new formats * rename getClock to clock * rig up JS ebt tests * cleanup test setup * upgrade muxrpc to v2.0.11 * update ebt state after replicate hops walks (updates #75) * add noop to fill if called with zero feeds * make sure to remove feeds from ebt if someone stops following them * update js metafeeds * update to latest changes in JS ebt branch
* use feature negotiation via muxrpc.HasMethod * expose new muxrpc ebt calls * duplex:replicateFormat * async: getClock * update go ebt to conform with latest JS ssb-ebt branch * open ebt.replicate for classic * just use replicateFormat for new formats * rename getClock to clock * rig up JS ebt tests * cleanup test setup * upgrade muxrpc to v2.0.11 * update ebt state after replicate hops walks (updates #75) * add noop to fill if called with zero feeds * make sure to remove feeds from ebt if someone stops following them * update js metafeeds * update to latest changes in JS ebt branch * add some ebt frontier and note tests
* use feature negotiation via muxrpc.HasMethod * expose new muxrpc ebt calls * duplex:replicateFormat * async: getClock * update go ebt to conform with latest JS ssb-ebt branch * open ebt.replicate for classic * just use replicateFormat for new formats * rename getClock to clock * rig up JS ebt tests * cleanup test setup * upgrade muxrpc to v2.0.11 * update ebt state after replicate hops walks (updates #75) * add noop to fill if called with zero feeds * make sure to remove feeds from ebt if someone stops following them * update js metafeeds * update to latest changes in JS ebt branch * add some ebt frontier and note tests
* use feature negotiation via muxrpc.HasMethod * expose new muxrpc ebt calls * duplex:replicateFormat * async: getClock * update go ebt to conform with latest JS ssb-ebt branch * open ebt.replicate for classic * just use replicateFormat for new formats * rename getClock to clock * rig up JS ebt tests * cleanup test setup * upgrade muxrpc to v2.0.11 * update ebt state after replicate hops walks (updates #75) * add noop to fill if called with zero feeds * make sure to remove feeds from ebt if someone stops following them * update js metafeeds * update to latest changes in JS ebt branch * add some ebt frontier and note tests
It would be really good to cover more of the unchecked tests in #72 (comment)
but mostly I'd love to see this stuff:
Update the vector clock / want list not just once after connecting
Currently the sets of feeds we want to receive is just send once when the connection is established.
The JS documentation is a bit lacking about when to actually re-send those and thus confirm received messages.
But also it doesn't send a clock update once we call sbot.Replicate(feed) - which means reception is delayed until a reconnect... at least this a part should be fixed before merging this.
Implement a more scalable solution to canceling individual feed subscriptions
Right now there is a context per feed. This works but is inefficient. Probably need to re-work the multisink so that they can be unsubscribed directly.
Partition sets of feeds over multiple connections
One of the advantages of ebt, other then not asking about feeds that didn't change, is receiving feeds from the best known source. For this we need a bit of heuristics, like ping to the peer and who has messages sooner.
Preliminary testing results: functional but a couple of glitches.
rebuilding of the own/self network frontier after changes on the replication lister (graph walk)
The TODO above about this acknowledged a slight variation of this but it's actually worse. Right now, it just builds the network frontier for ebt when it doesn't have one.. any (un)follows after the first ebt connection are without effect.
Followgraph walk finishes to soon
Tangential and slightly sure it was the case before these changes. The graph/builder walk along the follow messages sometimes collapses and returns to few results.
Untangle circular dependency between state matrix and graph builder (See PR Untangle statemat from replication #97)
Before the graph builder was hooked up after the indexing system.
The way it's setup now, the combined index needs the state matrix to update it with new received messages... which also needs the graph now for the first fill.
This circle needs to be broken so that the server setup doesn't race.
The text was updated successfully, but these errors were encountered: