Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Follow up: EBT improvments #75

Open
1 of 6 tasks
cryptix opened this issue Feb 4, 2021 · 1 comment
Open
1 of 6 tasks

Follow up: EBT improvments #75

cryptix opened this issue Feb 4, 2021 · 1 comment
Labels
bug Something isn't working

Comments

@cryptix
Copy link
Member

cryptix commented Feb 4, 2021

It would be really good to cover more of the unchecked tests in #72 (comment)

but mostly I'd love to see this stuff:

  1. Update the vector clock / want list not just once after connecting
    Currently the sets of feeds we want to receive is just send once when the connection is established.
    The JS documentation is a bit lacking about when to actually re-send those and thus confirm received messages.
    But also it doesn't send a clock update once we call sbot.Replicate(feed) - which means reception is delayed until a reconnect... at least this a part should be fixed before merging this.

  2. Implement a more scalable solution to canceling individual feed subscriptions
    Right now there is a context per feed. This works but is inefficient. Probably need to re-work the multisink so that they can be unsubscribed directly.

  3. Partition sets of feeds over multiple connections
    One of the advantages of ebt, other then not asking about feeds that didn't change, is receiving feeds from the best known source. For this we need a bit of heuristics, like ping to the peer and who has messages sooner.

Preliminary testing results: functional but a couple of glitches.

  • rebuilding of the own/self network frontier after changes on the replication lister (graph walk)
    The TODO above about this acknowledged a slight variation of this but it's actually worse. Right now, it just builds the network frontier for ebt when it doesn't have one.. any (un)follows after the first ebt connection are without effect.

  • Followgraph walk finishes to soon
    Tangential and slightly sure it was the case before these changes. The graph/builder walk along the follow messages sometimes collapses and returns to few results.

  • Untangle circular dependency between state matrix and graph builder (See PR Untangle statemat from replication #97)
    Before the graph builder was hooked up after the indexing system.
    The way it's setup now, the combined index needs the state matrix to update it with new received messages... which also needs the graph now for the first fill.
    This circle needs to be broken so that the server setup doesn't race.

@cryptix cryptix added the bug Something isn't working label Feb 4, 2021
@cryptix
Copy link
Member Author

cryptix commented Mar 8, 2021

@arj03 found a bug ( ssbc/epidemic-broadcast-trees#42 ) where a peer received a message, confirmed it but then looses it somehow.

It's a great test case (dealing with misbehaving remote peers correctly). Reproducing this right now would require stopping the go-ssb peer, truncating the log, re-indexing and then connecting again.

"reboot" tests are on the table anyhow to check the state matrix persistence worked as intended.

cryptix added a commit that referenced this issue May 7, 2021
cryptix added a commit that referenced this issue Sep 14, 2021
cryptix added a commit that referenced this issue Sep 16, 2021
updates #75

also:

* add noop to fill if called with zero feeds
* make sure to remove feeds from ebt if someone stops following them
cryptix added a commit that referenced this issue Sep 24, 2021
* open ebt.replicate for classic and just use replicateFormat for new
formats
* rename getClock to clock
* rig up JS ebt tests
* cleanup test setup
* upgrade muxrpc to v2.0.11
* update ebt state after replicate hops walks (updates #75)
* add noop to fill if called with zero feeds
* make sure to remove feeds from ebt if someone stops following them
* update js metafeeds
* update to latest changes in JS ebt branch
cryptix added a commit that referenced this issue Oct 8, 2021
* open ebt.replicate for classic and just use replicateFormat for new
formats
* rename getClock to clock
* rig up JS ebt tests
* cleanup test setup
* upgrade muxrpc to v2.0.11
* update ebt state after replicate hops walks (updates #75)
* add noop to fill if called with zero feeds
* make sure to remove feeds from ebt if someone stops following them
* update js metafeeds
* update to latest changes in JS ebt branch
cryptix added a commit that referenced this issue Oct 8, 2021
* use feature negotiation via muxrpc.HasMethod
* expose new muxrpc ebt calls
 * duplex:replicateFormat
 * async: getClock
* update go ebt to conform with latest JS ssb-ebt branch
* open ebt.replicate for classic
* just use replicateFormat for new formats
* rename getClock to clock
* rig up JS ebt tests
* cleanup test setup
* upgrade muxrpc to v2.0.11
* update ebt state after replicate hops walks (updates #75)
* add noop to fill if called with zero feeds
* make sure to remove feeds from ebt if someone stops following them
* update js metafeeds
* update to latest changes in JS ebt branch
* add some ebt frontier and note tests
cryptix added a commit that referenced this issue Oct 8, 2021
* use feature negotiation via muxrpc.HasMethod
* expose new muxrpc ebt calls
 * duplex:replicateFormat
 * async: getClock
* update go ebt to conform with latest JS ssb-ebt branch
* open ebt.replicate for classic
* just use replicateFormat for new formats
* rename getClock to clock
* rig up JS ebt tests
* cleanup test setup
* upgrade muxrpc to v2.0.11
* update ebt state after replicate hops walks (updates #75)
* add noop to fill if called with zero feeds
* make sure to remove feeds from ebt if someone stops following them
* update js metafeeds
* update to latest changes in JS ebt branch
* add some ebt frontier and note tests
cryptix added a commit that referenced this issue Dec 20, 2021
* use feature negotiation via muxrpc.HasMethod
* expose new muxrpc ebt calls
 * duplex:replicateFormat
 * async: getClock
* update go ebt to conform with latest JS ssb-ebt branch
* open ebt.replicate for classic
* just use replicateFormat for new formats
* rename getClock to clock
* rig up JS ebt tests
* cleanup test setup
* upgrade muxrpc to v2.0.11
* update ebt state after replicate hops walks (updates #75)
* add noop to fill if called with zero feeds
* make sure to remove feeds from ebt if someone stops following them
* update js metafeeds
* update to latest changes in JS ebt branch
* add some ebt frontier and note tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant