add periodic bootstrapping #583

jbenet · 2015-01-16T22:40:55Z

This PR adds periodic bootstrapping. This includes
the code from #582 as i want to test it in solarnet
before merging.

btc · 2015-01-16T23:57:29Z

core/bootstrap.go

@@ -86,7 +86,7 @@ func bootstrap(ctx context.Context,

 	// we can try running dht bootstrap even if we're connected to all bootstrap peers.
 	if len(h.Network().Conns()) > 0 {
-		if err := r.Bootstrap(ctx, numDHTBootstrapQueries); err != nil {
+		if _, err := r.Bootstrap(); err != nil {


Does this mean that core bootstrap does not return?

Oh. Nevermind. It does return.

Does the closer need to be returned to the caller so the core can take ownership?

yes, but if we do that, we should do that with regular supervise connections process, thats what the core should have a handle to.

Spitballing: maybe we want to extract the connection supervisor just as we've extracted the reprovider. (We'd still invoke it in the same place.)

jbenet · 2015-01-17T11:58:58Z

the test that failed seems to not have run: https://build.protocol-dev.com/job/epic/471/console

btc · 2015-01-17T12:01:11Z

re-running...

jbenet · 2015-01-19T12:36:26Z

rebased on new master, including #602

jbenet · 2015-01-19T14:15:10Z

this is a pretty good improvement: http://ipfs.benet.ai:8080/ipfs/QmbesKpGyQGd5jtJFUGEB1ByPjNFpukhnKZDnkfxUiKn38/chord#QmNtWftjuEzj4F2bWv2jiZGABNS47DhFWkjGrqzhWRropw

there's still something odd with the "clusters" (the routing table seems to divide into sections). if these are proper XOR-distance kbucket forming, great. but there's pretty sharp boundaries, which doesnt make sense. It may have to do with the containers unable to dial to each other.

@whyrusleeping let's get net-diag to output the routing table (still send to all connected peers, but each diag output is just your routing table).

jbenet · 2015-01-19T14:15:28Z

anyway, this is all RFCR -- but CR #602 first

whyrusleeping · 2015-01-19T20:33:13Z

I dont think the tests are large scale enough to show proper K-Buckets forming. In order to show this, im looking into giving dhthell the ability to use mocknet so we can build networks of over 1000 ( 9000 ) nodes.

jbenet · 2015-01-19T23:10:26Z

👍 that would help test the algorithm

—
Sent from Mailbox

On Mon, Jan 19, 2015 at 12:33 PM, Jeromy Johnson notifications@github.com
wrote:

I dont think the tests are large scale enough to show proper K-Buckets forming. In order to show this, im looking into giving dhthell the ability to use mocknet so we can build networks of over 1000 ( 9000 ) nodes.

Reply to this email directly or view it on GitHub:
#583 (comment)

jbenet · 2015-01-20T17:07:11Z

RFCR @whyrusleeping @briantigerchow -- it should be merged today.

Fixes #572

See the discussion below. A future commit will implement the closer change below, and rebase this one on top. <•jbenet> `n.Diagnostics.GetDiagnostic(time.Second * 20)` is not being respected. should it use a context instead? or is it a timeout because the timeout is sent to other nodes? <•jbenet> oh it's that the io doesnt respect the context so we're stuck waiting for responses. <•jbenet> this is that complex interface point between the world of contexts, and the world of io. ctxutil.Reader/Writer is made for this, but you have to make sure to defer close the stream. (see how dht_net uses it). i'd love to find a safer interface. not sure what it is, but we have to a) respect contexts, and b) allow using standard io.Reader/Writers. Maybe TRTTD <•jbenet> is have ctxutil.Reader/Writer take ReadCloser and WriteClosers and always close them. the user _must_ pass an ioutil. NopCloser to avoid ctxutil closing on you when you dont want it to. <•jbenet> this seems safer to me in the general case.

@whyrusleeping

Not sure this works. we dont have tests for net diag. We should make some. cc @whyrusleeping.

When some queries finished, but we got no result, it should be a simple NotFoundError. Only when every single query ended in error do we externalize those to the client, in case something major is going wrong

s/kademlia calls for makign sure to query all peers we have in our routing table, not just those closest. this helps ensure most queries resolve properly.

Moved it to its own package to isolate scope.

Many times, a node will start up only to shut down immediately. In these cases, reproviding is costly to both the node, and the rest of the network. Also note: the probability of a node being up another minute increases with uptime. TODO: maybe this should be 5 * time.Minute

jbenet · 2015-01-23T13:43:41Z

dd9c1b6...5c33b75 is the new part. If there are remaining concerns about the Bootstrap setup/teardown, I can address them as another PR. We're all merging a ton of stuff today, and best to stay close to one branch.

add periodic bootstrapping

…com/libp2p/go-libp2p-routing-helpers-0.2.1 build(deps): bump github.com/libp2p/go-libp2p-routing-helpers from 0.2.0 to 0.2.1

jbenet added the status/in-progress In progress label Jan 16, 2015

jbenet force-pushed the bootstrap-fix branch from 45b8833 to dcdd63e Compare January 16, 2015 23:56

btc reviewed Jan 16, 2015
View reviewed changes

jbenet force-pushed the bootstrap-fix branch 5 times, most recently from b3e3432 to 1982210 Compare January 19, 2015 12:36

jbenet force-pushed the bootstrap-fix branch from 1982210 to 71aa5f9 Compare January 19, 2015 13:17

whyrusleeping added this to the α milestone Jan 19, 2015

jbenet force-pushed the bootstrap-fix branch 6 times, most recently from fce10c6 to 0e27820 Compare January 20, 2015 17:05

jbenet changed the title ~~add periodic bootstrapping #572~~ add periodic bootstrapping Jan 20, 2015

jbenet force-pushed the bootstrap-fix branch 2 times, most recently from 9bae25c to aae09ae Compare January 20, 2015 17:15

jbenet self-assigned this Jan 20, 2015

jbenet mentioned this pull request Jan 21, 2015

multiaddrs with .../ipfs/... #608

Merged

jbenet force-pushed the bootstrap-fix branch from aae09ae to 8a5f9a1 Compare January 21, 2015 00:20

jbenet and others added 17 commits January 23, 2015 02:08

routing/dht: periodic bootstrapping #572

82d38a2

diag/net: add timeout param to cmd

898b969

net/diag: recursively decrement timeouts.

f627873

Not sure this works. we dont have tests for net diag. We should make some. cc @whyrusleeping.

stream back diagnostic responses as they are received

65b657e

try less aggressive bootstrap

8966743

core: call dht bootstrap

ec848c4

dht/bootstrap: logging

1493c9d

dht/bootstrap: timeout queries

9cd975c

dht/query: err return NotFound case

4865361

When some queries finished, but we got no result, it should be a simple NotFoundError. Only when every single query ended in error do we externalize those to the client, in case something major is going wrong

dht: kick off all the queries wit every node in our rt

5259cf0

s/kademlia calls for makign sure to query all peers we have in our routing table, not just those closest. this helps ensure most queries resolve properly.

p2p/proto/mux: make log more useful

8e9413b

p2p/proto/id: more helpful log

773ee2e

ipfs swarm peers: sort output

010cedf

updated goprocess, for periodic

c43f97d

core/bootstrap: cleaned up bootstrapping

d6ce837

Moved it to its own package to isolate scope.

core/bootstrap: CR comments

dd9c1b6

jbenet force-pushed the bootstrap-fix branch from d002bad to dd9c1b6 Compare January 23, 2015 10:08

jbenet added 3 commits January 23, 2015 05:25

core: cleaned up bootstrap process

95d58b2

p2p/net/conn: timeouts are real failures.

5c33b75

jbenet force-pushed the bootstrap-fix branch from 41745d7 to 5c33b75 Compare January 23, 2015 13:25

jbenet added a commit that referenced this pull request Jan 23, 2015

Merge pull request #583 from jbenet/bootstrap-fix

343940d

add periodic bootstrapping

jbenet merged commit 343940d into master Jan 23, 2015

jbenet deleted the bootstrap-fix branch January 23, 2015 13:43

jbenet removed the status/in-progress In progress label Jan 23, 2015

This was referenced Jan 23, 2015

various failures in multicore configuration jbenet/goprocess#2

Closed

DHT Bootstrap issue #572

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add periodic bootstrapping #583

add periodic bootstrapping #583

jbenet commented Jan 16, 2015

btc Jan 16, 2015

btc Jan 16, 2015

jbenet Jan 17, 2015

btc Jan 17, 2015

jbenet commented Jan 17, 2015

btc commented Jan 17, 2015

jbenet commented Jan 19, 2015

jbenet commented Jan 19, 2015

jbenet commented Jan 19, 2015

whyrusleeping commented Jan 19, 2015

jbenet commented Jan 19, 2015

I dont think the tests are large scale enough to show proper K-Buckets forming. In order to show this, im looking into giving dhthell the ability to use mocknet so we can build networks of over 1000 ( 9000 ) nodes.

jbenet commented Jan 20, 2015

jbenet commented Jan 23, 2015

add periodic bootstrapping #583

add periodic bootstrapping #583

Conversation

jbenet commented Jan 16, 2015

btc Jan 16, 2015

Choose a reason for hiding this comment

btc Jan 16, 2015

Choose a reason for hiding this comment

jbenet Jan 17, 2015

Choose a reason for hiding this comment

btc Jan 17, 2015

Choose a reason for hiding this comment

jbenet commented Jan 17, 2015

btc commented Jan 17, 2015

jbenet commented Jan 19, 2015

jbenet commented Jan 19, 2015

jbenet commented Jan 19, 2015

whyrusleeping commented Jan 19, 2015

jbenet commented Jan 19, 2015

I dont think the tests are large scale enough to show proper K-Buckets forming. In order to show this, im looking into giving dhthell the ability to use mocknet so we can build networks of over 1000 ( 9000 ) nodes.

jbenet commented Jan 20, 2015

jbenet commented Jan 23, 2015