Find self on bootstrap

[Stebalien]

We need to find ourself in the DHT to properly bootstrap the routing table. Finding random IDs will connect us to the rest of the DHT but we need to find ourself to:

1. Connect to our neighbors.
2. Distribute our peer addresses to our neighbors.

This is kind of critical for proper DHT functionality. Actually, in addition to finding our own peer, I wonder if we should be trying to find a key in each bucket (or some sampling thereof)?

[florianlenz]

I would like to work on this issue, since the project I am working on depends on it. We need to find ourself in the DHT to properly bootstrap the routing table. What exactly does that mean? When I create a instance of the DHT I am passing in a host so the DHT exist in the context of a host (or am I wrong)?

[ghost]

What exactly does that mean

Take our own PeerID, call findpeer with it, and see where the results lead us :)

It'll result in the same peers as the ones we'd write IPNS records to.

Finding random IDs will connect us to the rest of the DHT

This is already being done and is dubbed "random walk". Here we generate random PeerIDs and call findpeer. We'll of course never find addresses for exactly these PeerIDs, but the DHT will yield addresses for PeerIDs close by in address space.

[Stebalien]

This is already being done and is dubbed "random walk". Here we generate random PeerIDs and call findpeer. We'll of course never find addresses for exactly these PeerIDs, but the DHT will yield addresses for PeerIDs close by in address space.

FYI, we only do one random walk as far as I can tell.

[florianlenz]

@lgierth I tried that but I get a panic: routing: not found. Just to be 100% sure that I got you right: I have a peer A and create a DHT (Ad) with it. Then I call findPeer on Ad.

[Stebalien]

@florianlenz Yes. That's happening because, the first time you call FindPeer, none of the nodes actually know where to find your peer. Therefore, they'll return a "not found" error; you should ignore it. However, by trying to find your own peer, you'll end up connecting to the peers responsible for remembering your routing information which will implicitly (unfortunately) give them this information.

[florianlenz]

@Stebalien I tried to find my own peer but it didn't help. I also tried to put / get a value to / from the DHT with the same error. 17 / 20 reject the dial when I put a value in the DHT. Is it a requirement that the value needs to be put to 20 of my closer peers or something that is nice to have? It would be great to have a few examples like in the go-libp2p repo. I would add them but I am failing to get something that works running.

[upperwal]

If i am getting it right, any peer, let's say A executes IpfsDHT.FindPeer(A) it should return PeerInfo of A and not not found. This happens because when we initialise a new IpfsDHT, rounting table won't contain "self" peer.

Now, if A executes IpfsDHT.FindPeer(Z) where Z is some other peer. A might get PeerInfo of Z but from some other peer which was storing Z's information even if Z received a dhtQuery for the same.

A won't receive any PeerInfo if no other peer has Z's information even if Z received a dhtQuery message as Z won't be able to identify self.

If the above is correct, we can try to update (insert) the LocalPeer after creating the routing table. This way the routing table will contain the LocalPeer in the first bucket and IpfsDHT.FindPeer will always return it.

@Stebalien let me know if this is correct or not.

[Stebalien]

@florianlenz

Is it a requirement that the value needs to be put to 20 of my closer peers or something that is nice to have?

20 of the peers closest to the key you're trying to use, yes. This is an issue that @whyrusleeping is currently tackling. The problem is that GetClosestPeers returns a list of 20 peers closest to the key you're trying to put but it doesn't check if you can actually connect to those peers (and PutValue requires that all of those connections succeed).

I tried to find my own peer but it didn't help.

Are you now able to actually connect to the other peers in your cluster (is peer discovery working properly)?

[Stebalien]

@upperwal

The point here is not to actually find our own peer info in the DHT, it's to connect to the nodes responsible for remembering our peer info (so they can learn our peer info and answer future FindPeer requests).

If i am getting it right, any peer, let's say A executes IpfsDHT.FindPeer(A) it should return PeerInfo of A and not not found. This happens because when we initialise a new IpfsDHT, rounting table won't contain "self" peer.

The opposite. The first time A calls FindPeer(A), it will likely return not found. This is because the peers responsible for remembering information about us won't know about us yet because we've never contacted them. Future calls to FindPeer(A) should succeed.

Note: If our routing table is empty, we'll always get an error. That's why connecting to some well known bootstrap nodes is an important first step.

[florianlenz]

@Stebalien

Are you now able to actually connect to the other peers in your cluster (is peer discovery working properly)?

It works when calling findPeer on my self first.

[Stebalien]

Got it. Just making sure I had diagnosed that issue correctly.

[upperwal]

@Stebalien I see. Understood now. Thanks.