Peer discovery with kademlia for gossipsub

[Frederik-Baetens]

1. Is kademlia intended/useful for peers in a gossipsub network to discover eachother? If so, should I put the peer information in a kademlia DHT, or should I use kademlia's built in peer discovery directly to link my gossipsub peers with eachother as well?
2. How do I make my own kademlia cluster for peer discovery? I tried modifying the ipfs-kad example, but i'm not clear on how to replace the DNS with just an ip here, or even what information should be contained in the dns record:

        let bootaddr = Multiaddr::from_str("/dnsaddr/bootstrap.libp2p.io")?;
        for peer in &BOOTNODES {
            behaviour.add_address(&PeerId::from_str(peer)?, bootaddr.clone());
        }

[mxinden]

Is kademlia intended/useful for peers in a gossipsub network to discover eachother? If so, should I put the peer information in a kademlia DHT, or should I use kademlia's built in peer discovery directly to link my gossipsub peers with eachother as well?

In case you just want to discover any nodes within the network, I recommend doing a random walk (FIND_NODE with a random peer ID) on the DHT.

How do I make my own kademlia cluster for peer discovery?

1. Make sure to set a custom protocol name:

   rust-libp2p/protocols/kad/src/behaviour.rs

   Lines 218 to 226 in 320a1cd

   	/// Sets a custom protocol name.
   	///
   	/// Kademlia nodes only communicate with other nodes using the same protocol
   	/// name. Using a custom name therefore allows to segregate the DHT from
   	/// others, if that is desired.
   	pub fn set_protocol_name(&mut self, name: impl Into<Cow<'static, [u8]>>) -> &mut Self {
   	self.protocol_config.set_protocol_name(name);
   	self
   	}

2. Call Kademlia::add_address with the Multiaddr of one or many bootstrap nodes of your custom network.

I tried modifying the ipfs-kad example, but i'm not clear on how to replace the DNS with just an ip here, or even what information should be contained in the dns record:

Note, that any supported Multiaddr would work (e.g. /ip6/xxx). You can read up on /dnsaddr here.

Let me know if this helps @Frederik-Baetens.

[Frederik-Baetens]

Thanks a lot, that helps, I'm now able to run queries against ipfs using an ip adress I got by using dig on the dns records. My mistake was that I had left out the p2p/peerid from the multiaddr.

To create my own kademlia nodes and enable connecting to them, I also had to add swarm.listen_on("/ip4/0.0.0.0/tcp/0".parse()?).unwrap(); This enabled me to query my own kademlia nodes. Thanks a lot!

I'll try to check back once I'm finished to see if I there's interest in me contributing an example with combined kademlia + gossibsub chat.

[Frederik-Baetens]

I've gotten a bit further now, but I'm not clear on how to successfully bootstrap a new kademlia node so that other peers can find it.

My experiments thus far are at (modified version of the ipfs-kad example): https://github.com/Frederik-Baetens/libp2p-kad-peer_discovery/blob/main/src/main.rs

I have managed to start up a first node, cargo run -- 1234 (1234 being the port number)
Then I connect a second node to it, and am able to successfully query the peer id of the first node with cargo run -- 1235 /ip4/127.0.0.1/tcp/1234/p2p/12D3KooWM5ZRjdKyVy6e2UkKPJjDHcoeRix3gUMVbwCqGVsR1pxx (the required peer address is printed upon startup of the first node)
However, when I try to connect a third node to the first node, in similar fashion, I only get the peerId of the first node, and not also of the second node. cargo run -- 1236 /ip4/127.0.0.1/tcp/1234/p2p/12D3KooWM5ZRjdKyVy6e2UkKPJjDHcoeRix3gUMVbwCqGVsR1pxx
However, if I connect the third node to the second node, I do get the peerIds of both the first and second node.

So I don't know how I would inform the first node of the existence of the second node, and make the first node "advertise" the second node when queried.

I am bootstrapping my nodes with swarm.behaviour_mut().bootstrap().unwrap(); but that didn't really seem to make a difference.

[Frederik-Baetens]

On my first node, I am getting some unroutablepeer events:

[src/main.rs:146] evnt = UnroutablePeer {
    peer: PeerId(
        "12D3KooWQY2c8cBW8XBusby2MrcgKyztx7BWLtNEyXeQBFhKd9ZN",
    ),
}

But i don't know what to change in my second and third node so that they become routablepeer events on the first node.

[mxinden]

The documentation of UnroutablePeer might help:

    /// A peer has connected for whom no listen address is known.
    ///
    /// If the peer is to be added to the routing table, a known
    /// listen address for the peer must be provided via [`Kademlia::add_address`].

I am guessing that 2 connects to 1. Given that 1 does not know the listen address (but only the ephemeral dial address) of 2, 1 does not add 2 to its routing table. Given that 1 does not add 2 to its routing table, 1 can not return 2 to 3 when 3 queries 1.

For now I recommend doing the following: Use libp2p-identify to discover a remotes listen address. Once discovered, add the remote to the Kademlia routing table with that listen address. You can find an example here:

https://github.com/mxinden/rust-libp2p-server/blob/35aad8be33962d565ac8fe1cf63679418a1b1189/src/main.rs#L129-L156

[Frederik-Baetens]

Don't worry, it already got approved again shortly after I posted that comment here 😄

[deputinizer]

For now I recommend doing the following: Use libp2p-identify to discover a remotes listen address. Once discovered, add the remote to the Kademlia routing table with that listen address. You can find an example here:

https://github.com/mxinden/rust-libp2p-server/blob/35aad8be33962d565ac8fe1cf63679418a1b1189/src/main.rs#L129-L156

I think there is a mistake. You should add local_peer_id into DHT, not the remote peer_id.

https://github.com/mxinden/rust-libp2p-server/blob/35aad8be33962d565ac8fe1cf63679418a1b1189/src/main.rs#L152

[Spoiler] proof of duplicate IPs and weird behaviour

Using the remote peer_id just like in the example gives me duplicate IP addresses on multiple peers.

peer:12D3KooWR2KSRQWyanR1dPvnZkXt296xgf3FFn8135szya3zYYwY has addr:/ip4/146.1.2.3/tcp/18211
peer:12D3KooWR2KSRQWyanR1dPvnZkXt296xgf3FFn8135szya3zYYwY has addr:/ip4/213.1.2.3/tcp/21290

Debugging done with:

SwarmEvent::Behaviour(MyEvent::Kademlia(KademliaEvent::OutboundQueryCompleted{
    id,
    result: QueryResult::GetClosestPeers(Ok(GetClosestPeersOk { peers, .. })),
    ..
})) => {
    for peer in peers {
        let addrs = self.swarm.behaviour_mut().kademlia.addresses_of_peer(&peer);
        for addr in addrs {
            println!("peer:{} has addr:{}", peer, addr);
        }
    }
}

[mxinden]

I think there is a mistake. You should add local_peer_id into DHT, not the remote peer_id.

I am not sure I understand @deputinizer. One adds the remote peer with its addresses to your local routing table.

[deputinizer]

Ohh, sorry. I see you're using listen_addrs, not the identify_info.observed_addr. But then it's kind of useless to add LAN IP addresses into the routing table.

[mxinden]

But then it's kind of useless to add LAN IP addresses into routing the table.

Yes, that is an optimization worth doing.

Also, once we have Kademlia client mode (#2032) a node should not advertise the Kademlia protocol and thus not be added to any routing table when not public anyways.