peer discovery fail

[florianlenz]

Hi guy's,

in the last few day's I have been playing around with content / peer discovery. Discovery of content work's quite nice tho it's a bid slow (I wonder what it's so fast when using IPFS directly) but peer discovery doesn't work. When I search for a bootstrapping node (e.g. QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd) it fails after 2 - 5 minutes because the peer can't be found (error message is: routing: not found). I created an test repo here which you can use to reproduce the problem. The interesting thing is that I can call FindPeer on the DHT with any string. There is no validation, which might mean that I am using the wrong ID (tho I am sure that this QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd is a peer id in the libpp2 context). Can someone point me in the right direction where I can find help? I also had a look at whyrusleeping's chat example since it makes use of the DHT. But there it seem's to work.

Thanks for you help,
Florian

[Stebalien]

That's the base58 encoded peer ID. In this case, you need to pass in the decoded peer ID (which you can get by calling peer.IDB58Decode(myPeerId).

If you look at the signature, it actually takes a peer.ID. However, your code works because:

1. peer.ID is a "new type" of string (defined as type ID string).
2. You're passing in an untyped constant (with a default type string). In this case, however, go infers the type to be peer.ID.

[florianlenz]

@Stebalien Thanks. It's work's now. Peer was found in like 2 seconds (What a stupid mistake of me).

[florianlenz]

Ok, I am able to discover a bootstrap node, but I am not able to discover a second peer that I create. I updated the test repo. I create two peer's and two DHT's that I connect to the same bootstrapping nodes. I then call FindPeer on the second DHT with the ID from peer one, but I am not able to find the peer (this time I am using the ID correct). Maybe I am missing something obviously.

@Stebalien I think It would be great to have an example for how this work's. I am happy to add it when I figured it out. Should all the examples stay in this repo? In order to build the example I at least need to add the DHT as a dependencies to this go-libp2p project so I guess the dependency list will become quite huge ...

[Stebalien]

Yeah... So, I thought this might be an issue. libp2p/go-libp2p-kad-dht#147

[florianlenz]

I am closing this issue - my problem regarding peer discovery is solved. Look here for details: libp2p/go-libp2p-kad-dht#147

[Stebalien]

@florianlenz could you test libp2p/go-libp2p-kad-dht#160 against your code?

[florianlenz]

Sure - what's the best way to test it? the github import doesn't work. Do I have to make an gx release in order to test it?

[florianlenz]

@Stebalien it doesn't work for me. My code:

package main

import (
	"context"
	"fmt"
	dht "github.com/libp2p/go-libp2p-kad-dht"
	libp2p "github.com/libp2p/go-libp2p"
	"github.com/ipfs/go-datastore"
	"github.com/libp2p/go-libp2p-host"
	"github.com/libp2p/go-libp2p-peerstore"
	"github.com/multiformats/go-multiaddr"
)

var BootstrapPeers = []string{
	"/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
	"/ip4/104.236.76.40/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64",
	"/ip4/104.236.176.52/tcp/4001/ipfs/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z",
	"/ip4/104.236.179.241/tcp/4001/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM",
	"/ip4/162.243.248.213/tcp/4001/ipfs/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm",
	"/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu",
}

func config(cfg *libp2p.Config) error {
	return libp2p.Defaults(cfg)
}

func peerFactory() (*dht.IpfsDHT, host.Host) {

	//Create host
	h, err := libp2p.New(context.Background(), config)
	if err != nil {
		panic(err)
	}

	//Create DHT
	d := dht.NewDHTClient(context.Background(), h, datastore.NewMapDatastore())

	for _, p := range BootstrapPeers {
		//Bootstrap object
		ma, err := multiaddr.NewMultiaddr(p)
		if err != nil {
			panic(err)
		}
		
		info, err := peerstore.InfoFromP2pAddr(ma)
		err = h.Connect(context.Background(), *info)
		if err != nil {
			panic(err)
		}
	}
	
	//Exit on DHT bootstrap error
	if err := d.Bootstrap(context.Background()); err != nil {
		panic(err)
	}

	return d, h

}

func main() {

	dhtAlice, _ := peerFactory()
	_, hostBob := peerFactory()
	
	fmt.Println("created peers")

	fmt.Println(dhtAlice.FindPeer(context.Background(), hostBob.ID()))

}

I checked out fix/147 in the dht repo.

[Stebalien]

Thanks! I'll take a look.

[Stebalien]

It looks like we wait 5m before the initial bootstrap... 😞

[Stebalien]

Also, this is really slow. It shouldn't be that slow.

[florianlenz]

By initial bootstrap you mean the part where we connect to the bootstrapping nodes? It took ~40 seconds for me to create an bootstrap the two dht's.

[florianlenz]

ok, now calling findPeer on your self and then calling findPeer with another DHT on the first peer doesn't work as well anymore. Very strange - since i tested that a few days ago.

[Stebalien]

So, turns out you needed to tell libp2p to listen on some external port (e.g., libp2p.ListenAddrStrings("/ip4/0.0.0.0/tcp/0")`).

I wonder if there's a good way to warn about this. It's a rather nasty silent bug.

[florianlenz]

It's still failing for me. I added /ip4/0.0.0.0/tcp/0 via the config passed in when creating a new instance. Code:

package main

import (
	"context"
	"fmt"
	dht "github.com/libp2p/go-libp2p-kad-dht"
	libp2p "github.com/libp2p/go-libp2p"
	"github.com/ipfs/go-datastore"
	"github.com/libp2p/go-libp2p-host"
	"github.com/libp2p/go-libp2p-peerstore"
	"github.com/multiformats/go-multiaddr"
)

var BootstrapPeers = []string{
	"/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
	"/ip4/104.236.76.40/tcp/4001/ipfs/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64",
	"/ip4/104.236.176.52/tcp/4001/ipfs/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z",
	"/ip4/104.236.179.241/tcp/4001/ipfs/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM",
	"/ip4/162.243.248.213/tcp/4001/ipfs/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm",
	"/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu",
}

func config(cfg *libp2p.Config) error {
	addr, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/0")
	if err != nil {
		return err
	}
	cfg.ListenAddrs = []multiaddr.Multiaddr{
		addr,
	}
	return libp2p.Defaults(cfg)
}

func peerFactory() (*dht.IpfsDHT, host.Host) {

	//Create host
	h, err := libp2p.New(context.Background(), config)
	if err != nil {
		panic(err)
	}
	
	//Create DHT
	d := dht.NewDHTClient(context.Background(), h, datastore.NewMapDatastore())
	
	for _, p := range BootstrapPeers {
		//Bootstrap object
		ma, err := multiaddr.NewMultiaddr(p)
		if err != nil {
			panic(err)
		}
		
		info, err := peerstore.InfoFromP2pAddr(ma)
		err = h.Connect(context.Background(), *info)
		if err != nil {
			panic(err)
		}
	}

	//Exit on DHT bootstrap error
	if err := d.Bootstrap(context.Background()); err != nil {
		panic(err)
	}

	return d, h

}

func main() {

	dhtAlice, _ := peerFactory()
	_, hostBob := peerFactory()
	
	fmt.Println(dhtAlice.FindPeer(context.Background(), hostBob.ID()))
	
}

[Stebalien]

That's libp2p/go-libp2p-kad-dht#163, unfortunately.

[Stebalien]

Honestly, this entire thing is embarrassing. Thanks for sticking with it.

[florianlenz]

Never mind. LibP2P is such an awesome project and safed us sooo many hours of work (the organisation I am working with / in) that those little bugs are totally acceptable :)

[Stebalien]

safed us sooo many hours of work (the organisation I am working with / in

Awesome! That's why it exists. It's just kind of awkward when the simple test case of creating two peers and connecting them to each other doesn't just work out of the box 😆.

[florianlenz]

Btw, that just brings up another question. Do we (community) / LibP2P plan to build a integration testing framework? I guess that those problems can be spotted earlier by having integration tests. It would also nice to build test suites that show how the network behaves under certain situations. Or to test new implementations. I am e.g. implementing Quasar based on LibP2P. I would like to see how the implementation behaves in a network with 30K, 60K and 90K people and if the message delivery rate stays the same. I think there would definitely be a benefit of having such a testing framework. I will need such an framework in the future. It would be nice to create it based on community needs.

[Stebalien]

Yes. protocol/research-grants#4

[anacrolix]

What's the status of this issue? Should it be closed?

[Stebalien]

Yes. This is fixed. Libp2p is now configured with default transports.