Refactor go-kademlia to remove dependency on go-libp2p.

[iand]

ETA: 2023-08-18

Description: go-kademlia should not depend on go-libp2p but should be compatible with it. All libp2p functionality should move to go-libp2p-kad-dht version 2 (see libp2p/go-libp2p-kad-dht#855)

[guillaumemichel]

I have a mixed feelings about this.

On the one hand, if go-kademlia (to be renamed) is to be a tool box to build DHTs it can contain standard networking transports that could be used to build any DHT. I think that go-kademlia should define what a transport endpoint is, and ideally provide a few generic endpoint implementations (e.g libp2p, QUIC, TCP, UDP). This would allow go-kademlia users to build their DHTs without having to reimplement a libp2p or TCP endpoint themselves, or depend on go-libp2p-kad-dht which is a full DHT implementation.

However, on the other hand, I agree that go-kademlia should be a tool box to build DHTs, and hence shouldn't depend on libp2p, it is libp2p that has to depend on go-kademlia because it is using it to implement the go-libp2p-kad-dht. Also, if the libp2p endpoint contains go-libp2p-kad-dht specific enhancements (e.g backpressure), it wouldn't be considered to be generic enough to remain in go-kademlia.

I think that having a dependency on go-libp2p (not go-libp2p-kad-dht) should be seen like a dependency on TCP or UDP. So for me the question should be: do we want to give go-kademlia generic transport endpoints that they can already use in their DHT implementation, or should they build their transport endpoint themselves?

[Wondertan]

libp2p already abstracts over any known transport. There is not so much value in adding another layer of indirection to avoid depending on it, as user can choose whatever transport on the libp2p level.

[guillaumemichel]

This is not what I am saying.

This repository shouldn't solely depend on libp2p as a transport. It should be possible to build a DHT with go-kademlia using standard networking libraries as well (e.g net package).

[iand]

go-kademlia should definitely support more than libp2p by abstracting the transports, addressing and node identification (the last two are already abstracted)

I think the libp2p implementation of those types should live in go-libp2p-kad-dht. If you are in the libp2p world then you should be using go-libp2p-kad-dht. If you are building a custom DHT that uses, for example, the Ethereum p2p stack then that should also be possible to do cleanly.

[guillaumemichel]

OK, but in this case, I am in favor of having a clean module e.g go-libp2p-kad-dht/net that could be easily imported in different projects. We need to make it clear that go-kademlia + go-libp2p != go-libp2p-kad-dht.

For instance, if I want to create a DHT for my custom app, that would be using libp2p as transport, but I don't want provider records nor IPNS, I should be able to easily find and import go-libp2p-kad-dht/net only and not the rest of the go-libp2p-kad-dht repo that is not relevant for my project. The libp2p endpoint is useful beyond just the go-libp2p-kad-dht implementation.

I am slightly concerned that it becomes too complex to build a basic DHT using the building blocks that are available at go-kademlia if there are no implementations of endpoint. We shouldn't expect from developers that they build they own transport endpoint when exploring the repo. As an alternative, we could write a simple TCP endpoint using the net package and have the libp2p endpoint in the go-libp2p-kad-dht repo.

I think that transport endpoints that are generic enough to be used in multiple DHT implementations (e.g TCP, UDP, QUIC, libp2p, eth p2p, etc.) could be part of go-kademlia (or at least 1 should to serve as example).

[iand]

Superceded by migration of core types to go-libdht