DHT support

[HelloZeroNet]

Similar to torrent's DHT solution, required features:

• Find peer to specified site file (to allow sites to add optionally download files)
• Ipv6 and tor address support for peers
• It should work using on Tor (TCP only, maybe UDP on clearnet)
• No other library dependency if possible
• Store peer ips to allow work without bootstrap server

Maybe its requred to separate the DHT from Peer Exchange

Please comment if you have other ideas/suggestions.

[defnax]

why not use libtorrent?, twister uses libtorrent for DHT
http://twister.net.co/?page_id=54

[HelloZeroNet]

zeronet protocol is different from torrent, so libtorrent will not work. Also bittorrent DHT is UDP based, so it will not work on Tor without proxying it to TCP which I don't want. And it does not support storing Tor hidden service addresses, so you would not be able to create sites on Tor.

[TheNain38]

@HelloZeroNet You could add to your first comment:

• Be able to store I2P and Tor hidden service addresses
• Don't use UDP to be compatible with Tor

[HelloZeroNet]

Its already there:

• Ipv6 and tor address support for peers
• It should work using on Tor (TCP only, maybe UDP on clearnet)

[TheNain38]

@HelloZeroNet Ho, didn't see that, sorry

[alxbob]

https://github.com/Ayms/node-Tor

https://pypi.python.org/pypi/pyp2p

[alxbob]

Full tor support and hidden services can work inside zeronet by creating a dht (that would be a tor hidden service it self) that holds the .onion addresses and load balances them. In clearnet it could work as a common dht. So its users connecting to zeronet through tor will have a hidden service created for its site he/she visits. And let tor hdirs take care the rest!

https://pypi.python.org/pypi/OnionBalance

https://stem.torproject.org/tutorials/over_the_river.html

[pldubouilh]

Interesting issue. Zeronet would drastically benefit from a DHT, as trackers tend to come and go with time...

My bet is that it would be best to bootstrap and use Bittorrent's DHT (Mainline DHT - MLDHT), as although it has a few problems, a lot of effort is put into it to keep it reliable and stable. A simple approach could be done using announce_peer and get_peers. Some more fancy stuff could be done using the post-signed-mutable-content thing (see bep_0044), but that might not be interesting for Zeronet.

For instance I (new user) query the MLDHT : get_peers(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D)) and retrieve a list of the peers on it - simple. The peers would have to first announced themselves using (roughly) announce_peer(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D), my_ip_and_port)

About the dependency, I think it's inevitable, at least for some things like bencode and the MLDHT, although there seems to exists relatively small python libraries. I'll have a look at this.

[SeniorPlayer]

Can the DHT solves the problem that how a new client first join the zeronet when the tracker server is down?

[HelloZeroNet]

@stillwarter DHT still need a bootstrap server, so it does not help on new "virgin" clients.

[obv-mikhail]

@HelloZeroNet Maybe this https://github.com/maidsafe/routing or this https://github.com/maidsafe/crust can be used.

[alugarius]

Custom bootstrap clients (storing all of them is nice too) , some proxys and an option to save an anonymous list of users to use as emergency bootstrap, it can be stored(exported and imported)! The "unvirginizer"
( IPFS tactics)

[Bachstelze]

Could we use https://github.com/closeio/redis-hashring in Python or https://github.com/RJ/ketama in C?

[sim590]

Hi,

I propose to you to consider using OpenDHT. It has the following features:

• C++11 (maybe C++14 somewhat soon);
• Python bindings;
• IPv4 and IPv6 support;
• Nice crypto layer for handling encrypted blobs and signatures;
• Transition from 160-bits SHA1 hashes to 256-bits hashes is in progress;
• TCP support is work in progress (@aberaud may comment on the status);
• Distributed indexing: it literally permits to do searches on complex queries rather than exact key lookups. This is based on PHT data structure although we have added substantial changes to support Fully distributed indexing which are documented into an article to be presented at UNet2017. This is more than relevant for this project. You can compare this to providing Google search features on ZeroNet;
• It is actively used by the software called Ring which is driven to deliver distributed and secure p2p communications to its users. More people use the same DHT, more the apps based on it will behave well.

I personally would like more projects to join like Syncthing.

[slothbag]

IPv6 for CJDNS would be fantastic.

[MuxZeroNet]

I talked with Shortcutme with this issue. Here are the questions he is trying to find an answer to.

Some months ago I did some experiments and checked out some [DHT] libraries, but there are still lots of questions left.

• Can it reliably work over Tor (or over TCP at all)?
• DHT routing in ZeroNet must support .onion and .i2p addresses.
• Should we use an existing DHT implementation (eg. Mainline, OpenDHT) or create a new one?
• Is there any alternative to DHT that worth checking out?

Here are my questions:

• Nodes in the network hosts their own sets of sites and optional files, based on user interests. We still want to choose what to store and what not to store. There is no strictly defined peer responsibilities. Should me adjust current DHT implementations to reflect the current circumstances of the ZeroNet network?

[d-roak]

Which DHT protocol do you use ? @HelloZeroNet

[alugarius]

Does ZNet made any progress in this issue?

[HelloZeroNet]

New progress made yet.
Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

[skwerlman]

this might be of interest. It's not reimplementing DHT using TCP, but instead it just carries the UDP traffic through a TCP tunnel (with the express intent of letting it be carried by Tor)

[gitbugged]

Very interested in getting DHT, but in the interim with trackers getting blocked and dropped, maybe you could add more trackers?
https://github.com/ngosang/trackerslist

[MuxZeroNet]

Here is my own implementation of the Pastry DHT algorithm. https://github.com/MuxZeroNet/pastry

Pastry basics:

• Dynamic Distributed Hash Tables.
• Selected DHT algorithms: Chord and Pastry. pp. 23-34.
• Distributed Hash Tables: An Overview. pp. 14-16.

Joining, leaving and repairing the network:

• Pastry, Tapestry and Kademlia. p. 5.
• Structured P2P Networks: Pastry and Tapestry. p. 5

The packets you may need to provide:

1. A packet that implements this interface: has_value, other_nodes = send_dht_request(peer, key)
2. A "gossip" packet that allows nodes to exchange parts of the routing table or leaf set.

Edit: Removed "Toy DHT." Added Pastry, which is not a toy.

[MuxZeroNet]

We got a new problem:

Tamas Kocsis wrote:
The handshake and the encryption has pretty big overhead
so probably it would be more efficient if we would use a separate UDP port

Yes. In practice, latency is what makes DHT protocols slow, which is one of the reasons DHT protocols run over datagram, but rarely over TCP streams.

In Tor, you have to use TCP streams, because this is how Tor works.

I2P SAM provides datagram API. Running DHT over I2P has "encryption overhead" and has observable latency, but it is better than nothing. DHT has been used by I2PSnark and BiglyBT for torrents.

Do you think if the DHT should be part of ZeroNet protocol?
[use the protocol provided by] one of the already existent [libraries]. (maybe libp2p or any other dht lib)

I am more on the "implement a DHT that fits ZeroNet's needs" side, while I am not familiar with the API libp2p or any DHT library provides.

Even if you will be using a DHT library, you should always think about how to write the protocol docs. By using proper means of abstraction, the less you leak implementation details in the specification, the more flexible the protocol is. You really don't want to make future developers who will be implementing ZeroNet in another programming language get stuck on a particular library.

[HulaHoopWhonix]

For a DHT alternative you might want to look at how syncthing implements TCP relays to allow hole punching and communication over Tor. Magic-Wormhole a secure simple file transfer tool is looking to adopt their design to solve scalability.

https://docs.syncthing.net/users/strelaysrv.html

list of servers: http://relays.syncthing.net/

cc/ @adrelanos

[sim590]

@HulaHoopWhonix The thing with relay servers, is that you still have a form of centralization: indexation system of IP addresses. If the indexation system is put down, you cannot query the list of IP addresses. Even worst: censorship, mitm and all centralized scheme attacks are possible. When using DHT, a new routing protocol emerges on top of the IP network which is (really) hard to take down and attack once you have passed a certain point in the number of nodes. After you have resolved the hash query, you can exchange with your peer through DHT put and get operations to go on with hole punching.

However, what you speak of makes me think that onion services may be a good fit for syncthing's use case. Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other. I'm throwing that idea out there, but I don't know a lot about the onion service design... Also, it seems funny to use an anonymizing network to exchange IP address. May be syncthing could even run directly on tor, but I digress from the original subject.

[HulaHoopWhonix]

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network. This is unavoidable. DHT can never be used over Tor and that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other.

You realize that onions don't have IPs?

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

[sim590]

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network.

Yes. However, listing relays on a centralized machine, contrary to using a DHT, doesn't scale.

DHT can never be used over Tor

I didn't say that. However, I assumed the opposite. In fact, onion services use DHT.

that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

As I acknowledged above, bootstrap nodes are needed at first, but you don't need to solely rely on fixed relays over time. Once you're connected to the network, to a DHT for example, you save that list of ip addresses you have contacted, you don't rely on the centralized ip address list and you continuously learn about new nodes through the protocol itself, hence the scaling capability. Opting for centralized relay indexation system implies lacking of scale or reinventing the wheel, like the DHT that is. I'm sure that you know that.

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

Excuse me. I recognize(d) that I digressed from the subject in the last paragraph of my last post. I was not speaking of a solution for ZeroNet.

You realize that onions don't have IPs?

What exacly do you imply? Please, can you avoid making sentences in the form of "You realize that [place some supposed well-known fact here]?" They're not clear and tend to be taken with offense.

I think I understand what's the concern here. You're thinking that I'm speaking of onion services for ZeroNet, but I'm not. I'm speaking about the case of Syncthing. Onion services are resolved with a distributed hash table lookup if that's what you meant by your sentense above. What I meant is that Syncthing devices could use onion services to exchange their ip address like you'd do in a distributed hash table, except that they'd do it through the tor network in addition to looking up the DHT. That may not be adequate indeed after a second thought. A DHT is sufficient, I guess.

I think that we should avoid continuing to speak about Syncthing now as it is not the concern of this issue.

[weimilianqiao]

@HelloZeroNet

New progress made yet.
Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

Please consider retro share. Dark net mode of Retro share let tor or i2p do the peer discovery. It is NOT DHT over TOR.
http://retroshare.readthedocs.io/en/latest/user-guide/settings/#network

Dark net mode is used if neither DHT nor Discovery is wanted. Tor and I2P hidden nodes use this by default as they already have a fixed address to connect to.

[vRobM]

#57 (comment)
Definitely Kademlia.
https://github.com/bmuller/kademlia
@HelloZeroNet

[HelloZeroNet]

https://github.com/bmuller/kademlia: "The nodes communicate using RPC over UDP to communiate, meaning that it is capable of working behind a NAT."
Tor is TCP-only so it's not going to work

[mwarning]

How about DHT only without Tor for now?

[agentofuser]

@gpestana is very knowledgeable about onion-routing + DHTs, maybe he can help. There are some useful links here:

• https://hashmatter.com/
• https://github.com/hashmatter/p3lib
• https://www.gpestana.com/blog/in-pursuit-of-private-dhts/

[gpestana]

hey! ZeroNet is great, good job! 🚀🚀

We at hashmatter are very interested in understanding how different projects are using DHTs and P2P routing protocols and what implication it may have in privacy and metadata leaks - and to try solving privacy vulnerabilities. In the context of ZeroNet, what would be the goal of a DHT? Would it be for service/peer discovery or/and storing and requesting content? How about the latency requirements? In terms of metadata leaks, DHTs may become very problematic and proxying the DHT traffic through Tor may sometimes not be the best solution (depending on the latency requirements, peer identity requirements, etc.) I believe the best way is to define 1) what are the goals for the DHT 2) latency requirements 3) threat model (this might exist somewhere already, and I'm sorry if that's the case and I didn't research enough). There are interesting protocols that can be built on top of the DHT routing protocol (check some here https://github.com/hashmatter/p3lib) that can enhance considerably the privacy of DHT interactions without the need for routing all traffic over Tor.

Happy to keep the discussion and helping if needed 👍

[HulaHoopWhonix]

@gpestana
I'm sorry but nothing you can come up with can match or surpass the anonymity of Tor which has been around forever and has a proven track-record.

Unless your solution is compatible with Tor, it is not relevant for the goal of this ticket.

[erik777]

The red herring seems to be the requirement:

It should work using on Tor (TCP only, maybe UDP on clearnet)

Can we generalize this requirement? What are the requirement for Tor? Anonymity? Privacy? So, how can you use DHT in a way that preserves these basic requirements, while not necessarily using Tor and not impeding the ability to use Tor for the rest of ZeroNet's functions?

So, when I look at OpenDHT, I wonder to what extent this capability helps meet these core requirements:

Public key cryptography layer providing optional data signature and encryption (using GnuTLS)

[EternityForest]

One of OpenDHTs very nice features is a defined REST API for DHT nodes, which act as gateways to look up keys without actually using the DHT yourself. Why not just switch over to using this mode of access when using TOR, but use the DHT as is when using the clear internet?

Trackers can come and go all they want, just switch to another, they all use the same DHT backend.

[nerd36]

One of my blog (zite) visitor complained about there is no peers error but there was 20 peers. The tracker was lying.
I created this bounty. At this point myself cannot pay anything because anonymous gift card is not available in my country
Next step could be block-chain bootstrapping

[nerd36]

it will not work on Tor without proxying it to TCP

You should use Lokinet instead of Tor it is safer because monetized and it support UDP. Even maybe they can sponsor you