New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DHT support #57

Open
HelloZeroNet opened this Issue Mar 16, 2015 · 29 comments

Comments

Projects
None yet
@HelloZeroNet
Owner

HelloZeroNet commented Mar 16, 2015

Similar to torrent's DHT solution, required features:

  • Find peer to specified site file (to allow sites to add optionally download files)
  • Ipv6 and tor address support for peers
  • It should work using on Tor (TCP only, maybe UDP on clearnet)
  • No other library dependency if possible
  • Store peer ips to allow work without bootstrap server

Maybe its requred to separate the DHT from Peer Exchange

Please comment if you have other ideas/suggestions.

@defnax

This comment has been minimized.

defnax commented Nov 9, 2015

why not use libtorrent?, twister uses libtorrent for DHT
http://twister.net.co/?page_id=54

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Nov 9, 2015

zeronet protocol is different from torrent, so libtorrent will not work. Also bittorrent DHT is UDP based, so it will not work on Tor without proxying it to TCP which I don't want. And it does not support storing Tor hidden service addresses, so you would not be able to create sites on Tor.

@TheNain38

This comment has been minimized.

Contributor

TheNain38 commented Nov 9, 2015

@HelloZeroNet You could add to your first comment:

  • Be able to store I2P and Tor hidden service addresses
  • Don't use UDP to be compatible with Tor
@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Nov 9, 2015

Its already there:

  • Ipv6 and tor address support for peers
  • It should work using on Tor (TCP only, maybe UDP on clearnet)
@TheNain38

This comment has been minimized.

Contributor

TheNain38 commented Nov 9, 2015

@HelloZeroNet Ho, didn't see that, sorry

@alxbob

This comment has been minimized.

@alxbob

This comment has been minimized.

alxbob commented Nov 20, 2015

Full tor support and hidden services can work inside zeronet by creating a dht (that would be a tor hidden service it self) that holds the .onion addresses and load balances them. In clearnet it could work as a common dht. So its users connecting to zeronet through tor will have a hidden service created for its site he/she visits. And let tor hdirs take care the rest!

https://pypi.python.org/pypi/OnionBalance

https://stem.torproject.org/tutorials/over_the_river.html

@pldubouilh

This comment has been minimized.

pldubouilh commented Apr 18, 2016

Interesting issue. Zeronet would drastically benefit from a DHT, as trackers tend to come and go with time...

My bet is that it would be best to bootstrap and use Bittorrent's DHT (Mainline DHT - MLDHT), as although it has a few problems, a lot of effort is put into it to keep it reliable and stable. A simple approach could be done using announce_peer and get_peers. Some more fancy stuff could be done using the post-signed-mutable-content thing (see bep_0044), but that might not be interesting for Zeronet.

For instance I (new user) query the MLDHT : get_peers(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D)) and retrieve a list of the peers on it - simple. The peers would have to first announced themselves using (roughly) announce_peer(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D), my_ip_and_port)

About the dependency, I think it's inevitable, at least for some things like bencode and the MLDHT, although there seems to exists relatively small python libraries. I'll have a look at this.

@stillwarter

This comment has been minimized.

stillwarter commented Aug 15, 2016

Can the DHT solves the problem that how a new client first join the zeronet when the tracker server is down?

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Aug 15, 2016

@stillwarter DHT still need a bootstrap server, so it does not help on new "virgin" clients.

@obv-mikhail

This comment has been minimized.

obv-mikhail commented Aug 29, 2016

@alugarius

This comment has been minimized.

alugarius commented Sep 10, 2016

Custom bootstrap clients (storing all of them is nice too) , some proxys and an option to save an anonymous list of users to use as emergency bootstrap, it can be stored(exported and imported)! The "unvirginizer"
( IPFS tactics)

@Bachstelze

This comment has been minimized.

Bachstelze commented Nov 19, 2016

@sim590

This comment has been minimized.

sim590 commented Apr 21, 2017

Hi,

I propose to you to consider using OpenDHT. It has the following features:

  • C++11 (maybe C++14 somewhat soon);
  • Python bindings;
  • IPv4 and IPv6 support;
  • Nice crypto layer for handling encrypted blobs and signatures;
  • Transition from 160-bits SHA1 hashes to 256-bits hashes is in progress;
  • TCP support is work in progress (@aberaud may comment on the status);
  • Distributed indexing: it literally permits to do searches on complex queries rather than exact key lookups. This is based on PHT data structure although we have added substantial changes to support Fully distributed indexing which are documented into an article to be presented at UNet2017. This is more than relevant for this project. You can compare this to providing Google search features on ZeroNet;
  • It is actively used by the software called Ring which is driven to deliver distributed and secure p2p communications to its users. More people use the same DHT, more the apps based on it will behave well.

I personally would like more projects to join like Syncthing.

@slothbag

This comment has been minimized.

slothbag commented May 1, 2017

IPv6 for CJDNS would be fantastic.

@MuxZeroNet

This comment has been minimized.

Contributor

MuxZeroNet commented Jun 20, 2017

I talked with Shortcutme with this issue. Here are the questions he is trying to find an answer to.

Some months ago I did some experiments and checked out some [DHT] libraries, but there are still lots of questions left.

  • Can it reliably work over Tor (or over TCP at all)?
  • DHT routing in ZeroNet must support .onion and .i2p addresses.
  • Should we use an existing DHT implementation (eg. Mainline, OpenDHT) or create a new one?
  • Is there any alternative to DHT that worth checking out?

Here are my questions:

  • Nodes in the network hosts their own sets of sites and optional files, based on user interests. We still want to choose what to store and what not to store. There is no strictly defined peer responsibilities. Should me adjust current DHT implementations to reflect the current circumstances of the ZeroNet network?
@jpldcarvalho

This comment has been minimized.

jpldcarvalho commented Jul 28, 2017

Which DHT protocol do you use ? @HelloZeroNet

@alugarius

This comment has been minimized.

alugarius commented Aug 3, 2017

Does ZNet made any progress in this issue?

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Aug 4, 2017

New progress made yet.
Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

@skwerlman

This comment has been minimized.

skwerlman commented Aug 7, 2017

this might be of interest. It's not reimplementing DHT using TCP, but instead it just carries the UDP traffic through a TCP tunnel (with the express intent of letting it be carried by Tor)

@mkg20001 mkg20001 referenced this issue Aug 26, 2017

Closed

ZeroNet DHT #1098

@gitbugged

This comment has been minimized.

gitbugged commented Sep 11, 2017

Very interested in getting DHT, but in the interim with trackers getting blocked and dropped, maybe you could add more trackers?
https://github.com/ngosang/trackerslist

@MuxZeroNet

This comment has been minimized.

Contributor

MuxZeroNet commented Sep 30, 2017

Here is my own implementation of the Pastry DHT algorithm. https://github.com/MuxZeroNet/pastry

Pastry basics:

Joining, leaving and repairing the network:

The packets you may need to provide:

  1. A packet that implements this interface: has_value, other_nodes = send_dht_request(peer, key)
  2. A "gossip" packet that allows nodes to exchange parts of the routing table or leaf set.

Edit: Removed "Toy DHT." Added Pastry, which is not a toy.

@MuxZeroNet

This comment has been minimized.

Contributor

MuxZeroNet commented Jan 11, 2018

We got a new problem:

Tamas Kocsis wrote:
The handshake and the encryption has pretty big overhead
so probably it would be more efficient if we would use a separate UDP port

Yes. In practice, latency is what makes DHT protocols slow, which is one of the reasons DHT protocols run over datagram, but rarely over TCP streams.

In Tor, you have to use TCP streams, because this is how Tor works.

I2P SAM provides datagram API. Running DHT over I2P has "encryption overhead" and has observable latency, but it is better than nothing. DHT has been used by I2PSnark and BiglyBT for torrents.

Do you think if the DHT should be part of ZeroNet protocol?
[use the protocol provided by] one of the already existent [libraries]. (maybe libp2p or any other dht lib)

I am more on the "implement a DHT that fits ZeroNet's needs" side, while I am not familiar with the API libp2p or any DHT library provides.

Even if you will be using a DHT library, you should always think about how to write the protocol docs. By using proper means of abstraction, the less you leak implementation details in the specification, the more flexible the protocol is. You really don't want to make future developers who will be implementing ZeroNet in another programming language get stuck on a particular library.

@HulaHoopWhonix

This comment has been minimized.

HulaHoopWhonix commented Feb 27, 2018

For a DHT alternative you might want to look at how syncthing implements TCP relays to allow hole punching and communication over Tor. Magic-Wormhole a secure simple file transfer tool is looking to adopt their design to solve scalability.

https://docs.syncthing.net/users/strelaysrv.html

list of servers: http://relays.syncthing.net/

cc/ @adrelanos

@sim590

This comment has been minimized.

sim590 commented Feb 27, 2018

@HulaHoopWhonix The thing with relay servers, is that you still have a form of centralization: indexation system of IP addresses. If the indexation system is put down, you cannot query the list of IP addresses. Even worst: censorship, mitm and all centralized scheme attacks are possible. When using DHT, a new routing protocol emerges on top of the IP network which is (really) hard to take down and attack once you have passed a certain point in the number of nodes. After you have resolved the hash query, you can exchange with your peer through DHT put and get operations to go on with hole punching.

However, what you speak of makes me think that onion services may be a good fit for syncthing's use case. Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other. I'm throwing that idea out there, but I don't know a lot about the onion service design... Also, it seems funny to use an anonymizing network to exchange IP address. May be syncthing could even run directly on tor, but I digress from the original subject.

@HulaHoopWhonix

This comment has been minimized.

HulaHoopWhonix commented Feb 28, 2018

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network. This is unavoidable. DHT can never be used over Tor and that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other.

You realize that onions don't have IPs?

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

@sim590

This comment has been minimized.

sim590 commented Feb 28, 2018

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network.

Yes. However, listing relays on a centralized machine, contrary to using a DHT, doesn't scale.

DHT can never be used over Tor

I didn't say that. However, I assumed the opposite. In fact, onion services use DHT.

that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

As I acknowledged above, bootstrap nodes are needed at first, but you don't need to solely rely on fixed relays over time. Once you're connected to the network, to a DHT for example, you save that list of ip addresses you have contacted, you don't rely on the centralized ip address list and you continuously learn about new nodes through the protocol itself, hence the scaling capability. Opting for centralized relay indexation system implies lacking of scale or reinventing the wheel, like the DHT that is. I'm sure that you know that.

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

Excuse me. I recognize(d) that I digressed from the subject in the last paragraph of my last post. I was not speaking of a solution for ZeroNet.

You realize that onions don't have IPs?

What exacly do you imply? Please, can you avoid making sentences in the form of "You realize that [place some supposed well-known fact here]?" They're not clear and tend to be taken with offense.

I think I understand what's the concern here. You're thinking that I'm speaking of onion services for ZeroNet, but I'm not. I'm speaking about the case of Syncthing. Onion services are resolved with a distributed hash table lookup if that's what you meant by your sentense above. What I meant is that Syncthing devices could use onion services to exchange their ip address like you'd do in a distributed hash table, except that they'd do it through the tor network in addition to looking up the DHT. That may not be adequate indeed after a second thought. A DHT is sufficient, I guess.

I think that we should avoid continuing to speak about Syncthing now as it is not the concern of this issue.

@weimilianqiao

This comment has been minimized.

weimilianqiao commented Apr 19, 2018

@HelloZeroNet

New progress made yet.
Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

Please consider retro share. Dark net mode of Retro share let tor or i2p do the peer discovery. It is NOT DHT over TOR.
http://retroshare.readthedocs.io/en/latest/user-guide/settings/#network

Dark net mode is used if neither DHT nor Discovery is wanted. Tor and I2P hidden nodes use this by default as they already have a fixed address to connect to.

@DaniellMesquita DaniellMesquita referenced this issue Sep 10, 2018

Open

Chinese users needs IPv6 and DHT #1589

0 of 2 tasks complete
@vRobM

This comment has been minimized.

vRobM commented Sep 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment