Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DHT support #57

Open
HelloZeroNet opened this issue Mar 16, 2015 · 36 comments
Open

DHT support #57

HelloZeroNet opened this issue Mar 16, 2015 · 36 comments

Comments

@HelloZeroNet
Copy link
Owner

@HelloZeroNet HelloZeroNet commented Mar 16, 2015

Similar to torrent's DHT solution, required features:

  • Find peer to specified site file (to allow sites to add optionally download files)
  • Ipv6 and tor address support for peers
  • It should work using on Tor (TCP only, maybe UDP on clearnet)
  • No other library dependency if possible
  • Store peer ips to allow work without bootstrap server

Maybe its requred to separate the DHT from Peer Exchange

Please comment if you have other ideas/suggestions.

@defnax
Copy link

@defnax defnax commented Nov 9, 2015

why not use libtorrent?, twister uses libtorrent for DHT
http://twister.net.co/?page_id=54

@HelloZeroNet
Copy link
Owner Author

@HelloZeroNet HelloZeroNet commented Nov 9, 2015

zeronet protocol is different from torrent, so libtorrent will not work. Also bittorrent DHT is UDP based, so it will not work on Tor without proxying it to TCP which I don't want. And it does not support storing Tor hidden service addresses, so you would not be able to create sites on Tor.

@TheNain38
Copy link
Contributor

@TheNain38 TheNain38 commented Nov 9, 2015

@HelloZeroNet You could add to your first comment:

  • Be able to store I2P and Tor hidden service addresses
  • Don't use UDP to be compatible with Tor
@HelloZeroNet
Copy link
Owner Author

@HelloZeroNet HelloZeroNet commented Nov 9, 2015

Its already there:

  • Ipv6 and tor address support for peers
  • It should work using on Tor (TCP only, maybe UDP on clearnet)
@TheNain38
Copy link
Contributor

@TheNain38 TheNain38 commented Nov 9, 2015

@HelloZeroNet Ho, didn't see that, sorry

@alxbob
Copy link

@alxbob alxbob commented Nov 20, 2015

Full tor support and hidden services can work inside zeronet by creating a dht (that would be a tor hidden service it self) that holds the .onion addresses and load balances them. In clearnet it could work as a common dht. So its users connecting to zeronet through tor will have a hidden service created for its site he/she visits. And let tor hdirs take care the rest!

https://pypi.python.org/pypi/OnionBalance

https://stem.torproject.org/tutorials/over_the_river.html

@pldubouilh
Copy link

@pldubouilh pldubouilh commented Apr 18, 2016

Interesting issue. Zeronet would drastically benefit from a DHT, as trackers tend to come and go with time...

My bet is that it would be best to bootstrap and use Bittorrent's DHT (Mainline DHT - MLDHT), as although it has a few problems, a lot of effort is put into it to keep it reliable and stable. A simple approach could be done using announce_peer and get_peers. Some more fancy stuff could be done using the post-signed-mutable-content thing (see bep_0044), but that might not be interesting for Zeronet.

For instance I (new user) query the MLDHT : get_peers(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D)) and retrieve a list of the peers on it - simple. The peers would have to first announced themselves using (roughly) announce_peer(sha1(1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D), my_ip_and_port)

About the dependency, I think it's inevitable, at least for some things like bencode and the MLDHT, although there seems to exists relatively small python libraries. I'll have a look at this.

@SeniorPlayer
Copy link

@SeniorPlayer SeniorPlayer commented Aug 15, 2016

Can the DHT solves the problem that how a new client first join the zeronet when the tracker server is down?

@HelloZeroNet
Copy link
Owner Author

@HelloZeroNet HelloZeroNet commented Aug 15, 2016

@stillwarter DHT still need a bootstrap server, so it does not help on new "virgin" clients.

@alugarius
Copy link

@alugarius alugarius commented Sep 10, 2016

Custom bootstrap clients (storing all of them is nice too) , some proxys and an option to save an anonymous list of users to use as emergency bootstrap, it can be stored(exported and imported)! The "unvirginizer"
( IPFS tactics)

@sim590
Copy link

@sim590 sim590 commented Apr 21, 2017

Hi,

I propose to you to consider using OpenDHT. It has the following features:

  • C++11 (maybe C++14 somewhat soon);
  • Python bindings;
  • IPv4 and IPv6 support;
  • Nice crypto layer for handling encrypted blobs and signatures;
  • Transition from 160-bits SHA1 hashes to 256-bits hashes is in progress;
  • TCP support is work in progress (@aberaud may comment on the status);
  • Distributed indexing: it literally permits to do searches on complex queries rather than exact key lookups. This is based on PHT data structure although we have added substantial changes to support Fully distributed indexing which are documented into an article to be presented at UNet2017. This is more than relevant for this project. You can compare this to providing Google search features on ZeroNet;
  • It is actively used by the software called Ring which is driven to deliver distributed and secure p2p communications to its users. More people use the same DHT, more the apps based on it will behave well.

I personally would like more projects to join like Syncthing.

@slothbag
Copy link

@slothbag slothbag commented May 1, 2017

IPv6 for CJDNS would be fantastic.

@MuxZeroNet
Copy link
Contributor

@MuxZeroNet MuxZeroNet commented Jun 20, 2017

I talked with Shortcutme with this issue. Here are the questions he is trying to find an answer to.

Some months ago I did some experiments and checked out some [DHT] libraries, but there are still lots of questions left.

  • Can it reliably work over Tor (or over TCP at all)?
  • DHT routing in ZeroNet must support .onion and .i2p addresses.
  • Should we use an existing DHT implementation (eg. Mainline, OpenDHT) or create a new one?
  • Is there any alternative to DHT that worth checking out?

Here are my questions:

  • Nodes in the network hosts their own sets of sites and optional files, based on user interests. We still want to choose what to store and what not to store. There is no strictly defined peer responsibilities. Should me adjust current DHT implementations to reflect the current circumstances of the ZeroNet network?
@jpldcarvalho
Copy link

@jpldcarvalho jpldcarvalho commented Jul 28, 2017

Which DHT protocol do you use ? @HelloZeroNet

@alugarius
Copy link

@alugarius alugarius commented Aug 3, 2017

Does ZNet made any progress in this issue?

@HelloZeroNet
Copy link
Owner Author

@HelloZeroNet HelloZeroNet commented Aug 4, 2017

New progress made yet.
Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

@skwerlman
Copy link

@skwerlman skwerlman commented Aug 7, 2017

this might be of interest. It's not reimplementing DHT using TCP, but instead it just carries the UDP traffic through a TCP tunnel (with the express intent of letting it be carried by Tor)

@mkg20001 mkg20001 mentioned this issue Aug 26, 2017
@gitbugged
Copy link

@gitbugged gitbugged commented Sep 11, 2017

Very interested in getting DHT, but in the interim with trackers getting blocked and dropped, maybe you could add more trackers?
https://github.com/ngosang/trackerslist

@MuxZeroNet
Copy link
Contributor

@MuxZeroNet MuxZeroNet commented Sep 30, 2017

Here is my own implementation of the Pastry DHT algorithm. https://github.com/MuxZeroNet/pastry

Pastry basics:

Joining, leaving and repairing the network:

The packets you may need to provide:

  1. A packet that implements this interface: has_value, other_nodes = send_dht_request(peer, key)
  2. A "gossip" packet that allows nodes to exchange parts of the routing table or leaf set.

Edit: Removed "Toy DHT." Added Pastry, which is not a toy.

@MuxZeroNet
Copy link
Contributor

@MuxZeroNet MuxZeroNet commented Jan 11, 2018

We got a new problem:

Tamas Kocsis wrote:
The handshake and the encryption has pretty big overhead
so probably it would be more efficient if we would use a separate UDP port

Yes. In practice, latency is what makes DHT protocols slow, which is one of the reasons DHT protocols run over datagram, but rarely over TCP streams.

In Tor, you have to use TCP streams, because this is how Tor works.

I2P SAM provides datagram API. Running DHT over I2P has "encryption overhead" and has observable latency, but it is better than nothing. DHT has been used by I2PSnark and BiglyBT for torrents.

Do you think if the DHT should be part of ZeroNet protocol?
[use the protocol provided by] one of the already existent [libraries]. (maybe libp2p or any other dht lib)

I am more on the "implement a DHT that fits ZeroNet's needs" side, while I am not familiar with the API libp2p or any DHT library provides.

Even if you will be using a DHT library, you should always think about how to write the protocol docs. By using proper means of abstraction, the less you leak implementation details in the specification, the more flexible the protocol is. You really don't want to make future developers who will be implementing ZeroNet in another programming language get stuck on a particular library.

@HulaHoopWhonix
Copy link

@HulaHoopWhonix HulaHoopWhonix commented Feb 27, 2018

For a DHT alternative you might want to look at how syncthing implements TCP relays to allow hole punching and communication over Tor. Magic-Wormhole a secure simple file transfer tool is looking to adopt their design to solve scalability.

https://docs.syncthing.net/users/strelaysrv.html

list of servers: http://relays.syncthing.net/

cc/ @adrelanos

@sim590
Copy link

@sim590 sim590 commented Feb 27, 2018

@HulaHoopWhonix The thing with relay servers, is that you still have a form of centralization: indexation system of IP addresses. If the indexation system is put down, you cannot query the list of IP addresses. Even worst: censorship, mitm and all centralized scheme attacks are possible. When using DHT, a new routing protocol emerges on top of the IP network which is (really) hard to take down and attack once you have passed a certain point in the number of nodes. After you have resolved the hash query, you can exchange with your peer through DHT put and get operations to go on with hole punching.

However, what you speak of makes me think that onion services may be a good fit for syncthing's use case. Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other. I'm throwing that idea out there, but I don't know a lot about the onion service design... Also, it seems funny to use an anonymizing network to exchange IP address. May be syncthing could even run directly on tor, but I digress from the original subject.

@HulaHoopWhonix
Copy link

@HulaHoopWhonix HulaHoopWhonix commented Feb 28, 2018

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network. This is unavoidable. DHT can never be used over Tor and that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

Each device could host an onion service in the tor network when it's online so that it can exchange ip address with each other.

You realize that onions don't have IPs?

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

@sim590
Copy link

@sim590 sim590 commented Feb 28, 2018

Any network, including DHT and P2P ones will rely on bootstrap nodes to help them reach the wider network.

Yes. However, listing relays on a centralized machine, contrary to using a DHT, doesn't scale.

DHT can never be used over Tor

I didn't say that. However, I assumed the opposite. In fact, onion services use DHT.

that's why a commonly emerging solution in the Tox project and Syncthing is to rely on some form of federated relays.

As I acknowledged above, bootstrap nodes are needed at first, but you don't need to solely rely on fixed relays over time. Once you're connected to the network, to a DHT for example, you save that list of ip addresses you have contacted, you don't rely on the centralized ip address list and you continuously learn about new nodes through the protocol itself, hence the scaling capability. Opting for centralized relay indexation system implies lacking of scale or reinventing the wheel, like the DHT that is. I'm sure that you know that.

While Tor support in the form of ephemeral services using Txtorcon are a neat addition it still doesn't solve the problem of a Tor user interoperating with the outside zeronet network that runs on on the clearnet. Unless Tor use is enforced by default, you will end up with a fragmented network.

Excuse me. I recognize(d) that I digressed from the subject in the last paragraph of my last post. I was not speaking of a solution for ZeroNet.

You realize that onions don't have IPs?

What exacly do you imply? Please, can you avoid making sentences in the form of "You realize that [place some supposed well-known fact here]?" They're not clear and tend to be taken with offense.

I think I understand what's the concern here. You're thinking that I'm speaking of onion services for ZeroNet, but I'm not. I'm speaking about the case of Syncthing. Onion services are resolved with a distributed hash table lookup if that's what you meant by your sentense above. What I meant is that Syncthing devices could use onion services to exchange their ip address like you'd do in a distributed hash table, except that they'd do it through the tor network in addition to looking up the DHT. That may not be adequate indeed after a second thought. A DHT is sufficient, I guess.

I think that we should avoid continuing to speak about Syncthing now as it is not the concern of this issue.

@weimilianqiao
Copy link

@weimilianqiao weimilianqiao commented Apr 19, 2018

@HelloZeroNet

New progress made yet.
Does anyone know already working P2P application that uses DHT over Tor? (preferably with >1000 nodes)

Please consider retro share. Dark net mode of Retro share let tor or i2p do the peer discovery. It is NOT DHT over TOR.
http://retroshare.readthedocs.io/en/latest/user-guide/settings/#network

Dark net mode is used if neither DHT nor Discovery is wanted. Tor and I2P hidden nodes use this by default as they already have a fixed address to connect to.

@DaniellMesquita DaniellMesquita mentioned this issue Sep 10, 2018
0 of 2 tasks complete
@vRobM
Copy link

@vRobM vRobM commented Sep 11, 2018

@HelloZeroNet
Copy link
Owner Author

@HelloZeroNet HelloZeroNet commented Apr 6, 2019

https://github.com/bmuller/kademlia: "The nodes communicate using RPC over UDP to communiate, meaning that it is capable of working behind a NAT."
Tor is TCP-only so it's not going to work

@mwarning
Copy link

@mwarning mwarning commented May 16, 2019

How about DHT only without Tor for now?

@agentofuser
Copy link

@agentofuser agentofuser commented Jun 18, 2019

@gpestana is very knowledgeable about onion-routing + DHTs, maybe he can help. There are some useful links here:

@gpestana
Copy link

@gpestana gpestana commented Jun 18, 2019

hey! ZeroNet is great, good job! 🚀🚀

We at hashmatter are very interested in understanding how different projects are using DHTs and P2P routing protocols and what implication it may have in privacy and metadata leaks - and to try solving privacy vulnerabilities. In the context of ZeroNet, what would be the goal of a DHT? Would it be for service/peer discovery or/and storing and requesting content? How about the latency requirements? In terms of metadata leaks, DHTs may become very problematic and proxying the DHT traffic through Tor may sometimes not be the best solution (depending on the latency requirements, peer identity requirements, etc.) I believe the best way is to define 1) what are the goals for the DHT 2) latency requirements 3) threat model (this might exist somewhere already, and I'm sorry if that's the case and I didn't research enough). There are interesting protocols that can be built on top of the DHT routing protocol (check some here https://github.com/hashmatter/p3lib) that can enhance considerably the privacy of DHT interactions without the need for routing all traffic over Tor.

Happy to keep the discussion and helping if needed 👍

@HulaHoopWhonix
Copy link

@HulaHoopWhonix HulaHoopWhonix commented Jun 19, 2019

@gpestana
I'm sorry but nothing you can come up with can match or surpass the anonymity of Tor which has been around forever and has a proven track-record.

Unless your solution is compatible with Tor, it is not relevant for the goal of this ticket.

@filips123 filips123 mentioned this issue Jul 2, 2019
1 of 16 tasks complete
@erik777
Copy link

@erik777 erik777 commented Aug 12, 2019

The red herring seems to be the requirement:

It should work using on Tor (TCP only, maybe UDP on clearnet)

Can we generalize this requirement? What are the requirement for Tor? Anonymity? Privacy? So, how can you use DHT in a way that preserves these basic requirements, while not necessarily using Tor and not impeding the ability to use Tor for the rest of ZeroNet's functions?

So, when I look at OpenDHT, I wonder to what extent this capability helps meet these core requirements:

Public key cryptography layer providing optional data signature and encryption (using GnuTLS)

@EternityForest
Copy link

@EternityForest EternityForest commented Dec 18, 2020

One of OpenDHTs very nice features is a defined REST API for DHT nodes, which act as gateways to look up keys without actually using the DHT yourself. Why not just switch over to using this mode of access when using TOR, but use the DHT as is when using the clear internet?

Trackers can come and go all they want, just switch to another, they all use the same DHT backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet