Anti DoS: give Tor exits a lower priority than clearnet peers

[mikehearn]

Summary

@gavinandresen reported a Tor-based DoS against one of his nodes a few days ago. He found his node had run out of connection slots and his peers were almost all Tor exits, additionally, his node had stopped following the block chain (likely an inv/getdata exploit but not known for sure).

There are other patches tackling the inv/getdata issue. This patch tackles the other problem, of a jammed node using all its resources to accept connections from a DoS attacker. It lays the foundation for a resource scheduling approach to handling DoS attacks, which runs in parallel to the existing blacklist-on-badness approach.

Approach

A notion of IP groups is introduced. An IP group is a logically related set of subnets with a name and a score. The score of an ungrouped IP is zero. When a node reaches nMaxConnections, the IP of a newly accepted connection is checked against the groups and its priority thus calculated. If the priority is higher than an existing connection, that connection is dropped to make room for the new connection. If it is equal or lower then the existing behaviour of rejecting the new connection is used.

Next, a list of Tor exits is compiled into the binary (a later patch could make the download dynamic, or overridable with a local file). Tor exits get a score of -10 by default, reflecting their tendency to be used for attacks. Thus if a peer is connected to by lots of Tor exits, those connections will be serviced, but clearnet clients will take priority and if need be, kick out one of the Tor connections to make room.

This behaviour is much preferable to the more obvious solution of requiring node operators to manually ban all Tor exits during the type of attack Gavin witnessed.

Future evolution

This patch creates a base that can be evolved in future. For example, the protocol could allow peers to effectively 'buy' priority by proving ownership of coin age. This would allow e.g. an SPV wallet connecting via Tor to take priority over a DoS attack from the clearnet, thus inverting things. However such a protocol is not in this patch.

Smaller improvements might be:

1. Allow loading of IP groups from a file, so people witnessing a jamming attack from other netblocks can easily de-prioritise them.
2. Refresh the Tor exits list from https://check.torproject.org/ from time to time.
3. Queue transactions and signature checks according to peer priority, to fix the "ban every Tor exit by sending bad signatures" attack.
4. Boost the priority for peers that engage in useful behaviour, like relaying valid blocks and transactions (possibly only with an open port?).
5. Slowly lower the priority for connections that seem to be idle. Thus new connections would kick out long term idle connections. This is more a solution to temporary organic load spikes than DoS attacks because of course this rule would be trivially gamed by an attacker who just disconnects and reconnects every so often.

[mikehearn]

@jonasschnelli asked the following questions on the original commit:

What if an attacker uses Tor Bridges?

It makes no difference, traffic that enters Tor via a bridge still leaves via a regular exit.

Is there no way of detecting a misbehaving node that tries to feed jam to my node?

The current notion of misbehaviour is insufficient and we have all known this for a long time. By analogy, is someone who sits at a restaurant table for 5 hours "misbehaving" just because they're taking up a table some other guest could use? No, not really. They're just loitering. If the restaurant gets full the owner may ask them to make space for another guest, but that would not imply misbehaviour on their part.

The notion of priority this patch introduces fixes this, by allowing peers to be distinguished based on how important they are to the node (or perhaps to Bitcoin as a whole), with a score that only matters when resources start to run low. If resources are sufficient to serve everyone then peer priority does not make any difference.

If a node calls getblock of the same block(s) repeatedly, or does other irrational things, this could also reduce it's score.

Yes, good suggestion. I think priority can eventually subsume the existing misbehaviour system almost entirely. The problem with misbehaviour being, it lets jerks or devs writing buggy code ban shared IPs like carrier NATs, universities, VPNs, Tor exits etc. This has been pointed out in an academic paper and is one of the problems that make Tor-by-default hard to enable for bitcoinj clients.

What if we would set limits for block relying for the whole full node operation, maybe also per subnet and or also per IP.

I didn't understand this question,s orry.

I just try to find a way for auto banning nodes without a net QoS with a centralized fixed IP list because i think this would open the cat and mouse game.

Yes, a sufficiently motivated attacker will just find other sets of IPs to use. However this might involve not being so anonymous or having to more actively break the law by e.g. renting a botnet. Tor is convenient and easy and using it is legal, even if DoS attacks generally aren't.

So it's a start, and it addresses an actual observed DoS that someone clearly has the ability to mount right now, even if more work is required to handle every case. Giant oaks tiny acorns, etc.

[mikehearn]

I'm not sure why Travis is unhappy. The header is included in the patch and it builds fine here.

[jonasschnelli]

@mikehearn: you need to add ipgroups.h to src/Makefile.am (around L101).

[petertodd]

You know, I gotta wonder if going to all this engineering effort trying to blacklist Tor really is worthwhile, given we're still just talking about 125 or something connection slots - getting that many non-Tor IP addresses really isn't all that hard. Just look at how F2Pool was subject to a DoS attack recently that actually knocked a few % off their hashing power - script kiddies routinely play around with botnets with way more than 125 IP addresses. (though this attack would be a good way to discredit Tor! lots of potential suspects there)

Equally I can't see us extending these blacklists much beyond a few hundred nodes without making them highly centralized way; to be effective they need to be updated in real-time like the anti-spam blacklists, and that implies a lack of oversight, which for a technology that needs information to flow freely to be secure probably isn't a good tradeoff to make. It's easy to see how a bad blacklist update that isn't caught could cause serious issues for the network if it's widely deployed. Equally effectively saying to people "you're nodes will be unreliable if you run them over Tor" has big problems given the need for business and miners to run anonymized nodes to resist targeted attacks. (given the lack of reports from others, I have to wonder if Gavin was experiencing a targeted attack?)

However! That's not so say we can't have the best of both worlds! Because information propagation security is additive, if we have nodes on the network using both blacklists to resist attack, and more decentralized schemes to resist attack, the security benefit is additive. Some attacks will affect the decentralized schemes more, other attacks will affect the centralized blacklist approach more. So long as an attack can't stall both simultaneously the network will continue to function and maintain consensus. Equally we really could use a second implementation of the P2P logic for the sake of diversity against bugs and exploits, even if it starts off based on Bitcoin Core and gradually moves away from that codebase with design changes.

In short, I'd strongly encourage you to adopt these changes in Bitcoin XT. It'd be a big benefit to the community, and would allow the two approaches to go in directions that best optimize against attack, separately and thus together, without having to deal with divisive politics getting in the way.

[gmaxwell]

The system already has a IP address group (called netgroups). This hardcoded blacklist of tor nodes is problematic in a couple respects-- it's not sensitive to the host actually being able to exit to the user in particular or even Bitcoin at all-- so it effectively punishes people for running tor exits, even ones that can't reach Bitcoin-- and of course the list will rot.

The generation procedure gives the operator of a website free reign to add arbitrary hosts to the list as well.

The claim of tor being convenient isn't quite correct; it takes some fair technical sophistication to figure out how to spread your traffic across multiple exits (as tor doesn't do that by default)-- and in my experience mitigating attacks on Wikimedia attackers used tor as a secondary recourse over other proxies. On IRC the Bitcoin channels have been attacked by thousands of hosts at once with no tor use at all. Also keep in mind that outbound connections which we're normally counting on for partition resistance; so if you're seeing partitioning something very concerning is going wrong that this change will not fix.

So I suspect the benefit is pretty minimal for the cost of carrying around an always-inaccurate-constantly-bitrotting blacklist.... doubly so I find it weird to do this to specifically target our recommended privacy service while turning a blind eye to the many other known network surveillance and sybil attack systems out there which don't use tor. (E.g. prior issues requesting blocking some of these services have been closed with no action, even though they'd be easily blockable).

[gmaxwell]

If someone does want to take PT's advice and do this in some implementation; at the very minimum it should download the tor directory and verify its signatures from the tor directory authorities; and filter hosts to actually test if they can exit to the port/host in question. A nice way to handle fetching the directory securely would be gain the feature only when you're running tor alongside the node, and just let tor fetch it for you (the directory it writes is just plain ascii and easily parsed, IIRC); this would avoid having a static phone-home. Similarly, something less aggressive than instantly kicking all "low priority" peers when it fills up should really be used-- else the priority part of this is mostly pretext.

I still think doing that is inadvisable relative to an actual sensible general solution, but at least it wouldn't have quite as severe a level of additional damage.

[pstratem]

The issues around connection slots as a limited resource has been known since it was first implemented in 2011 (5a3e82f).

@jgarzik has done some great work in designing a UDP protocol for p2p messaging ( https://bitcointalk.org/index.php?topic=156769.0 )

This is not a problem that calls for a blacklist.

Oh and your codes wrong.

[pstratem]

This will disconnect all connections from tor exit nodes.

[pstratem]

I'm working on an alternative proposal.

[jwilkins]

How much effort do you think any judicial agency is going to put into chasing someone DOSing a node through even a basic VPS that they paid for using their own name. If you complain, the best you can realistically hope for is a quick response from the hosting provider. There is essentially no recourse beyond that unless you're Visa and someone feels like making an example of Wikileaks supporters. This is just security theater.

[btcdrak]

A hard coded list? Really?

[laanwj]

I deeply dislike the blacklist idea.

• This creates hardcoded discrimination on IP addresses, singled out based on anecdotal attacks. What is next, a proposal to make all Chinese IPs second class, because many DoSes originate there? Determined attackers can hire a botnet to get thousands of IPs wherever they want. This only hurts legitimate users - Tor isn't even very suitable for DoS because the capacity is inherently bottlenecked by the exit nodes.
• This makes it less reliable to run a node behind Tor, which is essential for privacy. We have put much work into making the software usable behind Tor, and this would undo some of that.
• Fighting DoS in a distributed network should not rely on hardcoded or centralized blacklists. It'd be ripe for abuse to have a central group decide on that. There is need for something more neutral, in the decentralized spirit of P2P.

I'm not opposed to have an option to regard certain networks configured by the node operator as higher-abuse-risk e.g. as an extension of the ban system (note that as of #6310, bans hare persistent). A "halfban", so to say. If that is useful - I suppose in most cases what one wants is simply a full ban. But in no case a list will be hardcoded.

Looking forward to @pstratem's proposal.

[jwilkins]

If you're having trouble, here's a better way to deal with blocking tor exit nodes

ipset -N tor_exit_nodes iphash
for IP in `curl https://check.torproject.org/cgi-bin/TorBulkExitList.py\?ip=$YOUR_IP\&port=8333 | grep -v \# `; do 
 ipset -q -A tor_exit_nodes $IP
done
iptables -A INPUT -m set --match-set tor_exit_nodes src -j DROP

[mikehearn]

Yes, there should be a break statement. Thanks, I'll add it.

Of course, a future patch can add support for smarter download mechanisms and smarter priority calculations. As I said in the patch description, in fact. It would require more code because Core does not have any way to download files over HTTPS. But there isn't much point in me or anyone else implementing such a thing until the base concept of peer priority is accepted, and a default rule that uses it is pretty much required for such infrastructure to be useful. Given that Tor has been observed taking part in an actual jamming attack, such a rule is as good a place to start as any.

[davout]

If bitcoind is vulnerable to DDoS, bitcoind should be fixed.

cat hearn > /dev/null

[coinx-ltc]

@davout You cann't prevent an attacker from blocking all connections of a node with a simple fix in bitcoind.
Since Tor makes it easier to launch such an attack this patch is a step in the right direction.

In case of such an attack non-tor connections should be prioritized since they are more likely to be nodes/miners/merchants. Therefore more important to the network than end user.

[gits7r]

Don't the netgroups in bitcoin core have a group for 'onions' (Tor hidden service peers). When a genuine client (one not trying to DoS other nodes) uses bitcoind over Tor, bitcoind will connect to more onion full nodes as opposite to clearnet IP nodes. This should prevent the attack, if not at least limit it. One of my full nodes (dual stack onion + clearnet) currently has 96 connections out of which 38 with onion peers.

If this is the current behavior, it would make sense to give Tor exit IP addresses a lower priority than other clearnet IP addresses - but I doubt this will stop a DoS attempt, as @petertodd says there are script kiddies out there with thousands of IP addresses, and getting 125 unique non Tor IP addresses is pretty trivial for anyone. It will require more code and additional effort for bitcoind to fetch regularly the dynamically updated Tor exist list and process it, etc. - maybe it's not worth it.

[mikehearn]

I used to work with the anti-DoS system at Google, about 10 years ago I used it quite extensively. It was very sophisticated even back then ..... but it started out as a simple text file of IPs that were bad, being scpd to servers.

The notion that "good is the enemy of the perfect" isn't one based in engineering, it's an academic viewpoint. One day bitcoind might have a sophisticated, dynamic scoring system that rapidly adapts to complex DoS attacks launched by professional hackers without any manual effort required. But it's gotta start in the same place as all the other DoS systems do.

[paveljanik]

So lets start with a script in contrib doing something like @jwilkins said.

[gits7r]

@mikehearn if the current behavior is not as I stated above, might be better to make bitcoind check if it can reach onion peers. Assuming it can reach onion peers if it has a proxy set is not reliable since Tor can be used transparently very easy. The rule is simple, bitcoind should try to connect and if foo.onion is reachable, prefer onion peers vs. clearnet peers.

This will not affect the clearnet nodes or the dual stack (onion + clearnet reachable nodes). Bitcoind can be easily configured with onlynet=tor ; onion=127.0.0.1:9150 which means one can choose to run a node only behind Tor or Tor and clearnet also.

I don't see why when used with Tor bitcoind would use normal circuits and connect to clearnet peers, when it can prefer rendezvous circuits and connect to onion nodes. This will also decrease the load on the Tor network exit capacity, since Tor exits are scarce, and in a rendezvous circuit to an onion peer Tor will use only non-exit relays. We should get more people running full nodes accessible via Tor hidden services, it's very easy to do so: externalip=foo.onion ; bind=127.0.0.1:8333 ; listen=1 + 2 lines in the torrc file.

[mikehearn]

@paveljanik That script is Linux specific, wouldn't be shipped with the binary and (the biggest problem) requires manual communication with and intervention from every node operator. Normally when designing DoS defences you want to automate as much as possible. Most node operators won't know there's a DoS happening and find the script, etc.

@gits7r Making bitcoind prefer to connect to hidden service nodes is a different change, it's not related to this one. I guess the concern is that it's then easier to sybil attack outbound connections as well as inbound (there is no real cost to creating lots of .onion addresses that are owned by the same person). However it should be discussed elsewhere.

[laanwj]

@mikehearn A while ago you said you wanted me to behave more like a dictator.
I still refuse to do this with regard to the consensus rules, but I'm fine with doing it for technical changes.

So: I reject an anti-Tor blacklist in Bitcoin Core.

Go ahead and merge this into your own fork, but the discussion here is done. Every pull you touch turns into a cesspool, a big controversy that detracts from getting day-to-day work done. You are behaving in a way that is toxic to this project. Instead of considered step-by-step development and reasoned discussion, like all other people here, you throw something over the wall and start a forceful argument on how you're right and every alternative suggestion is a mistake that will lead to doom and gloom. This is draining our energy. Stop it.

[mikehearn]

Well, I do appreciate the clear and quick decision making at least :) OK then, go ahead and stick with your current approach. Just remember that relying entirely on 'misbehaviour' lets anyone switch off Tor for everyone, globally: that's the opposite of what you want.

With respect to my patches being "cesspools", I'm afraid I feel everything you just wrote applies to you guys as well. This patch is a considered step by step approach: it's the first step, with many other steps described in pull request message. But if you reject the first step because you don't like the facts of a real attack that actually happened (Tor was abused), you'll never make it to the second step, or third, or fourth. You'll stick around waiting for some perfect "non controversial" solution forever.

Don't let "good is the enemy of the perfect" stay your modus operandi. That may work in academia. It does not work for a software engineering project.

[petertodd]

@gits7r

This will also decrease the load on the Tor network exit capacity, since Tor exits are scarce, and in a rendezvous circuit to an onion peer Tor will use only non-exit relays.

Actually I brought up that point at a Tor dev's workshop last year and apparently there has been so much new support that - at least at that point - the situation had reversed to relay capacity being the bottleneck, with plenty of exit capacity available. Still, Jacob Applebaum and others I talked to were highly supportive of making bitcoind support Tor by default with a onion peer strategy - they could definitely make good use of the extra cover traffic and it'd be a gain for Bitcoin as well in terms of censorship and attack resistance.

@mikehearn re: that attack, it'd be helpful if @gavinandresen did a proper writeup on it with details, log files, etc. As @gmaxwell suggested this would make a good issue for the issue tracker as a first step so we can monitor and evaluate the (apparent) problem. At this point the evidence of attack hasn't been peer reviewed after all - could be an isolated incident, or even a misunderstanding.

[petertodd]

@laanwj +1

You know, @mikehearn simplifying the issue to 'Don't let "good is the enemy of the perfect" stay your modus operandi.' is probably a big part of the problem - changes to Bitcoin aren't things that can be evaluated in a simple, linear, better or worse framework, but rather they have multi-dimensional effects with simultaneous good and bad outcomes. Often the outcomes are significantly time delayed and/or synergistic.

@mikehearn seems to have a particular talent for making proposals that have some good outcomes, but a lot of (often unstated) negative outcomes for other goals. That's a sure-fire recipe for drama and is something that's easily avoided. I myself have made some of the most controversial pull-reqs, but at least things like #6352 start off with explicit compromises and cost/benefit analysis. They're also a relatively small % of my pull-reqs as a whole - the only thing @mikehearn seems to do with his is generate controversy.

[kristovatlas]

@mikehearn sorry to beat a dead horse/closed PR, but would this not make it fairly trivial to block access to all full nodes for Tor users using a very small (125 strong?) botnet? Still wondering of the merits of this now that it has been merged into XT.

[gmaxwell]

@kristovatlas It was my belief that it could be done with a single host-- including a new tor exit that just wasn't in the list at fetch time, in fact. But I didn't try it and might be missing some detail.

[mikehearn]

Yes, you can. However, Tor is significantly easier to use/get access to than most botnets.

Please read the roadmap at the top. The next step is to handle multiple connections from single IPs better, and to allow user-configurable priority lists. Then a quick fix would be to drop your little botnet IPs into a file and give it lower priority than Tor. After that, Tor connections would evict the botnet.

Then after that, there needs to be ways for a node to figure out priorities automatically, for instance by defining a utility function that doesn't rate all mobile connections as low utility. This is very hard, especially as the code would be open source, but is obviously the gold standard to aim for.

This is all a lot of work. Contributions of some additional logic to get us further along the path would be appreciated.

[dcousens]

Then a quick fix would be to drop your little botnet IPs into a file and give it lower priority than Tor.

How is anyone to keep up with manually putting IPs into a file?

Never mind, you mention automating this in the next paragraph.

[mikehearn]

Yes. I think people are forgetting that DoS attackers don't politely wait around for you to come up with the perfect system. Sometimes you need a quick kludge if the automated systems you have aren't working.

The problem of handling DoS attacks in an open network like Bitcoin bears a strong resemblance to spam filtering in the email network. Random IPs are sending you messages and you don't have time to handle all of them. So you have to figure out which are important, which are probably junk but might not be, and which can definitely be blocked without damaging levels of false positives.

Needless to say, if the maintainers of popular email servers had said "Eww, you want to have actual lists of bad IPs, how centralizing" then they'd have never got anywhere ....

[dcousens]

@mikehearn that is all fine. Hacks are a reality in a lot of software. I think most people understand that.
But remember this is the core client, it is literally the software run by the entire network in one way or another.

This isn't some hack that will only occur for some guys email server, or a small business building out an new email platform. This is the protocol and back bone of the bitcoin network.
I'd say if high-bandwidth nodes are having DoS issues, until a well-thought, formal solution is found, they can employ their own tactics to do exactly as you proposed.

[find] a quick kludge if the automated systems you have aren't working.

I don't think we need to formalize that with manually adulterated lists of IPs.
You can use iptables for that.

[kristovatlas]

Yes, you can. However, Tor is significantly easier to use/get access to than most botnets.

Granted, but the pool of people who might want to grief/DoS Tor users of bitcoin who also can get access to 1-125 IP addresses is pretty large. It would only take one, as a list of Bitcoin node IPs is pretty easy to discover.

OTOH it looks like the way Bitcoin Core works right now is not better, if it's super easy for people to use Tor to fill up all of the connection slots. @mikehearn seems right that, as-is, you don't have many options other than to outright ban all Tor exit nodes in this scenario.

Both of these seem like crappy options. In both, DoS is easy, and obstructing access to nodes via Tor exits is easy. :(

In principle, as long as a kludge does not make things worse and is not a foot-in-the-door tactic to undermine privacy in the long term, I see no issue with it. After the redlisting stuff, I can understand why people are concerned about a slippery slope of some kind.

I would like to see some solid proposals about how this DoS attack can be dealt with in a meaningful way.

For example, the protocol could allow peers to effectively 'buy' priority by proving ownership of coin age.

Authenticating the right to connect based on the ownership of coins without compromising privacy seems tricky. Probably some zero-knowledge proof would be required? IMHO any solution that tries to trade privacy for DoS resistance presents an unappealing trade-off. I expect privacy to become increasingly important for Bitcoin users, not less.

Boost the priority for peers that engage in useful behaviour, like relaying valid blocks and transactions (possibly only with an open port?).

This sounds interesting. I like the idea of rewarding positive behavior, though so far it seems unclear how you would separate the good actors from the bad ones.

Because information propagation security is additive, if we have nodes on the network using both blacklists to resist attack, and more decentralized schemes to resist attack, the security benefit is additive. Some attacks will affect the decentralized schemes more, other attacks will affect the centralized blacklist approach more.

In short, I'd strongly encourage you to adopt these changes in Bitcoin XT.

Agree with @petertodd on this one. I actually wonder if it would be helpful for some clients to have non-deterministic behavior in this regard; I vaguely recall someone pitching "chaos monkey" behavior in another thread on this topic.

[kristovatlas]

With respect to better DoS resistance, what would it take to increase the # of connection slots a would-be attacker would consume? When you scale up the # of connection slots, which consumed resources tend to be the most expensive? Can we make them less expensive?

[kristovatlas]

Another way to look at DoS is in terms of economics. I think the fundamental problem with DoS is asymmetry in costs. Right now, there aren't many connection slots to consume, and it's very cheap (Tor or not -- this seems nearly irrelevant to me) to consume them. Even if we were to increase the number of slots, I bet a DoS attacker can find some operation that will require a node to consume a bunch of resources in order to service the operation.

My preferred solution -- admittedly a huge engineering effort -- is to have a version of the Bitcoin client that does almost nothing for other nodes unless they pay for it. Processing payment signals would have to be done in such a way that the act of processing payment would not be a viable DoS route itself. Then, aspiring DoS attacks become expensive and probably infeasible in the long term; the more resources you try to consume from your victim, the more you are expected to pay.

Coincidentally, I think this would be helpful in addressing some of the issues related to block size limits as well, but that's a whole other topic.

[mikehearn]

I think the right place to discuss this stuff would be the Bitcoin XT list.