Denial of Service

[zebambam]

Scaling discussions don’t make sense unless the ultimate goal is to reduce the likelihood that users will not be able to use Zcash because of a lack of service.

There are lots of ways to fill up our service currently.

When we spot them, we should link them to this epic.

• A small number of shielded transaction of massive size.
• A small number of or many transactions with many empty outputs.
• Many small transactions filling the mempool currently crash the nodes that receive them. It’s possible to knock nodes offline. Transaction expiry helps reduce the cost for an attacker.

Additionally, the failure mode is currently catastrophic.

• Filling the mempool currently results in a node crash.

With such an attack, someone might be able to knock over mining pool operators, and thus significantly drop the hashpower of the network, facilitating a 51% attack.

Suggested course of action:

Set limits on the mempool size, and reject further transactions.
Other steps that result in the reduction of the likelihood of a successful DoS attack.

[leto]

@zebambam Are there any bounties for this? It's pretty trivial to DoS the entire Zcash network and prevent any real transaction from occurring, for a few dollars a day. One attack which I have discovered, in the process of designing defenses, is called the "Sapling Wood-Chipper", but without financial incentive from ECC, such as a real bug bounty program, the tool will remain private.

[zebambam]

Hi @leto there's currently no bounty for bugs on ECC, partly because (if you click the label "denial of service", you'll see for yourself) we already have quite a few filed. This ticket is just to coordinate and track them through our prioritization process.

[zebambam]

Also I love that movie, thanks for the reference!

[leto]

@zebambam Just to clarify: Currently it would cost 1/20th a ZEC (about $4) to fill all transactions of all blocks per day, on the Zcash mainnet via the Sapling Woodchipper. Looking at all the bugs filed with "Denial of service", I don't see anything similar. Do you want a dedicated Github Issue for this network-wide transaction Denial-of-Service ? This is a severe attack and again, it's very worrisome that there is no Zcash bug bounty program.

[zebambam]

Not all of the DoS bugs that we are aware of are filed as bugs currently, for what are probably obvious reasons. Far more worrisome to me than our lack of bug bounty program is our inability to resolve bugs for the amount of time that some of these have filed. It's something that I'm doing my best to work on, hence this ticket. Once our problem shifts from "we can't fix bugs in a timely fashion" to "we don't know about enough bugs" then the time would be right to start a bug bounty program.

[tearodactyl]

@zebambam, thanks for noting that over three dozen DoS Issues could use some attention. And so so sorry to hear that it looks like a bit of work. Hopefully they are mostly minor, compared with what @leto is describing. Then again, 20% of $412M could buy some engineering time, even in Silly Valley, to secure the network and plug the holes in this rather porous implementation. Your position to close out already filed Issues before there is value in looking at new ones assumes that you can close them faster than new ones can be found.

ECC management may entertain a remote possibility that there exist expert developers who have not been employed by ECC, but have several years of experience digging through and plugging holes in numerous versions of the Zcash codebase. Possibly these are not the typical bug bounty jocks, looking for quick hits by scoping out some peripheral bugs. Maybe such serious blockchain developers have already identified mitigations and actually tested the solutions. I'm sure you'd understand that with a mortgage due, they cannot set aside paying gigs, in order to walk salaried ECC staff through the intricacies of the discoveries and details of the fixes. As long-term participants in the ZK ecosystem, they may even consider being compensated in ZEC for their time.

[zebambam]

@tearodactyl I think it's possible that in your analysis of the situation you've committed the sin of singularity - in fact there's a plurality of activity at ECC and fixing security bugs is only a component of that, with investment proportioned accordingly. The majority of the funds go to developing code from which other neighboring projects can benefit for free. I'd be delighted to be able to pay for more help fixing issues, but not at the expense of ECC fixing bugs in the code that we write. I expect that supplanting internal quality processes with external help would align incentives incorrectly and result in further degraded performance on ECC's part. At least as a primary strategy. I'm not opposed to that strategy to supplement some internal future effort to remediate serious DoS issues like the ones of the magnitude that Leto has been describing.

[leto]

@mineZcash if ECC does not want to run a real bug bounty program like Monero (https://hackerone.com/monero) or Horizen (they run their own, and actually value bug reports), I guess ECC does not really care about the security of the Zcash network very much.

No reason to bring in paperwork-fetishizing bureaucrats to this technical discussion. It seems the decision to ignore the severeness of this network-wide Denial-of-Service has already been made.

[zebambam]

Hey.. everyone calm down. This ticket was raised precisely so that we can get these issues into a higher priority spot in the backlog. The decision about the priority of these issues is far from over, I raised this ticket precisely so that we can have a better internal conversation about it.

[nathan-at-least]

Just to clarify nowhere in this ticket has any ECC representative said that we do not want to run a bug bounty program. We do not currently have a bug bounty program.

[daira]

It's a good job this is the issue for the general DoS "epic" rather than something more specific, otherwise I would be ruthlessly deleting off-topic comments by this point :-/

[mineZcash]

Related:
https://twitter.com/dukeleto/status/1125306275812737024?s=19

https://saplingwoodchipper.github.io

[zebambam]

Duke @leto filed this issue as CVE-2019-11636 here: https://www.cvedetails.com/cve/CVE-2019-11636/

We are still working on scheduling fixes.

[campassi]

"Availability Impact: Partial" in the CVE detailed markup seems far off target. @leto do you have any suggestions available which may swiftly patch this issue?

Surprising that a knowledgeable onlooker has not yet shorted the market while simultaneously launching an attack.

With such an attack having the distinct capacity to cripple ECC's regular income stream, should this vuln not be bumped to highest priority?

[JonLittleIT]

Any update on getting this fixed? Curious if it was left un-fixed and why? Says issue is open so I assume yes but if someone can please kindly explain why delay?

[daira]

When we spot [denial of service problems], we should link them to this epic.

• A small number of shielded transaction of massive size.
• A small number of or many transactions with many empty outputs.
• Many small transactions filling the mempool currently crash the nodes that receive them. It’s possible to knock nodes offline. Transaction expiry helps reduce the cost for an attacker.

Additionally, the failure mode is currently catastrophic.

• Filling the mempool currently results in a node crash.

The node crashes were fixed by implementation of ZIP 401. The remainder have been adequately addressed by ZIP 317.