Overwinter SignatureHash

[str4d]

Implements ZIP 143.

Includes code cherry-picked from the following upstream PRs:

• Report non-mandatory script failures correctly bitcoin/bitcoin#7276
• Remove obsolete reference to CValidationState from UpdateCoins. bitcoin/bitcoin#7976
• Reduce unnecessary hashing in signrawtransaction bitcoin/bitcoin#8118
• Segregated witness rebased bitcoin/bitcoin#8149
  • Only amount validation and SignatureHash commits.
• Mempool: Use Consensus::CheckTxInputs direclty over main::CheckInputs bitcoin/bitcoin#8346
• Precompute sighashes bitcoin/bitcoin#8524

Part of #2074 and #2254. Closes #1408 and #2584.

[str4d]

This is a rebased and cleaned-up version of #2896. Still todo:

• Rebase on top of activation logic.
• Implement version-handling (making parts of the transaction validation branch ID-aware).
• Switch inner hashes to use BLAKE2b-256.
• Rebase on top of transaction format changes.
• Add / update tests.
• Update zcbenchmark to use new transaction format with caching.

[str4d]

Note to reviewers: do not trust the commit ordering on GitHub, it does not handle the rebasing well.

[str4d]

Just to check that nothing was horribly broken, I adjusted zcbenchmark to use the more efficient sighash, and compared it to current master. Results (measured on my dev laptop, so take values with a grain of salt):

Current SigHash

[
  {
    "runningtime": 0.122434
  },
  {
    "runningtime": 0.117106
  },
  {
    "runningtime": 0.117861
  },
  {
    "runningtime": 0.120877
  },
  {
    "runningtime": 0.119742
  }
]

New SigHash

[
  {
    "runningtime": 0.118283
  },
  {
    "runningtime": 0.108241
  },
  {
    "runningtime": 0.106504
  },
  {
    "runningtime": 0.110444
  },
  {
    "runningtime": 0.109168
  }
]

New SigHash, with caching

[
  {
    "runningtime": 0.041709
  },
  {
    "runningtime": 0.042964
  },
  {
    "runningtime": 0.040389
  },
  {
    "runningtime": 0.04138
  },
  {
    "runningtime": 0.041503
  }
]

[str4d]

Once I rebase this on #2925 (so all the underlying pieces are in place, and this code can be finished), I will collect benchmark data for both the old and new SignatureHash as a function of the number of inputs, to verify that this PR does indeed address the performance problem sufficiently.

[zkbot]

☔ The latest upstream changes (presumably #2898) made this pull request unmergeable. Please resolve the merge conflicts.

[zkbot]

☔ The latest upstream changes (presumably #2940) made this pull request unmergeable. Please resolve the merge conflicts.

[str4d]

Rebased on master to fix merge conflicts, namely removing the partial commit from bitcoin/bitcoin#6915 which was fully back-ported in #2940.

@zkbot try

[zkbot]

⌛ Trying commit 4553901 with merge 6db1012...

[zkbot]

💔 Test failed - pr-try

[daira]

ACK https://gist.github.com/str4d/ea30aa290d3a84555e973f44a9c0c0e0

[arielgabizon]

why are we running this test that always fails?

[str4d]

@arielgabizon so we know that the expected-failures builder is working. We added that builder so we had the abillity to merge known-failing tests before we merged something to make them pass. There have been several instances of wanting to do this, but obviously making them regular tests that fail would mean our CI system would refuse to merge the PR 🙃

[str4d]

The failure in the try above was a spurious hang in the RPC tests.

Thanks all! Any further issues raised can be addressed in subsequent PRs.

@zkbot r+

[zkbot]

📌 Commit 4553901 has been approved by str4d

[zkbot]

⌛ Testing commit 4553901 with merge 8487be8...

[zkbot]

☀️ Test successful - pr-merge
Approved by: str4d
Pushing 8487be8 to master...