Implement BIPs 141, 142, 143, 145 and 147 (Segregated Witness) by Roasbeef · Pull Request #656 · btcsuite/btcd

Roasbeef · 2016-04-08T20:09:35Z

This PR makes several modifications across btcd, and many of the sub-packages in order to add compatibility for the upcoming Segregated Witness soft-fork. As the set of changes is rather expansive, this PR has been split up into several commits to in order to facilitate thorough code review of each change.

This PR currently implements the changes dictated by the following BIPS:

As of the writing of this PR description, with this patch set btcd is able to successful sync up, and correctly validate all blocks and witness commitments on Bitcoin'stestnet. This PR is still being extensively tested in order to ensure all new consensus critical decision related to transactions, blocks, and script validation are functioning properly.

The changes across all packages are briefly summarized below:

blockchain:
- New functions have been added to calculate block, and transaction "cost".
- The previous BuildMerkleTreeStore function is now able to calculate the witness merkle root. This allowed the creation of new functions to extract, and validate the witness commitment.
- The new "cost" metrics are now used when validating blocks, and transactions.
- The txscript.ScriptVerifyWitness is now used by default when validating transaction scripts. Once BIP9 support is in place, this will instead be gated by the appropriate versionbits stage threshold.
- Script validation performed by txValidator now also uses the new HashCache in order to more efficiently validate inputs spending witness program outputs.
- When checking BIP34 for blocks below height 16, we now attempt to extract just a small integer. This behavior never came when BIP34 was rolled out, as it was deployed around height 100k or so.
chaincfg:
- The parameters for segnet4 have been added, along with the new address bytes for the new standard output types.
database:
- Blocks, and transactions are always stored using the witness encoding if they have witness data. Additionally, all blocks are attempted to be decoded using the witness encoding.
peer:
- Detecting if a peer supports segwit has been added.
- A new queue function QueueMessageWithEncoding has been added in order to allow callers to specify the encoding type to use when serializing messages. This was required in order to allow callers to selectively send witness data to peers.
- The MaxProtocolVersion has been increased to 70012 in order to signal to other peers we support segwit.
txscript:
- Validation logic for p2wsh, p2wkh, and nested p2sh outputs have been added. In the process txscript.Engine now takes an optional HashCache as well as the input value of the output being spent.
- Support for the new sighash digest algorithm defined in BIP143 has been added. In order to make full use of the new optimizations, a new caching structure, the HashCache has been introduced which caches partial sighashes to be re-used within the mempool, and block across all concurrent script validation goroutines.
- New functions have been added to calculate sig op cost using the new sig op calculation introduced as part of segwit.
- Several new utility functions have been introduced for analyzing, creating, and classifying the new standard output types.
wire:
- The new SFNodeWitness service bit has been added along with new inventory types for segwit, and and increase in the MaxBlockPayload size.
- New methods have been added to support the new encoding format for transaction with witness data. In the process, a new field has been added to the wire.Message interface in order to cleanly facilitate optinoally encoding or decoding the witness data. Finally, a new methods has been added to wire.MsgTx to compute the wtxid, and ascertain if a transaction has any witness data.
btcd itself:
- Only peers supporting segwit are chosen to be the syncPeer.
- All wire.MsgGetData sent will request with the new witness inv types by default. Additionally the new inv types are no longer ignored when received by other peers (TODO(roasbeef): thought they were only to be used within getdata?).
- The new cost metrics are used across the mempool when validating transactions.
- Peers created by the server advertise support for segwit.

Thanks to @T909 for the initial work which laid the base for much of this PR, along with coming up with the initial idea for the HashCache.

TODO

Closes #962.

davecgh · 2016-04-08T22:05:35Z

+		PrevBlock:  wire.ShaHash{},           // 0000000000000000000000000000000000000000000000000000000000000000
+		MerkleRoot: segNetGenesisMerkleRoot,  // 4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
+		Timestamp:  time.Unix(1452831101, 0), // 2016-01-15 04:11:41 +0000 UTC
+		Bits:       0x1e01ffff,               // ?? [000007ffff000000000000000000000000000000000000000000000000000000]


?? == 503447551

Also, the bits equate to 000001ffff000000000000000000000000000000000000000000000000000000 as opposed to the value in the comment.

Fixed. The value in the comment was left over from segnet3.

davecgh · 2016-04-09T07:20:15Z

Thanks for all of the work on this. I know it was a lot of it. I also really appreciate the detail put into PR description regarding the changes.

davecgh · 2016-04-09T07:24:14Z

@@ -0,0 +1,101 @@
+// Copyright (c) 2013-2016 The btcsuite developers


This commit adds a new function to btcec: IsCompressedPubKey. This function returns true iff the passed serialized public key is encoded in compressed format.

This commit modifies the op-code execution for OP_IF and OP_NOTIF to enforce the additional “minimal if” constraints which require the top-stack item when the op codes are encountered to be either an empty vector, or exactly [0x01].

This commit adds verification of the post-segwit standardness requirement that all pubkeys involved in checks operations MUST be serialized as compressed public keys. A new ScriptFlag has been added to guard this behavior when executing scripts.

This commit implements the new “weight” metric introduced as part of the segwit soft-fork. Post-fork activation, rather than limiting the size of blocks and transactions based purely on serialized size, a new metric “weight” will instead be used as a way to more accurately reflect the costs of a tx/block on the system. With blocks constrained by weight, the maximum block-size increases to ~4MB.

…ss commitment

This commit implements the new block validation rules as defined by BIP0141. The new rules include the constraints that if a block has transactions with witness data, then there MUST be a commitment within the conies transaction to the root of a new merkle tree which commits to the wtxid of all transactions. Additionally, rather than limiting the size of a block by size in bytes, blocks are now limited by their total weight unit. Similarly, a newly define “sig op cost” is now used to limit the signature validation cost of transactions found within blocks.

…itment This commit modifies the existing block selection logic to limit preferentially by weight instead of serialized block size, and also to adhere to the new sig-op cost limits which are weighted according to the witness discount.

This commit adds set of BIP0009 (Version Bits) deployment parameters for all networks detailing the activation parameters for the segwit soft-fork. Additionally, the BIP0009 integration test has been updated to test for the proper transitioning of version bits state for the segwit soft fork. Finally, the `getblockchaininfo` test has also been updated to properly display the state of segwit.

This commit updates the new segwit validation logic within block validation to be guarded by an initial check to the version bits state before conditionally enforcing the logic based off of the state.

This commit updates the block template generation logic to only include witness transactions once the soft-fork has activated and to also include the OP_RETURN witness commitment (with additional block weight accounting).

davecgh

I'll take care of the test failure in another PR since it's a false positive and this is ready otherwise.

dan-da · 2017-08-15T02:20:13Z

congrats on merging it. any eta for next btcd release?

davecgh reviewed Apr 8, 2016
View reviewed changes

davecgh reviewed Apr 9, 2016
View reviewed changes

Roasbeef added 13 commits July 31, 2017 11:55

btcec: add new IsCompressedPubKey function

2b928de

This commit adds a new function to btcec: IsCompressedPubKey. This function returns true iff the passed serialized public key is encoded in compressed format.

txscript: add new post segwit policies to standard script flags

2d4f350

txscript: convert all new segwit error types to ErrorCode's

1a55439

txscript: update reference tests to include new segwit related cases

9be93bc

BIP0141+wire: add a WitnessHash method to tx

9b6f519

chaincfg: add address bytes for p2wsh and p2wkh addresses

181dc4e

BIP0141+blockchain: add functions for extracting and validating witne…

fc9dd8a

…ss commitment

blockchain/indexers: add addrindex support for p2wkh and p2wsh

dac46ff

mempool: modify mempool sanity checks to be segwit aware

62e8982

Roasbeef force-pushed the segwit-final branch from fee6f82 to 26a3aed Compare July 31, 2017 18:55

Roasbeef added 8 commits July 31, 2017 12:46

btcjson: update RPC calls to return segwit related data

dbc262c

blockchain: guard enforcement of segwit rules by a BIP0009 state check

83a27be

This commit updates the new segwit validation logic within block validation to be guarded by an initial check to the version bits state before conditionally enforcing the logic based off of the state.

mining: update GBT generation to include witness commitment

a8f41fa

This commit updates the block template generation logic to only include witness transactions once the soft-fork has activated and to also include the OP_RETURN witness commitment (with additional block weight accounting).

blockchain: only populate hashcache if tx is segwitty

6d9d18c

integration/rpctest: update API usage due to segwit

c9d1a8c

build: update glide.lock to target latest btcutil commit

a925475

Roasbeef force-pushed the segwit-final branch from 812cb3f to a925475 Compare July 31, 2017 19:46

davecgh mentioned this pull request Aug 13, 2017

rpcserver: Decouple from server. #981

Merged

davecgh approved these changes Aug 14, 2017

View reviewed changes

davecgh merged commit 01f26a1 into btcsuite:master Aug 14, 2017

ysangkok mentioned this pull request Nov 14, 2017

Update to newer btcd btcsuite/btcwallet#487

Closed

timjrobinson mentioned this pull request Dec 24, 2017

Update INSTALL.md lightningnetwork/lnd#323

Closed

ysangkok mentioned this pull request Feb 7, 2018

README.md claims segwit support has not been merged into master lightningnetwork/lnd#740

Closed

Navigation Menu

Uh oh!

Implement BIPs 141, 142, 143, 145 and 147 (Segregated Witness) - #656

Implement BIPs 141, 142, 143, 145 and 147 (Segregated Witness) #656
davecgh merged 33 commits into
btcsuite:masterfrom
Roasbeef:segwit-final

Roasbeef commented Apr 8, 2016 •

edited

Loading

Uh oh!

davecgh Apr 8, 2016

Uh oh!

davecgh Apr 8, 2016

Uh oh!

Roasbeef Apr 10, 2016

Uh oh!

davecgh commented Apr 9, 2016

Uh oh!

davecgh Apr 9, 2016

Uh oh!

Roasbeef Apr 10, 2016

Uh oh!

davecgh left a comment

Uh oh!

dan-da commented Aug 15, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

		@@ -0,0 +1,101 @@
		// Copyright (c) 2013-2016 The btcsuite developers

Uh oh!

Conversation

Roasbeef commented Apr 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

davecgh Apr 8, 2016

Choose a reason for hiding this comment

Uh oh!

davecgh Apr 8, 2016

Choose a reason for hiding this comment

Uh oh!

Roasbeef Apr 10, 2016

Choose a reason for hiding this comment

Uh oh!

davecgh commented Apr 9, 2016

Uh oh!

davecgh Apr 9, 2016

Choose a reason for hiding this comment

Uh oh!

Roasbeef Apr 10, 2016

Choose a reason for hiding this comment

Uh oh!

davecgh left a comment

Choose a reason for hiding this comment

Uh oh!

dan-da commented Aug 15, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Roasbeef commented Apr 8, 2016 •

edited

Loading