Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement BIPs 141, 142, 143, 145 and 147 (Segregated Witness) #656

merged 33 commits into from Aug 14, 2017


Copy link

@Roasbeef Roasbeef commented Apr 8, 2016

This PR makes several modifications across btcd, and many of the sub-packages in order to add compatibility for the upcoming Segregated Witness soft-fork. As the set of changes is rather expansive, this PR has been split up into several commits to in order to facilitate thorough code review of each change.

This PR currently implements the changes dictated by the following BIPS:

As of the writing of this PR description, with this patch set btcd is able to successful sync up, and correctly validate all blocks and witness commitments on Bitcoin'stestnet. This PR is still being extensively tested in order to ensure all new consensus critical decision related to transactions, blocks, and script validation are functioning properly.

The changes across all packages are briefly summarized below:

  • blockchain:
    • New functions have been added to calculate block, and transaction "cost".
    • The previous BuildMerkleTreeStore function is now able to calculate the witness merkle root. This allowed the creation of new functions to extract, and validate the witness commitment.
    • The new "cost" metrics are now used when validating blocks, and transactions.
    • The txscript.ScriptVerifyWitness is now used by default when validating transaction scripts. Once BIP9 support is in place, this will instead be gated by the appropriate versionbits stage threshold.
    • Script validation performed by txValidator now also uses the new HashCache in order to more efficiently validate inputs spending witness program outputs.
    • When checking BIP34 for blocks below height 16, we now attempt to extract just a small integer. This behavior never came when BIP34 was rolled out, as it was deployed around height 100k or so.
  • chaincfg:
    • The parameters for segnet4 have been added, along with the new address bytes for the new standard output types.
  • database:
    • Blocks, and transactions are always stored using the witness encoding if they have witness data. Additionally, all blocks are attempted to be decoded using the witness encoding.
  • peer:
    • Detecting if a peer supports segwit has been added.
    • A new queue function QueueMessageWithEncoding has been added in order to allow callers to specify the encoding type to use when serializing messages. This was required in order to allow callers to selectively send witness data to peers.
    • The MaxProtocolVersion has been increased to 70012 in order to signal to other peers we support segwit.
  • txscript:
    • Validation logic for p2wsh, p2wkh, and nested p2sh outputs have been added. In the process txscript.Engine now takes an optional HashCache as well as the input value of the output being spent.
    • Support for the new sighash digest algorithm defined in BIP143 has been added. In order to make full use of the new optimizations, a new caching structure, the HashCache has been introduced which caches partial sighashes to be re-used within the mempool, and block across all concurrent script validation goroutines.
    • New functions have been added to calculate sig op cost using the new sig op calculation introduced as part of segwit.
    • Several new utility functions have been introduced for analyzing, creating, and classifying the new standard output types.
  • wire:
    • The new SFNodeWitness service bit has been added along with new inventory types for segwit, and and increase in the MaxBlockPayload size.
    • New methods have been added to support the new encoding format for transaction with witness data. In the process, a new field has been added to the wire.Message interface in order to cleanly facilitate optinoally encoding or decoding the witness data. Finally, a new methods has been added to wire.MsgTx to compute the wtxid, and ascertain if a transaction has any witness data.
  • btcd itself:
    • Only peers supporting segwit are chosen to be the syncPeer.
    • All wire.MsgGetData sent will request with the new witness inv types by default. Additionally the new inv types are no longer ignored when received by other peers (TODO(roasbeef): thought they were only to be used within getdata?).
    • The new cost metrics are used across the mempool when validating transactions.
    • Peers created by the server advertise support for segwit.

Thanks to @T909 for the initial work which laid the base for much of this PR, along with coming up with the initial idea for the HashCache.


  • Implement BIP 145 support
  • Update the rpcserver, and btcjson to display the new transactions, and block related data. The display changes can be summarized as: display of witness data needs to be added to the transaction display, add a new hash field to show the wtxid, add vsize (virtual size) to transaction display, show cost for blocks.
  • Significantly improve test coverage across all packages which have been modified, or had new functionality introduced.
  • Integrate Bitcoin Core's new tests within script_tests.json
  • Integrate Bitcoin Core's segwit transaction tests within tx_valid.json, and tx_invalid.json
  • Add new, proper error messages within txscript.Engine to be returned when a new segwit specific error is encountered. At the moment, it returns some ad-hoc errors I created to facilitate debugging.
  • Clean up the new witness validation logic in txscript.Engine.
  • Update cpuminer.go, and the GBT logic to account for the new cost metrics.
  • Add test coverage for the new sig op counting code, and witness commitment validation code.
  • Once BIP 9 has been implemented for btcd, guard new seg wit behavior with a checking of the versionbits state.
  • Add passive partial sighash eviction to HashCache
  • Modify the addrindex, and txindex to enable support for fully indexing blocks with witness data.

Closes #962.

PrevBlock: wire.ShaHash{}, // 0000000000000000000000000000000000000000000000000000000000000000
MerkleRoot: segNetGenesisMerkleRoot, // 4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
Timestamp: time.Unix(1452831101, 0), // 2016-01-15 04:11:41 +0000 UTC
Bits: 0x1e01ffff, // ?? [000007ffff000000000000000000000000000000000000000000000000000000]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?? == 503447551

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the bits equate to 000001ffff000000000000000000000000000000000000000000000000000000 as opposed to the value in the comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The value in the comment was left over from segnet3.

Copy link

davecgh commented Apr 9, 2016

Thanks for all of the work on this. I know it was a lot of it. I also really appreciate the detail put into PR description regarding the changes.

@@ -0,0 +1,101 @@
// Copyright (c) 2013-2016 The btcsuite developers
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


This commit adds a new function to btcec: IsCompressedPubKey. This
function returns true iff the passed serialized public key is encoded
in compressed format.
This commit modifies the op-code execution for OP_IF and OP_NOTIF to
enforce the additional “minimal if” constraints which require the
top-stack item when the op codes are encountered to be either an empty
vector, or exactly [0x01].
This commit adds verification of the post-segwit standardness
requirement that all pubkeys involved in checks operations MUST be
serialized as compressed public keys. A new ScriptFlag has been added
to guard this behavior when executing scripts.
This commit implements the new “weight” metric introduced as part of
the segwit soft-fork. Post-fork activation, rather than limiting the
size of blocks and transactions based purely on serialized size, a new
metric “weight” will instead be used as a way to more accurately
reflect the costs of a tx/block on the system. With blocks constrained
by weight, the maximum block-size increases to ~4MB.
This commit implements the new block validation rules as defined by
BIP0141. The new rules include the constraints that if a block has
transactions with witness data, then there MUST be a commitment within
the conies transaction to the root of a new merkle tree which commits
to the wtxid of all transactions. Additionally, rather than limiting
the size of a block by size in bytes, blocks are now limited by their
total weight unit. Similarly, a newly define “sig op cost” is now used
to limit the signature validation cost of transactions found within

This commit modifies the existing block selection logic to limit
preferentially by weight instead of serialized block size, and also to
adhere to the new sig-op cost limits which are weighted according to
the witness discount.
This commit adds set of BIP0009 (Version Bits) deployment parameters
for all networks detailing the activation parameters for the segwit

Additionally, the BIP0009 integration test has been updated to test for
the proper transitioning of version bits state for the segwit soft
fork. Finally, the `getblockchaininfo` test has also been updated to
properly display the state of segwit.
This commit updates the new segwit validation logic within block
validation to be guarded by an initial check to the version bits state
before conditionally enforcing the logic based off of the state.
This commit updates the block template generation logic to only include
witness transactions once the soft-fork has activated and to also
include the OP_RETURN witness commitment (with additional block weight
Copy link

@davecgh davecgh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll take care of the test failure in another PR since it's a false positive and this is ready otherwise.

@davecgh davecgh merged commit 01f26a1 into btcsuite:master Aug 14, 2017
Copy link

dan-da commented Aug 15, 2017

congrats on merging it. any eta for next btcd release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

9 participants