Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement BIPs 141, 142, 143, 145 and 147 (Segregated Witness) #656

Merged
merged 33 commits into from Aug 14, 2017
Merged

Implement BIPs 141, 142, 143, 145 and 147 (Segregated Witness) #656

merged 33 commits into from Aug 14, 2017

Conversation

Roasbeef
Copy link
Member

@Roasbeef Roasbeef commented Apr 8, 2016 •
edited

This PR makes several modifications across btcd, and many of the sub-packages in order to add compatibility for the upcoming Segregated Witness soft-fork. As the set of changes is rather expansive, this PR has been split up into several commits to in order to facilitate thorough code review of each change.

This PR currently implements the changes dictated by the following BIPS:

As of the writing of this PR description, with this patch set btcd is able to successful sync up, and correctly validate all blocks and witness commitments on Bitcoin'stestnet. This PR is still being extensively tested in order to ensure all new consensus critical decision related to transactions, blocks, and script validation are functioning properly.

The changes across all packages are briefly summarized below:

  • blockchain:
    • New functions have been added to calculate block, and transaction "cost".
    • The previous BuildMerkleTreeStore function is now able to calculate the witness merkle root. This allowed the creation of new functions to extract, and validate the witness commitment.
    • The new "cost" metrics are now used when validating blocks, and transactions.
    • The txscript.ScriptVerifyWitness is now used by default when validating transaction scripts. Once BIP9 support is in place, this will instead be gated by the appropriate versionbits stage threshold.
    • Script validation performed by txValidator now also uses the new HashCache in order to more efficiently validate inputs spending witness program outputs.
    • When checking BIP34 for blocks below height 16, we now attempt to extract just a small integer. This behavior never came when BIP34 was rolled out, as it was deployed around height 100k or so.
  • chaincfg:
    • The parameters for segnet4 have been added, along with the new address bytes for the new standard output types.
  • database:
    • Blocks, and transactions are always stored using the witness encoding if they have witness data. Additionally, all blocks are attempted to be decoded using the witness encoding.
  • peer:
    • Detecting if a peer supports segwit has been added.
    • A new queue function QueueMessageWithEncoding has been added in order to allow callers to specify the encoding type to use when serializing messages. This was required in order to allow callers to selectively send witness data to peers.
    • The MaxProtocolVersion has been increased to 70012 in order to signal to other peers we support segwit.
  • txscript:
    • Validation logic for p2wsh, p2wkh, and nested p2sh outputs have been added. In the process txscript.Engine now takes an optional HashCache as well as the input value of the output being spent.
    • Support for the new sighash digest algorithm defined in BIP143 has been added. In order to make full use of the new optimizations, a new caching structure, the HashCache has been introduced which caches partial sighashes to be re-used within the mempool, and block across all concurrent script validation goroutines.
    • New functions have been added to calculate sig op cost using the new sig op calculation introduced as part of segwit.
    • Several new utility functions have been introduced for analyzing, creating, and classifying the new standard output types.
  • wire:
    • The new SFNodeWitness service bit has been added along with new inventory types for segwit, and and increase in the MaxBlockPayload size.
    • New methods have been added to support the new encoding format for transaction with witness data. In the process, a new field has been added to the wire.Message interface in order to cleanly facilitate optinoally encoding or decoding the witness data. Finally, a new methods has been added to wire.MsgTx to compute the wtxid, and ascertain if a transaction has any witness data.
  • btcd itself:
    • Only peers supporting segwit are chosen to be the syncPeer.
    • All wire.MsgGetData sent will request with the new witness inv types by default. Additionally the new inv types are no longer ignored when received by other peers (TODO(roasbeef): thought they were only to be used within getdata?).
    • The new cost metrics are used across the mempool when validating transactions.
    • Peers created by the server advertise support for segwit.

Thanks to @T909 for the initial work which laid the base for much of this PR, along with coming up with the initial idea for the HashCache.

TODO

  • Implement BIP 145 support
  • Update the rpcserver, and btcjson to display the new transactions, and block related data. The display changes can be summarized as: display of witness data needs to be added to the transaction display, add a new hash field to show the wtxid, add vsize (virtual size) to transaction display, show cost for blocks.
  • Significantly improve test coverage across all packages which have been modified, or had new functionality introduced.
  • Integrate Bitcoin Core's new tests within script_tests.json
  • Integrate Bitcoin Core's segwit transaction tests within tx_valid.json, and tx_invalid.json
  • Add new, proper error messages within txscript.Engine to be returned when a new segwit specific error is encountered. At the moment, it returns some ad-hoc errors I created to facilitate debugging.
  • Clean up the new witness validation logic in txscript.Engine.
  • Update cpuminer.go, and the GBT logic to account for the new cost metrics.
  • Add test coverage for the new sig op counting code, and witness commitment validation code.
  • Once BIP 9 has been implemented for btcd, guard new seg wit behavior with a checking of the versionbits state.
  • Add passive partial sighash eviction to HashCache
  • Modify the addrindex, and txindex to enable support for fully indexing blocks with witness data.

Closes #962.

davecgh
davecgh reviewed Apr 8, 2016
View changes
chaincfg/genesis.go Outdated
PrevBlock: wire.ShaHash{}, // 0000000000000000000000000000000000000000000000000000000000000000
MerkleRoot: segNetGenesisMerkleRoot, // 4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
Timestamp: time.Unix(1452831101, 0), // 2016-01-15 04:11:41 +0000 UTC
Bits: 0x1e01ffff, // ?? [000007ffff000000000000000000000000000000000000000000000000000000]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?? == 503447551

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the bits equate to 000001ffff000000000000000000000000000000000000000000000000000000 as opposed to the value in the comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The value in the comment was left over from segnet3.

@davecgh
Copy link
Member

davecgh commented Apr 9, 2016

Thanks for all of the work on this. I know it was a lot of it. I also really appreciate the detail put into PR description regarding the changes.

davecgh
davecgh reviewed Apr 9, 2016
View changes
txscript/hashcache.go Outdated
@@ -0,0 +1,101 @@
// Copyright (c) 2013-2016 The btcsuite developers
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2016

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@Roasbeef
btcec: add new IsCompressedPubKey function
2b928de 
This commit adds a new function to btcec: IsCompressedPubKey. This
function returns true iff the passed serialized public key is encoded
in compressed format.
@Roasbeef
txscript: add verification of the post-segwit minimal if policy
064b869 
This commit modifies the op-code execution for OP_IF and OP_NOTIF to
enforce the additional “minimal if” constraints which require the
top-stack item when the op codes are encountered to be either an empty
vector, or exactly [0x01].
@Roasbeef
txscript: add verification of the post-segwit pub key type constraint
9984ea6 
This commit adds verification of the post-segwit standardness
requirement that all pubkeys involved in checks operations MUST be
serialized as compressed public keys. A new ScriptFlag has been added
to guard this behavior when executing scripts.
@Roasbeef
txscript: add new post segwit policies to standard script flags
2d4f350
@Roasbeef
txscript: convert all new segwit error types to ErrorCode's
1a55439
@Roasbeef
txscript: update reference tests to include new segwit related cases
9be93bc
@Roasbeef
BIP0141+wire: add a WitnessHash method to tx
9b6f519
@Roasbeef
chaincfg: add address bytes for p2wsh and p2wkh addresses
181dc4e
@Roasbeef
BIP0141+blockchain: implement tx/block weight calculation funcitons
73428ee 
This commit implements the new “weight” metric introduced as part of
the segwit soft-fork. Post-fork activation, rather than limiting the
size of blocks and transactions based purely on serialized size, a new
metric “weight” will instead be used as a way to more accurately
reflect the costs of a tx/block on the system. With blocks constrained
by weight, the maximum block-size increases to ~4MB.
@Roasbeef
BIP0141+blockchain: add functions for extracting and validating witne…
fc9dd8a 
…ss commitment
@Roasbeef
BIP0141+blockchain: implement segwit block validation rules
1972ff3 
This commit implements the new block validation rules as defined by
BIP0141. The new rules include the constraints that if a block has
transactions with witness data, then there MUST be a commitment within
the conies transaction to the root of a new merkle tree which commits
to the wtxid of all transactions. Additionally, rather than limiting
the size of a block by size in bytes, blocks are now limited by their
total weight unit. Similarly, a newly define “sig op cost” is now used
to limit the signature validation cost of transactions found within
blocks.
@Roasbeef
blockchain/indexers: add addrindex support for p2wkh and p2wsh
dac46ff
@Roasbeef
mempool: modify mempool sanity checks to be segwit aware
62e8982
@Roasbeef
btcjson: update RPC calls to return segwit related data
dbc262c
@Roasbeef
mining+config: modify GBT mining to limit by weight, add witness comm…
3ba4285 
…itment

This commit modifies the existing block selection logic to limit
preferentially by weight instead of serialized block size, and also to
adhere to the new sig-op cost limits which are weighted according to
the witness discount.
@Roasbeef
chaincfg+integration: add BIP0009 deployment parameters for segwit
65060a3 
This commit adds set of BIP0009 (Version Bits) deployment parameters
for all networks detailing the activation parameters for the segwit
soft-fork.

Additionally, the BIP0009 integration test has been updated to test for
the proper transitioning of version bits state for the segwit soft
fork. Finally, the `getblockchaininfo` test has also been updated to
properly display the state of segwit.
@Roasbeef
blockchain: guard enforcement of segwit rules by a BIP0009 state check
83a27be 
This commit updates the new segwit validation logic within block
validation to be guarded by an initial check to the version bits state
before conditionally enforcing the logic based off of the state.
@Roasbeef
mining: update GBT generation to include witness commitment
a8f41fa 
This commit updates the block template generation logic to only include
witness transactions once the soft-fork has activated and to also
include the OP_RETURN witness commitment (with additional block weight
accounting).
@Roasbeef
blockchain: only populate hashcache if tx is segwitty
6d9d18c
@Roasbeef
integration/rpctest: update API usage due to segwit
c9d1a8c
@Roasbeef
build: update glide.lock to target latest btcutil commit
a925475
@davecgh davecgh mentioned this pull request Aug 13, 2017
Merged
davecgh
davecgh approved these changes Aug 14, 2017
View changes
Copy link
Member

@davecgh davecgh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll take care of the test failure in another PR since it's a false positive and this is ready otherwise.

@davecgh davecgh merged commit 01f26a1 into btcsuite:master Aug 14, 2017
@dan-da
Copy link
Contributor

dan-da commented Aug 15, 2017

congrats on merging it. any eta for next btcd release?

@ysangkok ysangkok mentioned this pull request Nov 14, 2017
Closed
@timjrobinson timjrobinson mentioned this pull request Dec 24, 2017
Closed
@ysangkok ysangkok mentioned this pull request Feb 7, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ysangkok ysangkok ysangkok left review comments

@stevenroose stevenroose stevenroose left review comments

@davecgh davecgh davecgh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.13.0
Development

Successfully merging this pull request may close these issues.

None yet
9 participants
@Roasbeef @davecgh @justusranvier @dajohi @afk11 @stevenroose @dan-da @ysangkok @wallclockbuilder