New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement BIPs 141, 142, 143, 145 and 147 (Segregated Witness) #656

Merged
merged 33 commits into from Aug 14, 2017

Conversation

Projects
None yet
9 participants
@Roasbeef
Member

Roasbeef commented Apr 8, 2016

This PR makes several modifications across btcd, and many of the sub-packages in order to add compatibility for the upcoming Segregated Witness soft-fork. As the set of changes is rather expansive, this PR has been split up into several commits to in order to facilitate thorough code review of each change.

This PR currently implements the changes dictated by the following BIPS:

As of the writing of this PR description, with this patch set btcd is able to successful sync up, and correctly validate all blocks and witness commitments on Bitcoin'stestnet. This PR is still being extensively tested in order to ensure all new consensus critical decision related to transactions, blocks, and script validation are functioning properly.

The changes across all packages are briefly summarized below:

  • blockchain:
    • New functions have been added to calculate block, and transaction "cost".
    • The previous BuildMerkleTreeStore function is now able to calculate the witness merkle root. This allowed the creation of new functions to extract, and validate the witness commitment.
    • The new "cost" metrics are now used when validating blocks, and transactions.
    • The txscript.ScriptVerifyWitness is now used by default when validating transaction scripts. Once BIP9 support is in place, this will instead be gated by the appropriate versionbits stage threshold.
    • Script validation performed by txValidator now also uses the new HashCache in order to more efficiently validate inputs spending witness program outputs.
    • When checking BIP34 for blocks below height 16, we now attempt to extract just a small integer. This behavior never came when BIP34 was rolled out, as it was deployed around height 100k or so.
  • chaincfg:
    • The parameters for segnet4 have been added, along with the new address bytes for the new standard output types.
  • database:
    • Blocks, and transactions are always stored using the witness encoding if they have witness data. Additionally, all blocks are attempted to be decoded using the witness encoding.
  • peer:
    • Detecting if a peer supports segwit has been added.
    • A new queue function QueueMessageWithEncoding has been added in order to allow callers to specify the encoding type to use when serializing messages. This was required in order to allow callers to selectively send witness data to peers.
    • The MaxProtocolVersion has been increased to 70012 in order to signal to other peers we support segwit.
  • txscript:
    • Validation logic for p2wsh, p2wkh, and nested p2sh outputs have been added. In the process txscript.Engine now takes an optional HashCache as well as the input value of the output being spent.
    • Support for the new sighash digest algorithm defined in BIP143 has been added. In order to make full use of the new optimizations, a new caching structure, the HashCache has been introduced which caches partial sighashes to be re-used within the mempool, and block across all concurrent script validation goroutines.
    • New functions have been added to calculate sig op cost using the new sig op calculation introduced as part of segwit.
    • Several new utility functions have been introduced for analyzing, creating, and classifying the new standard output types.
  • wire:
    • The new SFNodeWitness service bit has been added along with new inventory types for segwit, and and increase in the MaxBlockPayload size.
    • New methods have been added to support the new encoding format for transaction with witness data. In the process, a new field has been added to the wire.Message interface in order to cleanly facilitate optinoally encoding or decoding the witness data. Finally, a new methods has been added to wire.MsgTx to compute the wtxid, and ascertain if a transaction has any witness data.
  • btcd itself:
    • Only peers supporting segwit are chosen to be the syncPeer.
    • All wire.MsgGetData sent will request with the new witness inv types by default. Additionally the new inv types are no longer ignored when received by other peers (TODO(roasbeef): thought they were only to be used within getdata?).
    • The new cost metrics are used across the mempool when validating transactions.
    • Peers created by the server advertise support for segwit.

Thanks to @T909 for the initial work which laid the base for much of this PR, along with coming up with the initial idea for the HashCache.

TODO

  • Implement BIP 145 support
  • Update the rpcserver, and btcjson to display the new transactions, and block related data. The display changes can be summarized as: display of witness data needs to be added to the transaction display, add a new hash field to show the wtxid, add vsize (virtual size) to transaction display, show cost for blocks.
  • Significantly improve test coverage across all packages which have been modified, or had new functionality introduced.
  • Integrate Bitcoin Core's new tests within script_tests.json
  • Integrate Bitcoin Core's segwit transaction tests within tx_valid.json, and tx_invalid.json
  • Add new, proper error messages within txscript.Engine to be returned when a new segwit specific error is encountered. At the moment, it returns some ad-hoc errors I created to facilitate debugging.
  • Clean up the new witness validation logic in txscript.Engine.
  • Update cpuminer.go, and the GBT logic to account for the new cost metrics.
  • Add test coverage for the new sig op counting code, and witness commitment validation code.
  • Once BIP 9 has been implemented for btcd, guard new seg wit behavior with a checking of the versionbits state.
  • Add passive partial sighash eviction to HashCache
  • Modify the addrindex, and txindex to enable support for fully indexing blocks with witness data.

Closes #962.

Show outdated Hide outdated chaincfg/genesis.go Outdated
Show outdated Hide outdated chaincfg/params.go Outdated
Show outdated Hide outdated chaincfg/params.go Outdated
Show outdated Hide outdated config.go Outdated
Show outdated Hide outdated params.go Outdated
Show outdated Hide outdated wire/msgtx.go Outdated
Show outdated Hide outdated wire/msgtx.go Outdated
Show outdated Hide outdated wire/msgtx.go Outdated
Show outdated Hide outdated wire/msgtx.go Outdated
return err
}
}
// Prevent more input transactions than could possibly fit into a
// message. It would be possible to cause memory exhaustion and panics
// without a sane upper bound on this count.

This comment has been minimized.

@davecgh

davecgh Apr 9, 2016

Member

Need to verify if maxTxInPerMessage is still a constant under a witness-style transaction. The same goes for maxTxOutPerMessage. It might not have changed, but need to double check that since the tx structure did change.

@davecgh

davecgh Apr 9, 2016

Member

Need to verify if maxTxInPerMessage is still a constant under a witness-style transaction. The same goes for maxTxOutPerMessage. It might not have changed, but need to double check that since the tx structure did change.

Show outdated Hide outdated wire/msgtx.go Outdated
// Then for witCount number of stack items, each item
// has a varint length prefix, followed by the witness
// item itself.
txin.Witness = make([][]byte, witCount)

This comment has been minimized.

@davecgh

davecgh Apr 9, 2016

Member

The witCount needs to be checked against some type of sane maximum based on the max possible that could fit in a transaction. Otherwise it would be trivial for an attacker to craft a message with a huge varint encoded here to force a massive allocation of memory and crash the process.

Remember the first rule of reading data from external sources is you can never trust it.

@davecgh

davecgh Apr 9, 2016

Member

The witCount needs to be checked against some type of sane maximum based on the max possible that could fit in a transaction. Otherwise it would be trivial for an attacker to craft a message with a huge varint encoded here to force a massive allocation of memory and crash the process.

Remember the first rule of reading data from external sources is you can never trust it.

This comment has been minimized.

@Roasbeef

Roasbeef Apr 10, 2016

Member

Excellent suggestion. I've changed this fragment to limit the possible size of witCount.

@Roasbeef

Roasbeef Apr 10, 2016

Member

Excellent suggestion. I've changed this fragment to limit the possible size of witCount.

Show outdated Hide outdated wire/msgtx.go Outdated
Show outdated Hide outdated wire/msgtx.go Outdated
Show outdated Hide outdated wire/msgtx.go Outdated
Show outdated Hide outdated wire/msgtx.go Outdated
Show outdated Hide outdated server.go Outdated
Show outdated Hide outdated txscript/engine.go Outdated
Show outdated Hide outdated txscript/engine.go Outdated
Show outdated Hide outdated txscript/engine.go Outdated
Show outdated Hide outdated txscript/engine.go Outdated
Show outdated Hide outdated blockchain/merkle.go Outdated
Show outdated Hide outdated blockchain/merkle.go Outdated
Show outdated Hide outdated blockchain/merkle.go Outdated
Show outdated Hide outdated blockchain/merkle.go Outdated
Show outdated Hide outdated blockchain/merkle.go Outdated
Show outdated Hide outdated blockchain/merkle.go Outdated
@@ -69,6 +69,87 @@ func IsPayToScriptHash(script []byte) bool {
return isScriptHash(pops)
}
// isWitnessScriptHash returns true if the passed script is a
// pay-to-witness-script-hash transaction, false otherwise.
func isWitnessScriptHash(pops []parsedOpcode) bool {

This comment has been minimized.

@davecgh

davecgh Apr 9, 2016

Member

Please move this and all of the others that represent standard scripts into standard.go since this is not consensus critical code.

@davecgh

davecgh Apr 9, 2016

Member

Please move this and all of the others that represent standard scripts into standard.go since this is not consensus critical code.

This comment has been minimized.

@davecgh

davecgh Apr 9, 2016

Member

Also, these should really only be true depending on the witness script version which I don't see anything for properly detecting and handling that. I suspect this function, and the others in the same vain, should be taking the witness version and checking it against 0.

The BIP says:

If the version byte is 0, and the witness program is 32 bytes:

Followed by:

If the version byte is 1 to 16, no further interpretation of the witness program or witness happens, and there is no size restriction for the witness. These versions are reserved for future extensions. 

In other words, currently, it is only a P2WSH for version 0.

@davecgh

davecgh Apr 9, 2016

Member

Also, these should really only be true depending on the witness script version which I don't see anything for properly detecting and handling that. I suspect this function, and the others in the same vain, should be taking the witness version and checking it against 0.

The BIP says:

If the version byte is 0, and the witness program is 32 bytes:

Followed by:

If the version byte is 1 to 16, no further interpretation of the witness program or witness happens, and there is no size restriction for the witness. These versions are reserved for future extensions. 

In other words, currently, it is only a P2WSH for version 0.

This comment has been minimized.

@Roasbeef

Roasbeef Jul 3, 2017

Member

I placed these functions here as the functions for the prior standard script templates resided in this file as well.

The predicates to decide if something is p2wkh or p2wsh don't need to take in the witness version as a parameter as their definition is based on the witness version being zero, hence the check to pops[0].opcode.value == OP_0.

@Roasbeef

Roasbeef Jul 3, 2017

Member

I placed these functions here as the functions for the prior standard script templates resided in this file as well.

The predicates to decide if something is p2wkh or p2wsh don't need to take in the witness version as a parameter as their definition is based on the witness version being zero, hence the check to pops[0].opcode.value == OP_0.

@davecgh

This comment has been minimized.

Show comment
Hide comment
@davecgh

davecgh Apr 9, 2016

Member

Thanks for all of the work on this. I know it was a lot of it. I also really appreciate the detail put into PR description regarding the changes.

Member

davecgh commented Apr 9, 2016

Thanks for all of the work on this. I know it was a lot of it. I also really appreciate the detail put into PR description regarding the changes.

Show outdated Hide outdated txscript/hashcache.go Outdated

Roasbeef added some commits Apr 26, 2017

btcec: add new IsCompressedPubKey function
This commit adds a new function to btcec: IsCompressedPubKey. This
function returns true iff the passed serialized public key is encoded
in compressed format.
txscript: add verification of the post-segwit minimal if policy
This commit modifies the op-code execution for OP_IF and OP_NOTIF to
enforce the additional “minimal if” constraints which require the
top-stack item when the op codes are encountered to be either an empty
vector, or exactly [0x01].
txscript: add verification of the post-segwit pub key type constraint
This commit adds verification of the post-segwit standardness
requirement that all pubkeys involved in checks operations MUST be
serialized as compressed public keys. A new ScriptFlag has been added
to guard this behavior when executing scripts.
BIP0141+blockchain: implement tx/block weight calculation funcitons
This commit implements the new “weight” metric introduced as part of
the segwit soft-fork. Post-fork activation, rather than limiting the
size of blocks and transactions based purely on serialized size, a new
metric “weight” will instead be used as a way to more accurately
reflect the costs of a tx/block on the system. With blocks constrained
by weight, the maximum block-size increases to ~4MB.
BIP0141+blockchain: implement segwit block validation rules
This commit implements the new block validation rules as defined by
BIP0141. The new rules include the constraints that if a block has
transactions with witness data, then there MUST be a commitment within
the conies transaction to the root of a new merkle tree which commits
to the wtxid of all transactions. Additionally, rather than limiting
the size of a block by size in bytes, blocks are now limited by their
total weight unit. Similarly, a newly define “sig op cost” is now used
to limit the signature validation cost of transactions found within
blocks.
mining+config: modify GBT mining to limit by weight, add witness comm…
…itment

This commit modifies the existing block selection logic to limit
preferentially by weight instead of serialized block size, and also to
adhere to the new sig-op cost limits which are weighted according to
the witness discount.
chaincfg+integration: add BIP0009 deployment parameters for segwit
This commit adds set of BIP0009 (Version Bits) deployment parameters
for all networks detailing the activation parameters for the segwit
soft-fork.

Additionally, the BIP0009 integration test has been updated to test for
the proper transitioning of version bits state for the segwit soft
fork. Finally, the `getblockchaininfo` test has also been updated to
properly display the state of segwit.
blockchain: guard enforcement of segwit rules by a BIP0009 state check
This commit updates the new segwit validation logic within block
validation to be guarded by an initial check to the version bits state
before conditionally enforcing the logic based off of the state.
mining: update GBT generation to include witness commitment
This commit updates the block template generation logic to only include
witness transactions once the soft-fork has activated and to also
include the OP_RETURN witness commitment (with additional block weight
accounting).
@ysangkok

This comment has been minimized.

Show comment
Hide comment
@ysangkok

ysangkok Aug 5, 2017

@Roasbeef this should be witnessMarkerBytes

ysangkok commented on wire/msgtx.go in b2efc18 Aug 5, 2017

@Roasbeef this should be witnessMarkerBytes

@davecgh

I'll take care of the test failure in another PR since it's a false positive and this is ready otherwise.

@davecgh davecgh merged commit 01f26a1 into btcsuite:master Aug 14, 2017

1 check failed

continuous-integration/travis-ci/pr The Travis CI build failed
Details
@dan-da

This comment has been minimized.

Show comment
Hide comment
@dan-da

dan-da Aug 15, 2017

Contributor

congrats on merging it. any eta for next btcd release?

Contributor

dan-da commented Aug 15, 2017

congrats on merging it. any eta for next btcd release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment