New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segregated witness #7910

Closed
wants to merge 128 commits into
base: master
from

Conversation

Projects
@sipa
Copy link
Member

sipa commented Apr 19, 2016

This is the complete code for segregated witness on top of master (implementing BIP141, BIP143, BIP144, BIP145). Apart from commentary and merging (see further), the code is never rewritten/squashed/rebased, in order not to invalidate review. A rebased/squashed version is available here: #8149.

See also the detailed and up-to-date list of commits here: #7910 (comment)

There are certainly some items left to do:

  • Incorporate changes to BIP145/BIP9 when finalized (see bitcoin/bips#365)
  • Define a deployment time for testnet
  • Seed nodes that provide segwit-capable peers
  • Do tests in a mixed network of upgraded and non-upgraded nodes
  • Remove segnet from mainline merge
  • Define a deployment time for mainnet
  • Per-transaction caching of sighashes to fix the O(n^2) sighash problem (see sipa#70)

Many thanks to all contributors so far (including @jl2012, @NicolasDorier, @CodeShark, @sdaftuar, @morcos, @luke-jr, @afk11, @theuni, @TheBlueMatt, @MarcoFalke, @LongShao007, @jonasnick, @mruddy), the ideas it is based on (by @gmaxwell and @luke-jr), and many reviewers (including @SergioDemianLerner, @instagibbs, @dcousens, and @btcdrak).

sipa added some commits Mar 26, 2016

Don't check the genesis block PoW
Inspired by Jorge Timon's "Testchain: Don't check the genesis block", but
restricted to just PoW check.

Code by Matt Corallo, originally submitted as a patch to segwit.
@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Apr 19, 2016

Since github shows the commits sorted by author date rather than dependency order, here is a list of all commits:

  • preparation (sipa/bitcoin@fb0ac48...e69265d)
    • 644b2c5 Keep addrman's nService bits consistent with outbound observations
    • afdc413 Verify that outbound connections have expected services
    • fdec2bc Only store and connect to NODE_NETWORK nodes
  • segnet (sipa/bitcoin@e69265d...bb4bb47)
    • 66dbd81 Don't check the genesis block PoW
    • 70ebe86 Create segnet4
    • f8bcd86 Add segnet seed nodes
    • 58847fe qt: Work (don't crash) with -segnet
  • P2P/node/consensus (sipa/bitcoin@bb4bb47...351a2fe)
    • 3b1ff49 Add segregated witness transaction serialization
    • e35d020 Removed ppszTypeName from protocol.cpp
    • adb1c09 getdata enum issue fix
    • ac6886d Introduce and preferentially peer with NODE_WITNESS service bit
    • cb9d4d3 Witness commitment validation
    • 2c87be1 Script validation logic for witnesses
    • 1e47805 Enable SCRIPT_VERIFY_WITNESS for mempool transactions
    • 97f34f7 Activate script consensus rules through BIP9
    • 6a8021f Only download blocks from witness peers after fork
    • 271e45e Observe input amounts: verification
    • d374139 Add signature version 1 with updated sighash
    • 92c6dc3 Return witness data
    • 3955218 Implement block size/sigop cost rules, limits, and GBT support
    • 5a47b98 Add command line options to loosen mempool acceptance rules
    • e573571 bitcoinconsensus: add method that accepts amount, and return error when verify_script receives VERIFY_WITNESS flag
    • a6386c2 Increase MAX_PROTOCOL_MESSAGE_LENGTH
    • 1b6c6f1 Add rewind logic to deal with post-fork software updates
  • wallet (sipa/bitcoin@351a2fe...a2d6f8a)
    • abbe085 Witness script signing
    • c3fe53f Add witness address RPCs (using P2SH)
    • ec385fa signrawtransaction can sign P2WSH
  • tests (sipa/bitcoin@a2d6f8a...cfb0a83)
    • 2ed1d11 Signing tests
    • 3c0b16f Add rpc test for segwit
    • 24746d2 Add transaction tests for segwit
    • 3409ad6 Add segwit support to script_tests
    • e5ffcc1 Autogeneration support for witness in script_tests
    • 5e6abe5 Update p2p test framework with segwit support
    • 7b539f9 P2P test for segwit
  • fixups (sipa/bitcoin@cfb0a83...77c613c)
    • bb2bfca fixup Implement block size/sigop cost rules, limits, and GBT support: use int64_t for sigopcost everywhere
    • 64c1527 fixup Implement block size/sigop cost rules, limits, and GBT support: update block cost comment
    • cc741f5 fixup Add segregated witness transaction serialization: comments
    • c28179f fixup Update p2p test framework with segwit support: fix comment
    • 6a5da9b fixup Witness commitment validation: list the mentioned BIPs
    • 88e0135 fixup Add command line options to loosen mempool acceptance rules: simplify comment
    • da60fae fixup Witness commitment validation: factor out GetWitnessCommitmentPos
    • 4f827be fixup Witness commitment validation: fix typo in comment
    • 058f495 fixup Implement block size/sigop cost rules, limits, and GBT support: improve GBT help
    • c1e7a95 fixup Add witness address RPCs (using P2SH): test hex & wallet available
    • dbe6391 fixup Add segregated witness transaction serialization: revert extformat-if-empty-vin
    • 97d7402 tidy up CInv::GetCommand
    • f0f6123 fixup Witness commitment validation: function UpdateUncommitedBlockStructures redefine
    • a613599 [qa] Use integer division, byte strings properly
    • 5d6b6e2 fixup Add segregated witness transaction serialization: add missing witness flags
  • fixups 2 (sipa/bitcoin@77c613c...f38671f)
    • f889bec fixup Add segregated witness transaction serialization: add negative flag
    • 3ddabe4 fixup Add segregated witness transaction serialization: test with no inputs no longer possible
    • c744a1e fixup Add segregated witness transaction serialization: deal with overwrite and inconsistencies
    • 4709595 fixup Witness commitment validation: revert adding of commitment in IncrementExtraNonce
    • 122ca81 fixup Witness commitment validation: correct comment
    • 5df3e51 fixup Witness commitment validation: add comment about witness checking in miner
    • 36e1656 fixup Witness commitment validation: better validation error message
    • f726402 fixup Implement block size/sigop cost rules, limits, and GBT support: update comment
    • cf2c531 fixup Enable SCRIPT_VERIFY_WITNESS for mempool transactions: do not reject/punish invalid witness orphans
    • 0fb6e4e fixup Return witness data: don't list nextblockhash twice
    • ccd0e6c [qa] mininode: Use hexlify wrapper from util
    • 3a62cb9 Implement RecursiveDynamicUsage for witness structures
    • a541f0b Use an enum for signature versions
  • fixups 3 (sipa/bitcoin@f38671f...8708de8)
    • ba7e292 test: WITNESS flag must be used with P2SH flag
    • e7821e9 BIP9 parameters for testnet
    • 76142af Segwit script error unit tests
  • fixups 4 (sipa/bitcoin@8708de8...306858f)
  • fixups 5 (sipa/bitcoin@306858f...869f26e)
    • 036fa47 Improve FindForkInGlobalIndex when locator contains chain tip
    • 7eb0d75 VerifyDB: don't check blocks that have been pruned
    • 8adb03a Improve RewindBlockIndex when pruning
    • a1d1d0c Make sure upgraded nodes don't ask for non-wit blocks
    • 019860e script_tests: witness tests can specify tx amount
    • c815c16 [Qt] Add support for NODE_WITNESS in formatServicesStr
    • f16067f bitcoinconsensus.h: Accept amount as int64_t
    • 059d4d1 Add GetTransactionSigOpCost unit tests
    • c1c38a2 segwit: fix gui wallet send transaction size calculation assertion failed
    • 0acd1dc segwit: txout dust threshold calculation update
  • fixups 6 (sipa/bitcoin@869f26e...f98de5f)
    • 14d4d1d Extend the max witness program length to 40 bytes
    • 4840f6d Prevent witness addresses from being constructed before fork
    • 3dbf852 Remove positive SERIALIZE_TRANSACTION_WITNESS flag
  • fixups 7 (sipa/bitcoin@f98de5f...7613bbb)
    • c06c40b Actually count the witness data in memusage of CTransaction
    • d8b5db9 Correctly count maximum size in mining
    • 57d4bd2 Rework -maxblocksize and -maxblockcost
    • 0bfbf60 Cache transaction cost in mempool
    • 496d8c0 Delete segnet
    • 7799a7c Do not send witnesses in response to bip37 blocks
    • 4c19c18 33 to 40 bytes push should now be considered a witness scriptPubKey
  • fixups 8 (sipa/bitcoin@7613bbb...1b9893f)
    • a9bff09 Tests: add getblocktemplate/segwit test
    • 92ab64c Add test for getrawtransaction
    • 40f7829 p2p-segwit.py: more RPC coverage
    • b644339 Rename deployment (witness -> segwit)
    • 00bf5e1 Update p2p-segwit.py for new deployment name
    • c8f2fb2 BIP143 P2WSH examples
    • f3a7ed4 Fix unused variable in sigopcount test
  • fixups 9 (sipa/bitcoin@1b9893f...b508f5b)
    • 3483e5c DEPLOYMENT_WITNESS -> DEPLOYMENT_SEGWIT
    • 2a6516f Behave as a non-witness node when start time is far away
    • c7b5de5 test: BIP143 examples fix and clarify
    • cc19adc Remove segnet from mininode
    • efc251d Tests: ensure that signrawtransaction failures are caught in segwit.py
    • 396f4b8 spelling fix: uncommited -> uncommitted
  • fixups 10 (sipa/bitcoin@b508f5b...a6840e5)
  • fixups 11 (sipa/bitcoin@d7fe873...fb348c6)
    • 0e177a2 Don't treat NODE_WITNESS as relevant before a fork is defined
    • c7795ee Revert "Don't check the genesis block PoW" as segnet has been dropped.
  • merge (sipa/bitcoin@fb348c6...e847337)
    • e847337 Merge remote-tracking branch 'upstream/master' into segwit-master

Code to generate this list:

PREV="$(git rev-parse HEAD)"; (git log --oneline upstream/master..HEAD; echo "$(git rev-parse upstream/master) --- [SEGWIT] begin: preparation ---") | while read C L; do if [ "d${L:0:13}" == "d--- [SEGWIT] " ]; then if [ "d$PREV" != "" ]; then echo "* ${L:20:-4} (https://github.com/sipa/bitcoin/compare/$C...$PREV)"; fi; PREV=$C; PREVL=$L; else echo "  * $C $L"; fi; done | tac

sipa and others added some commits Dec 31, 2015

Add segregated witness transaction serialization
Contains refactorings by Eric Lombrozo.
Contains fixup by Nicolas Dorier.
Witness commitment validation
Includes a fix by Suhas Daftuar
Add signature version 1 with updated sighash
Includes simplifications by Eric Lombrozo.
Implement block size/sigop cost rules, limits, and GBT support
Includes changes by Suhas Daftuar and Luke-jr.
Return witness data
Includes RPC field name changes by Luke-jr.
Increase MAX_PROTOCOL_MESSAGE_LENGTH
Witness blocks can be greater than 2MiB, but cannot be validly greater
than 4MB.
bitcoinconsensus: add method that accepts amount, and return error wh…
…en verify_script receives VERIFY_WITNESS flag

script_tests: always test bitcoinconsensus_verify_script_with_amount if VERIFY_WITNESS isn't set

Rename internal method + make it static

trim bitcoinconsensus_ prefix

Add SERIALIZE_TRANSACTION_WITNESS flag

@sipa sipa force-pushed the sipa:segwit-master branch from a29e899 Jun 16, 2016

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Jun 16, 2016

A few more changes:

  • Spelling and naming consistency nits (by @sdaftuar and @instagibbs)
  • Test improvements (by @jl2012 and @sdaftuar)
  • Make a node fully behave as a non-witness node as long as there is no softfork defined

@sipa sipa force-pushed the sipa:segwit-master branch Jun 16, 2016

@sipa sipa force-pushed the sipa:segwit-master branch 2 times, most recently to e847337 Jun 16, 2016

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Jun 16, 2016

New merge on top of #7600.


{
LOCK(cs_main);
if (!IsWitnessEnabled(chainActive.Tip(), Params().GetConsensus()) && !GetBoolArg("-walletprematurewitness", false)) {

This comment has been minimized.

@instagibbs

instagibbs Jun 16, 2016

Member

Do we want to do a similar thing for createwitnessaddress?

{
CScript witnessscript(result[0].begin(), result[0].end());
txnouttype subType;
solved = solved && SignStep(creator, witnessscript, result, subType, SIGVERSION_WITNESS_V0) && subType != TX_SCRIPTHASH && subType != TX_WITNESS_V0_SCRIPTHASH && subType != TX_WITNESS_V0_KEYHASH;

This comment has been minimized.

@dcousens

dcousens Jun 17, 2016

Contributor

Rather than reassign solved, maybe just return false?
Why are we waiting until the end of the scope?

This comment has been minimized.

@sipa

sipa Jun 17, 2016

Author Member

We want to push the potentially partial result to sigdata.scriptSig, which is done in line 179-182.

@sdaftuar

This comment has been minimized.

Copy link
Member

sdaftuar commented Jun 17, 2016

ACK e847337

(Will ACK in #8149 as well)

// * witness (when witness enabled in flags and excludes coinbase)
nSigOpsCost += GetTransactionSigOpCost(tx, view, flags);
if (nSigOpsCost > MAX_BLOCK_SIGOPS_COST)
return state.DoS(100, error("ConnectBlock(): too many sigops"),

This comment has been minimized.

@instagibbs

instagibbs Jun 18, 2016

Member

Upgraded nodes don't ask for blocks from non-upgraded?

This comment has been minimized.

@kazcw

kazcw Jun 18, 2016

Contributor

Oh, right. Nevermind..

@sipa sipa force-pushed the sipa:segwit-master branch from e847337 to c4e3c75 Jun 22, 2016

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Jun 22, 2016

New merge after #8068 and #8179. See the details in #8149 (comment).

@NicolasDorier

This comment has been minimized.

Copy link
Member

NicolasDorier commented Jun 23, 2016

I udpated the sig cache PR (sipa#101)

@sipa

This comment has been minimized.

Copy link
Member Author

sipa commented Jun 24, 2016

Merged as #8149.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment