Segregated witness #7910

sipa · 2016-04-19T15:22:33Z

This is the complete code for segregated witness on top of master (implementing BIP141, BIP143, BIP144, BIP145). Apart from commentary and merging (see further), the code is never rewritten/squashed/rebased, in order not to invalidate review. A rebased/squashed version is available here: #8149.

See also the detailed and up-to-date list of commits here: #7910 (comment)

There are certainly some items left to do:

Incorporate changes to BIP145/BIP9 when finalized (see BIP 141 and 145: GBT updates for SegWit bips#365)
Define a deployment time for testnet
Seed nodes that provide segwit-capable peers
Do tests in a mixed network of upgraded and non-upgraded nodes
Remove segnet from mainline merge
Define a deployment time for mainnet
Per-transaction caching of sighashes to fix the O(n^2) sighash problem (see Cache hashes sipa/bitcoin#70)

Many thanks to all contributors so far (including @jl2012, @NicolasDorier, @CodeShark, @sdaftuar, @morcos, @luke-jr, @afk11, @theuni, @TheBlueMatt, @MarcoFalke, @LongShao007, @jonasnick, @mruddy), the ideas it is based on (by @gmaxwell and @luke-jr), and many reviewers (including @SergioDemianLerner, @instagibbs, @dcousens, and @btcdrak).

Inspired by Jorge Timon's "Testchain: Don't check the genesis block", but restricted to just PoW check. Code by Matt Corallo, originally submitted as a patch to segwit.

sipa · 2016-04-19T16:09:32Z

Since github shows the commits sorted by author date rather than dependency order, here is a list of all commits:

preparation (sipa/bitcoin@fb0ac48...e69265d)
- 644b2c5 Keep addrman's nService bits consistent with outbound observations
- afdc413 Verify that outbound connections have expected services
- fdec2bc Only store and connect to NODE_NETWORK nodes
segnet (sipa/bitcoin@e69265d...bb4bb47)
- 66dbd81 Don't check the genesis block PoW
- 70ebe86 Create segnet4
- f8bcd86 Add segnet seed nodes
- 58847fe qt: Work (don't crash) with -segnet
P2P/node/consensus (sipa/bitcoin@bb4bb47...351a2fe)
- 3b1ff49 Add segregated witness transaction serialization
- e35d020 Removed ppszTypeName from protocol.cpp
- adb1c09 getdata enum issue fix
- ac6886d Introduce and preferentially peer with NODE_WITNESS service bit
- cb9d4d3 Witness commitment validation
- 2c87be1 Script validation logic for witnesses
- 1e47805 Enable SCRIPT_VERIFY_WITNESS for mempool transactions
- 97f34f7 Activate script consensus rules through BIP9
- 6a8021f Only download blocks from witness peers after fork
- 271e45e Observe input amounts: verification
- d374139 Add signature version 1 with updated sighash
- 92c6dc3 Return witness data
- 3955218 Implement block size/sigop cost rules, limits, and GBT support
- 5a47b98 Add command line options to loosen mempool acceptance rules
- e573571 bitcoinconsensus: add method that accepts amount, and return error when verify_script receives VERIFY_WITNESS flag
- a6386c2 Increase MAX_PROTOCOL_MESSAGE_LENGTH
- 1b6c6f1 Add rewind logic to deal with post-fork software updates
wallet (sipa/bitcoin@351a2fe...a2d6f8a)
- abbe085 Witness script signing
- c3fe53f Add witness address RPCs (using P2SH)
- ec385fa signrawtransaction can sign P2WSH
tests (sipa/bitcoin@a2d6f8a...cfb0a83)
- 2ed1d11 Signing tests
- 3c0b16f Add rpc test for segwit
- 24746d2 Add transaction tests for segwit
- 3409ad6 Add segwit support to script_tests
- e5ffcc1 Autogeneration support for witness in script_tests
- 5e6abe5 Update p2p test framework with segwit support
- 7b539f9 P2P test for segwit
fixups (sipa/bitcoin@cfb0a83...77c613c)
- bb2bfca fixup Implement block size/sigop cost rules, limits, and GBT support: use int64_t for sigopcost everywhere
- 64c1527 fixup Implement block size/sigop cost rules, limits, and GBT support: update block cost comment
- cc741f5 fixup Add segregated witness transaction serialization: comments
- c28179f fixup Update p2p test framework with segwit support: fix comment
- 6a5da9b fixup Witness commitment validation: list the mentioned BIPs
- 88e0135 fixup Add command line options to loosen mempool acceptance rules: simplify comment
- da60fae fixup Witness commitment validation: factor out GetWitnessCommitmentPos
- 4f827be fixup Witness commitment validation: fix typo in comment
- 058f495 fixup Implement block size/sigop cost rules, limits, and GBT support: improve GBT help
- c1e7a95 fixup Add witness address RPCs (using P2SH): test hex & wallet available
- dbe6391 fixup Add segregated witness transaction serialization: revert extformat-if-empty-vin
- 97d7402 tidy up CInv::GetCommand
- f0f6123 fixup Witness commitment validation: function UpdateUncommitedBlockStructures redefine
- a613599 [qa] Use integer division, byte strings properly
- 5d6b6e2 fixup Add segregated witness transaction serialization: add missing witness flags
fixups 2 (sipa/bitcoin@77c613c...f38671f)
- f889bec fixup Add segregated witness transaction serialization: add negative flag
- 3ddabe4 fixup Add segregated witness transaction serialization: test with no inputs no longer possible
- c744a1e fixup Add segregated witness transaction serialization: deal with overwrite and inconsistencies
- 4709595 fixup Witness commitment validation: revert adding of commitment in IncrementExtraNonce
- 122ca81 fixup Witness commitment validation: correct comment
- 5df3e51 fixup Witness commitment validation: add comment about witness checking in miner
- 36e1656 fixup Witness commitment validation: better validation error message
- f726402 fixup Implement block size/sigop cost rules, limits, and GBT support: update comment
- cf2c531 fixup Enable SCRIPT_VERIFY_WITNESS for mempool transactions: do not reject/punish invalid witness orphans
- 0fb6e4e fixup Return witness data: don't list nextblockhash twice
- ccd0e6c [qa] mininode: Use hexlify wrapper from util
- 3a62cb9 Implement RecursiveDynamicUsage for witness structures
- a541f0b Use an enum for signature versions
fixups 3 (sipa/bitcoin@f38671f...8708de8)
- ba7e292 test: WITNESS flag must be used with P2SH flag
- e7821e9 BIP9 parameters for testnet
- 76142af Segwit script error unit tests
fixups 4 (sipa/bitcoin@8708de8...306858f)
- b846298 f cb9d4d3: typo fix
- 38e3fcd f 3b1ff49: remove redundant witness clean
- fd85b77 f 1b6c6f1: Cleanup mapBlocksUnlinked in RewindBlockIndex
- 433daf4 f ac6886d: Get rid of leftover WITNESS_VERSION
- 9590289 f 2c87be1: fix typo
- 17277c9 f 3955218: GetBlockCost: Clarify comment description
- 1e9cba2 f 3955218: provide both -maxblocksize and -maxblockcost
- 8f50b96 [qa] segwit: Switch to py3
- 75335ca f 1b6c6f1: pass CChainParams rather than Consensus::Params to RewindBlockIndex
fixups 5 (sipa/bitcoin@306858f...869f26e)
- 036fa47 Improve FindForkInGlobalIndex when locator contains chain tip
- 7eb0d75 VerifyDB: don't check blocks that have been pruned
- 8adb03a Improve RewindBlockIndex when pruning
- a1d1d0c Make sure upgraded nodes don't ask for non-wit blocks
- 019860e script_tests: witness tests can specify tx amount
- c815c16 [Qt] Add support for NODE_WITNESS in formatServicesStr
- f16067f bitcoinconsensus.h: Accept amount as int64_t
- 059d4d1 Add GetTransactionSigOpCost unit tests
- c1c38a2 segwit: fix gui wallet send transaction size calculation assertion failed
- 0acd1dc segwit: txout dust threshold calculation update
fixups 6 (sipa/bitcoin@869f26e...f98de5f)
- 14d4d1d Extend the max witness program length to 40 bytes
- 4840f6d Prevent witness addresses from being constructed before fork
- 3dbf852 Remove positive SERIALIZE_TRANSACTION_WITNESS flag
fixups 7 (sipa/bitcoin@f98de5f...7613bbb)
- c06c40b Actually count the witness data in memusage of CTransaction
- d8b5db9 Correctly count maximum size in mining
- 57d4bd2 Rework -maxblocksize and -maxblockcost
- 0bfbf60 Cache transaction cost in mempool
- 496d8c0 Delete segnet
- 7799a7c Do not send witnesses in response to bip37 blocks
- 4c19c18 33 to 40 bytes push should now be considered a witness scriptPubKey
fixups 8 (sipa/bitcoin@7613bbb...1b9893f)
- a9bff09 Tests: add getblocktemplate/segwit test
- 92ab64c Add test for getrawtransaction
- 40f7829 p2p-segwit.py: more RPC coverage
- b644339 Rename deployment (witness -> segwit)
- 00bf5e1 Update p2p-segwit.py for new deployment name
- c8f2fb2 BIP143 P2WSH examples
- f3a7ed4 Fix unused variable in sigopcount test
fixups 9 (sipa/bitcoin@1b9893f...b508f5b)
- 3483e5c DEPLOYMENT_WITNESS -> DEPLOYMENT_SEGWIT
- 2a6516f Behave as a non-witness node when start time is far away
- c7b5de5 test: BIP143 examples fix and clarify
- cc19adc Remove segnet from mininode
- efc251d Tests: ensure that signrawtransaction failures are caught in segwit.py
- 396f4b8 spelling fix: uncommited -> uncommitted
fixups 10 (sipa/bitcoin@b508f5b...a6840e5)
- 10433cf Fix direct fetching of blocks after 2a6516f
- 2948c02 Consistency in serialization flags
fixups 11 (sipa/bitcoin@d7fe873...fb348c6)
- 0e177a2 Don't treat NODE_WITNESS as relevant before a fork is defined
- c7795ee Revert "Don't check the genesis block PoW" as segnet has been dropped.
merge (sipa/bitcoin@fb348c6...e847337)
- e847337 Merge remote-tracking branch 'upstream/master' into segwit-master

Code to generate this list:

PREV="$(git rev-parse HEAD)"; (git log --oneline upstream/master..HEAD; echo "$(git rev-parse upstream/master) --- [SEGWIT] begin: preparation ---") | while read C L; do if [ "d${L:0:13}" == "d--- [SEGWIT] " ]; then if [ "d$PREV" != "" ]; then echo "* ${L:20:-4} (https://github.com/sipa/bitcoin/compare/$C...$PREV)"; fi; PREV=$C; PREVL=$L; else echo "  * $C $L"; fi; done | tac

Contains refactorings by Eric Lombrozo. Contains fixup by Nicolas Dorier.

Service bit logic by Nicolas Dorier.

Includes a fix by Suhas Daftuar

Includes simplifications by Eric Lombrozo.

Includes RPC field name changes by Luke-jr.

Includes changes by Suhas Daftuar and Luke-jr.

…en verify_script receives VERIFY_WITNESS flag script_tests: always test bitcoinconsensus_verify_script_with_amount if VERIFY_WITNESS isn't set Rename internal method + make it static trim bitcoinconsensus_ prefix Add SERIALIZE_TRANSACTION_WITNESS flag

Witness blocks can be greater than 2MiB, but cannot be validly greater than 4MB.

sipa · 2016-06-16T18:35:22Z

New merge on top of #7600.

instagibbs · 2016-06-16T20:31:23Z

src/wallet/rpcwallet.cpp

+
+    {
+        LOCK(cs_main);
+        if (!IsWitnessEnabled(chainActive.Tip(), Params().GetConsensus()) && !GetBoolArg("-walletprematurewitness", false)) {


Do we want to do a similar thing for createwitnessaddress?

sdaftuar · 2016-06-17T05:04:30Z

ACK e847337

(Will ACK in #8149 as well)

instagibbs · 2016-06-18T01:15:31Z

src/main.cpp

+        // * witness (when witness enabled in flags and excludes coinbase)
+        nSigOpsCost += GetTransactionSigOpCost(tx, view, flags);
+        if (nSigOpsCost > MAX_BLOCK_SIGOPS_COST)
+            return state.DoS(100, error("ConnectBlock(): too many sigops"),


Upgraded nodes don't ask for blocks from non-upgraded?

Oh, right. Nevermind..

sipa · 2016-06-22T14:13:16Z

New merge after #8068 and #8179. See the details in #8149 (comment).

NicolasDorier · 2016-06-23T20:20:44Z

I udpated the sig cache PR (sipa#101)

sipa · 2016-06-24T16:20:48Z

Merged as #8149.

ZIP 239 preparations 3 Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#8080 - bitcoin/bitcoin#8082 - bitcoin/bitcoin#8126 - bitcoin/bitcoin#7910 - This is the unsquashed version of bitcoin/bitcoin#8149 - We take three cleanup commits to the protocol / `CInv` code. - bitcoin/bitcoin#8822 - bitcoin/bitcoin#8880 - Excluding the first commit (we don't have the comment it fixes yet). - bitcoin/bitcoin#19322

sipa added 5 commits April 19, 2016 15:57

Keep addrman's nService bits consistent with outbound observations

644b2c5

Verify that outbound connections have expected services

afdc413

Only store and connect to NODE_NETWORK nodes

fdec2bc

--- [SEGWIT] begin: segnet ---

e69265d

Don't check the genesis block PoW

66dbd81

Inspired by Jorge Timon's "Testchain: Don't check the genesis block", but restricted to just PoW check. Code by Matt Corallo, originally submitted as a patch to segwit.

jonasschnelli mentioned this pull request Apr 19, 2016

[WIP] Segregated Witness #7404

Closed

sipa and others added 23 commits April 19, 2016 18:27

Create segnet4

70ebe86

Add segnet seed nodes

f8bcd86

qt: Work (don't crash) with -segnet

58847fe

--- [SEGWIT] begin: P2P/node/consensus ---

bb4bb47

Add segregated witness transaction serialization

3b1ff49

Contains refactorings by Eric Lombrozo. Contains fixup by Nicolas Dorier.

Removed ppszTypeName from protocol.cpp

e35d020

getdata enum issue fix

adb1c09

Introduce and preferentially peer with NODE_WITNESS service bit

ac6886d

Service bit logic by Nicolas Dorier.

Witness commitment validation

cb9d4d3

Includes a fix by Suhas Daftuar

Script validation logic for witnesses

2c87be1

Enable SCRIPT_VERIFY_WITNESS for mempool transactions

1e47805

Activate script consensus rules through BIP9

97f34f7

Only download blocks from witness peers after fork

6a8021f

Observe input amounts: verification

271e45e

Add signature version 1 with updated sighash

d374139

Includes simplifications by Eric Lombrozo.

Return witness data

92c6dc3

Includes RPC field name changes by Luke-jr.

Implement block size/sigop cost rules, limits, and GBT support

3955218

Includes changes by Suhas Daftuar and Luke-jr.

Add command line options to loosen mempool acceptance rules

5a47b98

Increase MAX_PROTOCOL_MESSAGE_LENGTH

a6386c2

Witness blocks can be greater than 2MiB, but cannot be validly greater than 4MB.

Add rewind logic to deal with post-fork software updates

1b6c6f1

--- [SEGWIT] begin: wallet ---

351a2fe

Witness script signing

abbe085

sipa added 3 commits June 16, 2016 14:21

--- [SEGWIT] begin: fixups 10 ---

b508f5b

Fix direct fetching of blocks after 2a6516f

10433cf

Consistency in serialization flags

2948c02

sipa force-pushed the segwit-master branch from c42631c to 9923c49 Compare June 16, 2016 12:28

sipa added 3 commits June 16, 2016 16:37

--- [SEGWIT] begin: fixups 11 ---

d7fe873

Don't treat NODE_WITNESS as relevant before a fork is defined

0e177a2

Revert "Don't check the genesis block PoW" as segnet has been dropped.

c7795ee

sipa force-pushed the segwit-master branch 2 times, most recently from 8e3310c to e847337 Compare June 16, 2016 18:30

instagibbs reviewed Jun 16, 2016
View reviewed changes

instagibbs reviewed Jun 18, 2016
View reviewed changes

sipa added 2 commits June 22, 2016 15:42

--- [SEGWIT] begin: merge ---

c708c3a

Merge remote-tracking branch 'upstream/master' into segwit-master

c4e3c75

sipa force-pushed the segwit-master branch from e847337 to c4e3c75 Compare June 22, 2016 14:02

sipa closed this Jun 24, 2016

maflcko mentioned this pull request Oct 5, 2016

[Doc] Update bips.md for Segregated Witness #8891

Merged

rem0g mentioned this pull request Mar 13, 2017

Feature request BitcoinUnlimited/BitcoinUnlimited#369

Closed

scravy mentioned this pull request Oct 26, 2018

Documentation of review process dtr-org/unit-e#186

Merged

sipa mentioned this pull request Aug 28, 2020

Implement BIP 340-342 validation (Schnorr/taproot/tapscript) #17977

Closed

4 tasks

sipa mentioned this pull request Sep 14, 2020

Implement BIP 340-342 validation (Schnorr/taproot/tapscript) #19953

Merged

str4d mentioned this pull request Aug 13, 2021

ZIP 239 preparations 3 zcash/zcash#5274

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segregated witness #7910

Segregated witness #7910

sipa commented Apr 19, 2016 •

edited

Loading

sipa commented Apr 19, 2016 •

edited

Loading

sipa commented Jun 16, 2016

instagibbs Jun 16, 2016

sdaftuar commented Jun 17, 2016

instagibbs Jun 18, 2016

kazcw Jun 18, 2016

sipa commented Jun 22, 2016

NicolasDorier commented Jun 23, 2016

sipa commented Jun 24, 2016

Segregated witness #7910

Segregated witness #7910

Conversation

sipa commented Apr 19, 2016 • edited Loading

sipa commented Apr 19, 2016 • edited Loading

sipa commented Jun 16, 2016

instagibbs Jun 16, 2016

Choose a reason for hiding this comment

sdaftuar commented Jun 17, 2016

instagibbs Jun 18, 2016

Choose a reason for hiding this comment

kazcw Jun 18, 2016

Choose a reason for hiding this comment

sipa commented Jun 22, 2016

NicolasDorier commented Jun 23, 2016

sipa commented Jun 24, 2016

sipa commented Apr 19, 2016 •

edited

Loading

sipa commented Apr 19, 2016 •

edited

Loading