Segwit stands for "segregated witness" and is a backward-compatible upgrade or "soft fork" that activated on the Bitcoin network in August 2017. While the activation was controversial, the features of this technology require some explanation. In this chapter, we’ll explore how Segwit works, why it’s backward compatible, and what Segwit enables.
As a brief overview, Segwit incorporated a multitude of changes:
-
Block size increase
-
Transaction malleability fix
-
Segwit versioning for clear upgrade paths
-
Quadratic hashing fix
-
Offline wallet fee calculation security
It’s not entirely obvious what Segwit is without looking at how it’s implemented. We’ll start by examining the most basic type of Segwit transaction, pay-to-witness-pubkey-hash.
Pay-to-witness-pubkey-hash (p2wpkh) is one of four types of scripts defined by Segwit in BIP0141 and BIP0143. This is a smart contract that acts a lot like pay-to-pubkey-hash and is named similarly for that reason. The main change from p2pkh is that the data for the ScriptSig is now in the witness field. The rearrangement is to fix transaction malleability.
Transaction malleability is the ability to change the transaction’s ID without altering the transaction’s meaning. Mt. Gox CEO Mark Karpeles cited transaction malleability as the reason why his exchange was not allowing withdrawals back in 2013.
Malleability of the ID is an important consideration when creating payment channels, which are the atomic unit of the Lightning Network. A malleable transaction ID makes the safe creation of payment channel transactions much more difficult.
The reason why transaction malleability is a problem at all is because the transaction ID is calculated from the entire transaction. The ID of the transaction is the hash256 of the transaction. Most of the fields in a transaction cannot be changed without invalidating the transaction’s signature (and thus the transaction itself), so from a malleability standpoint, these fields are not a problem.
The one field that does allow for some manipulation without invalidating the signature is the ScriptSig field on each input. The ScriptSig is emptied before creating the signature hash (see [chapter_tx]), so it’s possible to change the ScriptSig without invalidating the signature. Also, as we learned in [chapter_elliptic_curve_cryptography], signatures contain a random component. This means that two different ScriptSigs can essentially mean the same thing but be different byte-wise.
This makes the ScriptSig field malleable—that is, able to be changed without changing the meaning—and means that the entire transaction, and the transaction ID, are malleable. A malleable transaction ID means that any dependent transactions (that is, any transaction spending one of the malleable transaction’s outputs) cannot be constructed in such a way as to guarantee validity. The previous transaction hash is uncertain, so the dependent transaction’s transaction input field cannot be guaranteed to be valid.
This is not usually a problem as once a transaction enters the blockchain, the transaction ID is fixed and no longer malleable (at least without finding a proof-of-work!). However, with payment channels, there are dependent transactions created before the funding transaction is added to the blockchain.
Transaction malleability is fixed by emptying the ScriptSig field and putting the data in another field that’s not used for ID calculation. For p2wpkh, the signature and pubkey are the items from ScriptSig, so those get moved to the witness field, which is not used for ID calculation. This way, the transaction ID stays stable as the malleability vector disappears. The witness field, and the whole Segwit serialization of a transaction, is only sent to nodes that ask for it. In other words, old nodes that haven’t upgraded to Segwit don’t receive the witness field and don’t verify the pubkey and signature.
If this sounds familiar, it should. This is similar to how p2sh works ([chapter_p2sh]) in that newer nodes do additional validation that older nodes do not, and is the basis for why Segwit is a soft fork (backward-compatible upgrade) and not a hard fork (backward-incompatible upgrade).
To understand Segwit, it helps to look at what a transaction looks like when sent to an old node (Pay-to-witness-pubkey-hash (p2wpkh) as seen by pre-BIP0141 software) versus a new node (Pay-to-witness-pubkey-hash (p2wpkh) as seen by post-BIP0141 software).
The difference between these two serializations is that the latter transaction (Segwit serialization) has the marker, flag, and witness fields. Otherwise, the two transactions look similar. The reason the transaction ID is not malleable is because the first serialization is used for calculating the transaction ID.
The witness field in p2wpkh has the signature and pubkey as its two elements. These will be used for validation for upgraded nodes only.
The ScriptPubKey for p2wpkh is OP_0 <20-byte hash>.
The ScriptSig, as seen in both serializations, is empty.
The combined script is shown in Pay-to-witness-pubkey-hash (p2wpkh) ScriptPubKey.
The processing of the combined script starts like p2wpkh start.
OP_0 pushes a 0 to the stack (p2wpkh step 1).
The 20-byte hash is an element, so it’s pushed to the stack (p2wpkh step 2).
At this point, older nodes will stop as there are no more Script commands to be processed.
Since the top element is nonzero, this will be counted as a valid script.
This is very similar to p2sh ([chapter_p2sh]) in that older nodes cannot validate further.
Newer nodes, however, have a special Segwit rule much like the special rule for p2sh (see [chapter_p2sh]).
Recall that with p2sh, the exact script sequence of <RedeemScript> OP_HASH160 <hash> OP_EQUAL triggers a special rule.
In the case of p2wpkh, the script sequence is OP_0 <20-byte hash>.
When that script sequence is encountered, the pubkey and signature from the witness field and the 20-byte hash are added to the command set in exactly the same sequence as with p2pkh, namely <signature> <pubkey> OP_DUP OP_HASH160 <20-byte hash> OP_EQUALVERIFY OP_CHECKSIG.
p2wpkh step 3 shows the state that is encountered next.
The rest of the processing of p2wpkh is the same as the processing of p2pkh, as seen in [chapter_script]. The end state is a single 1 on the stack if and only if the 20-byte hash is the hash160 of the pubkey and the signature is valid (p2wpkh step 4).
For an older node, processing stops at <20-byte hash> 0, as older nodes don’t know the special Segwit rule. Only upgraded nodes do the rest of the validation, much like with p2sh. Note that less data is sent over the network to older nodes. Also, nodes are given the option of not having to download (and hence not verify) transactions that are x blocks old if they don’t want to. In a sense, the signature has been witnessed by a bunch of people and a node can choose to trust that this is valid instead of validating directly if it so chooses.
Note also that this is a special rule for Segwit version 0.
Segwit version 1 can have a completely different processing path.
<20-byte hash> 1 could be the special script sequence that triggers a different rule.
Upgrades of Segwit can introduce Schnorr signatures, Graftroot, or even a different scripting system altogether, like Simplicity.
Segwit gives us a clear upgrade path.
Software that understands how to validate Segwit version X will validate such transactions, but software that isn’t aware of Segwit version X simply processes only up to the point of the special rule.
p2wpkh is great, but unfortunately, this is a new type of script and older wallets cannot send bitcoins to p2wpkh ScriptPubKeys. p2wpkh uses a new address format called Bech32, defined in BIP0173, whose ScriptPubKeys older wallets don’t know how to create.
The Segwit authors found an ingenious way to make Segwit backward compatible by "wrapping" p2wpkh inside p2sh. This is called "nested" Segwit as the Segwit script is nested in a p2sh RedeemScript.
A p2sh-p2wpkh address is a normal p2sh address, but the RedeemScript is OP_0 <20-byte hash>, or the ScriptPubKey of p2wpkh.
As with p2wpkh, different transactions are sent to older nodes (Pay-to-script-hash-pay-to-witness-pubkey-hash (p2sh-p2wpkh) to pre-BIP0141 software) versus newer nodes (p2sh-p2wpkh to post-BIP0141 software).
The difference from p2wpkh is that the ScriptSig is no longer empty. The ScriptSig has a RedeemScript, which is equal to the ScriptPubkey in p2wpkh. As this is a p2sh script, the ScriptPubKey is the same as in any other p2sh script. The combined script looks like p2sh-p2wpkh ScriptPubKey is the same as a normal p2sh ScriptPubKey.
We start the script evaluation like in p2sh-p2wpkh start.
Notice that the commands to be processed are exactly what triggers the p2sh special rule. The RedeemScript goes on the stack (p2sh-p2wpkh step 1).
The OP_HASH160 will return the RedeemScript’s hash (p2sh-p2wpkh step 2).
The hash will go on the stack, and we then get to OP_EQUAL (p2sh-p2wpkh step 3).
At this point, if the hashes are equal, pre-BIP0016 nodes will simply mark the input as valid, as they are unaware of the p2sh validation rules.
However, post-BIP0016 nodes recognize the special script sequence for p2sh, so the RedeemScript will then be evaluated as Script commands.
The RedeemScript is OP_0 <20-byte hash>, which is the same as the ScriptPubKey for p2wpkh.
This makes the script state look like p2sh-p2wpkh step 4.
This should look familiar, as this is the state that p2wpkh starts with.
After OP_0 and the 20-byte hash we are left with p2sh-p2wpkh step 5.
At this point, pre-Segwit nodes will mark this input as valid as they are unaware of the Segwit validation rules. However, post-Segwit nodes will recognize the special script sequence for p2wpkh. The signature and pubkey from the witness field along with the 20-byte hash will add the p2pkh commands (p2sh-p2wpkh step 6).
The rest of the processing is the same as p2pkh ([chapter_script]). Assuming the signature and pubkey are valid, we are left with p2sh-p2wpkh end.
As you can see, a p2sh-p2wpkh transaction is backward compatible all the way to before BIP0016. A pre-BIP0016 node would consider the script valid once the RedeemScripts were equal, and a post-BIP0016, pre-Segwit node would consider the script valid at the 20-byte hash. Both would not do the full validation and would accept the transaction. A post-Segwit node would do the complete validation, including checking the signature and pubkey.
|
Note
|
Can Anyone Spend Segwit Outputs?
Detractors of Segwit have referred to Segwit outputs as "anyone-can-spend." This would have been true if the Bitcoin community had rejected Segwit. In other words, if an economically significant part of the Bitcoin community had refused to do the Segwit validation and actively split from the network by accepting transactions that were not Segwit-valid, the outputs would have been anyone-can-spend. However, due to a variety of economic incentives, Segwit was activated on the network, there was no network split, a lot of bitcoins are now locked in Segwit outputs, and Segwit transactions are validated per the soft-fork rules by the vast economic majority of nodes. We can now say confidently that the detractors were wrong. |
The first change we’re going to make is to the Tx class, where we need to mark whether the transaction is Segwit or not:
link:code-ch13/tx.py[role=include]Next, we change the parse method depending on the serialization we receive:
class Tx:
...
link:code-ch13/tx.py[role=include]-
To determine whether we have a Segwit transaction or not, we look at the fifth byte. The first four are the version, the fifth is the Segwit marker.
-
The fifth byte being 0 is how we tell that this transaction is Segwit (this is not foolproof, but it’s what we’re going to use). We use different parsers depending on whether it’s Segwit.
-
We put the stream back to the position before we examined the first 5 bytes.
-
We’ve moved the old
parsemethod toparse_legacy.
Here’s a parser for the Segwit serialization:
class Tx:
...
link:code-ch13/tx.py[role=include]-
There are two new fields; one of them is the Segwit marker.
-
The second new field is witness, which contains items for each input.
We now code the corresponding changes to the serialization methods:
class Tx:
...
link:code-ch13/tx.py[role=include]-
What used to be called
serializeis nowserialize_legacy. -
The Segwit serialization adds the markers.
-
The witness is serialized at the end.
We also have to change the hash method to use the legacy serialization, even for Segwit transactions, as that will keep the transaction ID stable:
class Tx:
...
link:code-ch13/tx.py[role=include]The verify_input method requires a different z for Segwit transactions.
The Segwit transaction signature hash calculation is specified in BIP0143.
In addition, the witness field is passed through to the script evaluation engine:
class Tx:
...
def verify_input(self, input_index):
tx_in = self.tx_ins[input_index]
script_pubkey = tx_in.script_pubkey(testnet=self.testnet)
if script_pubkey.is_p2sh_script_pubkey():
command = tx_in.script_sig.commands[-1]
raw_redeem = int_to_little_endian(len(command), 1) + command
redeem_script = Script.parse(BytesIO(raw_redeem))
if redeem_script.is_p2wpkh_script_pubkey(): # (1)
z = self.sig_hash_bip143(input_index, redeem_script) # (2)
witness = tx_in.witness
else:
z = self.sig_hash(input_index, redeem_script)
witness = None
else:
if script_pubkey.is_p2wpkh_script_pubkey(): # (3)
z = self.sig_hash_bip143(input_index) # (2)
witness = tx_in.witness
else:
z = self.sig_hash(input_index)
witness = None
combined_script = tx_in.script_sig + tx_in.script_pubkey(self.testnet)
return combined_script.evaluate(z, witness) # (4)-
This handles the p2sh-p2wpkh case.
-
The BIP0143 signature hash generation code is in tx.py of this chapter’s code.
-
This handles the p2wpkh case.
-
The witness passes through to the evaluation engine so that p2wpkh can construct the right commands.
We also define what a p2wpkh script looks like in script.py:
link:code-ch13/script.py[role=include]
...
link:code-ch13/script.py[role=include]-
This is OP_0 <20-byte-hash>.
-
This checks if the current script is a p2wpkh ScriptPubKey.
Last, we need to implement the special rule in the evaluate method.
class Script:
...
def evaluate(self, z, witness):
...
while len(commands) > 0:
...
else:
stack.append(command)
...
link:code-ch13/script.py[role=include]-
This is where we execute witness program version 0 for p2wpkh. We make a p2pkh combined script from the 20-byte hash, signature, and pubkey and evaluate.
While p2wpkh takes care of a major use case, we need something more flexible if we want more complicated (e.g., multisig) scripts. This is where pay-to-witness-script-hash (p2wsh) comes in. p2wsh is like p2sh, but with all the ScriptSig data in the witness field instead.
As with p2wpkh, we send different data to pre-BIP0141 software (Pay-to-witness-script-hash as seen by pre-BIP0141 software) versus post-BIP0141 software (Pay-to-witness-script-hash as seen by post-BIP0141 software).
The ScriptPubKey for p2wsh script is OP_0 <32-byte hash>.
This sequence triggers another special rule.
The ScriptSig, as with p2wpkh, is empty.
When p2wsh outputs are being spent, the combined script looks like Pay-to-witness-script-hash (p2wsh) ScriptPubKey.
The processing of this script starts similarly to p2wpkh (Figures #p2wsh_start and #p2wsh_step_1).
The 32-byte hash is an element, so it is pushed to the stack (p2wsh step 2).
As with p2wpkh, older nodes will stop here because there are no more Script commands to be processed. Newer nodes will recognize the special sequence and do additional validation by looking at the witness field.
The witness field for p2wsh in our case is a 2-of-3 multisig (p2wsh witness).
The last item of the witness is called the WitnessScript and must sha256 to the 32-byte hash from the ScriptPubKey. Note this is sha256, not hash256. Once the WitnessScript is validated by having the same hash value, it is interpreted as script commands and put into the command set. The WitnessScript looks like p2wsh WitnessScript.
The rest of the witness field is put on top to produce the command set in p2wsh step 3.
As you can see, this is a 2-of-3 multisig much like what was explored in [chapter_p2sh] (p2wsh step 4).
If the signatures are valid, we end like p2wsh step 5.
The WitnessScript is very similar to the RedeemScript in that the sha256 of the serialization is addressed in the ScriptPubKey, but only revealed when the output is being spent. Once the sha256 of the WitnessScript is found to be the same as the 32-byte hash, the WitnessScript is interpreted as script commands and added to the command set. The rest of the witness field is then added to the command set as well, producing the final set of commands to be evaluated. p2wsh is particularly important, as unmalleable multisig transactions are required for creating bidirectional payment channels for the Lightning Network.
Like p2sh-p2wpkh, p2sh-p2wsh is a way to make p2wsh backward compatible. Again, different transactions are sent to older nodes (Pay-to-script-hash-pay-to-witness-script-hash (p2sh-p2wsh) to pre-BIP0141 software) versus newer nodes (p2sh-p2wsh to post-BIP0141 software).
As with p2sh-p2wpkh, the ScriptPubKey is indistinguishable from any other p2sh address and the ScriptSig is only the RedeemScript (p2sh-p2wsh ScriptPubKey).
We start the p2sh-p2wsh script evaluation in exactly the same way that p2sh-p2wpkh script evaluation starts (p2sh-p2wsh start).
The RedeemScript is pushed to the stack (p2sh-p2wsh step 1).
The OP_HASH160 will return the RedeemScript’s hash (p2sh-p2wsh step 2).
The hash is pushed to the stack, and we then get to OP_EQUAL (p2sh-p2wsh step 3).
As with p2sh-p2wpkh, if the hashes are equal, pre-BIP0016 nodes will mark the input as valid as they are unaware of the p2sh validation rules.
However, post-BIP0016 nodes will recognize the special script sequence for p2sh, so the RedeemScript will be interpreted as new script commands.
The RedeemScript is OP_0 32-byte hash, which is the same as the ScriptPubKey for p2wsh (p2sh-p2wsh RedeemScript).
This makes the script state look like p2sh-p2wsh step 4.
Of course, this is the exact same starting state as for p2wsh (p2sh-p2wsh step 5).
The 32-byte hash is an element, so it is pushed to the stack (p2sh-p2wsh step 6).
At this point, pre-Segwit nodes will mark this input as valid, as they are unaware of the Segwit validation rules. However, post-Segwit nodes will recognize the special script sequence for p2wsh. The witness field (p2sh-p2wsh witness) contains the WitnessScript (p2sh-p2wsh WitnessScript). The sha256 of the WitnessScript is checked against the 32-byte hash, and if they’re equal the WitnessScript is interpreted as script commands and put into the command set (p2sh-p2wsh step 7).
As you can see, this is a 2-of-3 multisig as in [chapter_p2sh]. If the signatures are valid, we end like p2sh-p2wsh end.
This makes p2wsh backward compatible, allowing older wallets to send to p2sh ScriptPubKeys that they can handle.
The parsing and serialization are exactly the same as before.
The main changes have to do with verify_input in tx.py and evaluate in script.py:
class Tx:
...
def verify_input(self, input_index):
tx_in = self.tx_ins[input_index]
script_pubkey = tx_in.script_pubkey(testnet=self.testnet)
if script_pubkey.is_p2sh_script_pubkey():
command = tx_in.script_sig.commands[-1]
raw_redeem = int_to_little_endian(len(command), 1) + command
redeem_script = Script.parse(BytesIO(raw_redeem))
if redeem_script.is_p2wpkh_script_pubkey():
z = self.sig_hash_bip143(input_index, redeem_script)
witness = tx_in.witness
elif redeem_script.is_p2wsh_script_pubkey(): # (1)
command = tx_in.witness[-1]
raw_witness = encode_varint(len(command)) + command
witness_script = Script.parse(BytesIO(raw_witness))
z = self.sig_hash_bip143(input_index,
witness_script=witness_script)
witness = tx_in.witness
else:
z = self.sig_hash(input_index, redeem_script)
witness = None
else:
if script_pubkey.is_p2wpkh_script_pubkey():
z = self.sig_hash_bip143(input_index)
witness = tx_in.witness
elif script_pubkey.is_p2wsh_script_pubkey(): # (2)
command = tx_in.witness[-1]
raw_witness = encode_varint(len(command)) + command
witness_script = Script.parse(BytesIO(raw_witness))
z = self.sig_hash_bip143(input_index,
witness_script=witness_script)
witness = tx_in.witness
else:
z = self.sig_hash(input_index)
witness = None
combined_script = tx_in.script_sig + tx_in.script_pubkey(self.testnet)
return combined_script.evaluate(z, witness)-
This takes care of p2sh-p2wsh.
-
This takes care of p2wsh.
Next, we code a way to identify p2wsh in script.py:
link:code-ch13/script.py[role=include]
...
class Script:
...
link:code-ch13/script.py[role=include]-
OP_0<32-byte script> is what we expect.
Last, we handle the special rule for p2wsh:
class Script:
...
def evaluate(self, z, witness):
...
while len(commands) > 0:
...
else:
stack.append(command)
...
link:code-ch13/script.py[role=include]-
The top element is the sha256 hash of the WitnessScript.
-
The second element is the witness version, 0.
-
Everything but the WitnessScript is added to the command set.
-
The WitnessScript is the last item of the witness field.
-
The WitnessScript must hash to the sha256 that was in the stack.
-
We parse the WitnessScript and add it to the command set.
Segwit also fixes the quadratic hashing problem through a different calculation of the signature hash. A lot of the calculations for the signature hash, z, can be reused instead of requiring a new hash256 hash for each input. The signature hash calculation is detailed in BIP0143 and can be seen in code-ch13/tx.py.
Another improvement is that by policy, uncompressed SEC pubkeys are now forbidden; only compressed SEC pubkeys are used for Segwit, saving space.
The chapter covered the details of Segwit as a taste of what’s now possible. [chapter_advanced] discusses the next steps that you can take on your Bitcoin developer journey.