multi: Enable vote for DCP0005.

[davecgh]

This requires PR #1904.

Testing Notes

As of this PR, the expected behavior is that there is a single migration that takes around 10 to 15 minutes to complete. after which it will no longer be possible to downgrade.

As the warning above notes, if you try to run an older software version after this migration has completed, you will get an error message similar to Unable to start server on [:9108]: the current blockchain database is no longer compatible with this version of the software (6 > 5).

---

This implements the agenda for voting on header commitments as defined in DCP0005 along with consensus tests to ensure its correctness. It consists of several commits that have been carefully crated to make the changes easier to reason about and review.

Each individual commit message more thoroughly describes its purpose.

The following is an overview of the changes:

• Generate new version blocks and reject old version blocks after a super majority has been reached
  • New block version on mainnet is version 7
  • New block version on testnet is version 8
• Generate block templates in accordance with the state of the vote
  • Set the merkle root to either the existing merkle root of the new combined merkle root
  • Set stake root to either the existing stake root or the new commitment root
  • Add new ErrCalcCommitmentRoot mining error code
  • Introduce a new function named calcBlockCommitmentRootV1 which handles creating the version 2 block filter and calculating the resulting commitment root for block templates
  • Remove the no longer used calcTxTreeMerkleRoot function
• Modify block validation to enforce new consensus semantics in accordance with the state of the vote
  • Enforce merkle root block header field as either existing value or new combined merkle root
  • Enforce stake root block header field as either existing value or new commitment root
  • Add new ErrBadCommitmentRoot rule error code to uniquely identify the new consensus violation
  • Introduce new unexported struct to house header commitment data and modify relevant funcs to accept an instance of it
• Introduce a convenience function for determining if the vote passed and is now active
  • Add tests for determining if the agenda is active for both mainnet and testnet
• Introduce several new functions to the standalone module
  • CalcCombinedTxTreeMerkleRoot for calculating the combined merkle root
  • GenerateInclusionProof to generate inclusion proofs for header commitments
  • VerifyInclusionProof to verify inclusion proofs for header commitments
  • Add comprehensive tests for the new functions
  • Update the documentation for the new functions
• Introduce a new package named blockcf2 for creating the filters
  • Provide the B and M constants which represent the predefined bin size and false positive rate for the block filters
  • Introduce PrevScripter interface and associated PrevScriptError
  • Implement Regular function for creating and returning the filter
  • Provide Key function to return the block-specific key for a filter from the merkle root of the associated block
  • Add a README.md for the new package
• Introduce a new blockchain.CalcCommitmentRootV1 function for calculating the commitment root
  • Add tests to ensure the commitment root is calculated properly
• Add a new blockchain.FetchUtxoViewParentTemplate method
• Make UtxoViewpoint satisfy the PrevScripter interface so it may be used as a source previous scripts when creating filters
• Introduce a new database bucket to house v2 block filters
• Implement database migration code to retroactively create the new filters for all historical blocks
  • Bump the chain database version to 6
  • Introduce code to allow spent txout entries from the spend journal to be used as a source of previous scripts to significantly optimize the filter creation as compared to what would be required to reconstruct all the utxo views as of each block
  • Mark all blocks that failed validation under the current consensus rules as eligible for validation again
• Create and store the new filters in the db when connecting blocks
• Introduce exported function named FilterByBlockHash to retrieve the new block filters so they are available to be served
• Export a new constant named HeaderCmtFilterIndex which indicates the header proof index for the filter header commitment
• Implement getcfilterv2 and cfilterv2 wire messages to request and deliver version 2 committed filters
  • Bump the wire protocol version to 7
  • Export a new constant named CFilterV2Version
  • Bump the default user agent to /dcrwire:0.4.0/
  • Add comprehensive tests for the new messages
• Update peer to provide listeners for the new getcfilterv2 and cfilterv2 messages
• Update server to support the new CFilterV2Version protocol version
  • Respond to getcfilterv2 with the requested filter and associated inclusion proof
• Implement getcfilterv2 JSON-RPC to request and deliver version 2 committed filter over RPC
  • Update rpcclient to support the new RPC and convert the result to concrete types
  • Update JSON-RPC API documentation

[davecgh]

The following is some example code showing how this all fits together from a lightweight client perspective:

package main

import (
	"encoding/hex"
	"fmt"

	"github.com/decred/dcrd/blockchain/standalone"
	"github.com/decred/dcrd/chaincfg/chainhash"
	"github.com/decred/dcrd/gcs/v2"
	"github.com/decred/dcrd/gcs/v2/blockcf2"
	"github.com/decred/dcrd/wire"
)

// block382251Data returns the actual version 2 cfilter wire message and block
// header for mainnet block 382251.
func block382251Data() (*wire.MsgCFilterV2, *wire.BlockHeader, error) {
	// Ordinarily this would be requested over the wire via the `getcfilterv2`
	// message, however, it is hard coded here for the purposes of the example.
	//
	// This data is from main chain block 382251.
	hashStr := "000000000000000004756ceb7d94730d0686dd9947a46fd1e896df289e75d86c"
	blockHash, err := chainhash.NewHashFromStr(hashStr)
	if err != nil {
		return nil, nil, err
	}
	filterHex := "2a2ccbedf4ab5a488cc1eaf328102d82951095e99cc99cf150603573853" +
		"f9aa9bb11809a46f19f1efc322efe04504dc5a304e89c4303f28010951c8ab420ccc" +
		"0a7cb010a95684dcdf02af6d7d0812f36503e7d3e0ec1d29e82aa934b5fd69d21597" +
		"4a7915a49d885e389d0bf2f122670"
	filterBytes, err := hex.DecodeString(filterHex)
	if err != nil {
		return nil, nil, err
	}
	cfilter := wire.MsgCFilterV2{
		BlockHash:   *blockHash,
		Data:        filterBytes,
		ProofIndex:  0,
		ProofHashes: nil,
	}

	// Ordinarily this would already be known to the client and proven to
	// properly connect to the rest of the chain back to the genesis block,
	// however, it is hard coded here for the purposes of the example.
	headerHex := "0600000095c4d60268260ac4f0b888a8efdcc3f23c852d30379e0807000" +
		"0000000000000931e41516eb7b6cea3c7c2f0d79cc02ab643e1780f8d305038c2a34" +
		"b44e3cd5ffda62ac6f21d26e0e0d9946735faaccea1f27b5659d7debf3853f307d02" +
		"e72a201007e54003cb7d305000200df9f000021e01b18da4cde05030000002bd5050" +
		"09f2e0000ea0c895d78df86eb374153005872e410903900010000000000000000000" +
		"00000000000000000000006000000"
	headerBytes, err := hex.DecodeString(headerHex)
	if err != nil {
		return nil, nil, err
	}
	var header wire.BlockHeader
	if err := header.FromBytes(headerBytes); err != nil {
		return nil, nil, err
	}

	return &cfilter, &header, nil
}

func main() {
	// Obtain the filter and block header for mainnet block 382251.
	cfilter, header, err := block382251Data()
	if err != nil {
		panic(err)
	}

	// Parse the raw filter bytes.
	filter, err := gcs.FromBytesV2(blockcf2.B, blockcf2.M, cfilter.Data)
	if err != nil {
		panic(err)
	}

	// Verify the filter inclusion proof.
	//
	// This will be false for this example because the consensus vote has not
	// taken place and hence the stake root field will not commit to the filter
	// yet.
	filterHash := filter.Hash()
	verified := standalone.VerifyInclusionProof(&header.StakeRoot, &filterHash,
		cfilter.ProofIndex, cfilter.ProofHashes)
	fmt.Printf("filter commitment verified?: %v\n", verified)

	// Match a script from one of the ticket commitment outputs in the block.
	pkScript, err := hex.DecodeString("76a9142c9ea7e95d37d6713269bbfb52038c98948996e788ac")
	if err != nil {
		panic(err)
	}
	matches := filter.Match(blockcf2.Key(&header.MerkleRoot), pkScript)
	fmt.Printf("matches script %x?: %v", pkScript, matches)
}

Output:

filter commitment verified?: false
matches script 76a9142c9ea7e95d37d6713269bbfb52038c98948996e788ac?: true

[matheusd]

First pass code review, only found a few nits.

Commit message for "gcs/blockcf2: Implement v2 block filter creation" has blockc.

[jholdstock]

Upgrade took around 1 minute on testnet https://pastebin.com/7N70CEkd
And around 2 minutes on mainnet https://pastebin.com/1B1ciTVm

[matheusd]

Small gotcha for clients of the filters: for all the talk of combining merkleRoot with stakeRoot, block filter data before the activation is keyed only by the regular tree (i.e. the merkleRoot header field), not by both merkleRoot and stakeRoot.

This is explicitly called out in the dcp doc:

"The key set to the first 16 bytes of the MerkleRoot (in the exact order produced by the hash function) of the block covered by the filter"

But the interpretation is that before the activation of the DCP the key derivation does not include the stake root. So if you're "manually" calculating the key you need to keep that in mind: https://0bin.net/paste/rbzta44vbmgbdWZh#0a9atexOMbjpVBi-Yq/OJ6IxiZe91sAxqvG8uy0QlbP

This might save someone a few minutes of head scratching :)

[davecgh]

Yes, that behavior is intentional. It is keyed based on the MerkleRoot of the block, whatever it happens to be at that point. As you note, it will be only the Merkle root of the regular transaction tree prior to the point the vote activates and the semantics change such that it combines the two.

Typically clients will need to download the block headers and verify they connect together properly all the back to the genesis block. Thus, they will simply be able use the MerkleRoot from the header exactly as it is without having to worry about the switchover point.

[matheusd]

🎉 🎉 🎉

[dajohi]

testnet miner tOK