Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multi: Enable vote for DCP0005. #1906

Merged
merged 11 commits into from Oct 13, 2019

Conversation

@davecgh
Copy link
Member

davecgh commented Sep 23, 2019

This requires PR #1904.

Testing Notes

As of this PR, the expected behavior is that there is a single migration that takes around 10 to 15 minutes to complete. after which it will no longer be possible to downgrade.

As the warning above notes, if you try to run an older software version after this migration has completed, you will get an error message similar to Unable to start server on [:9108]: the current blockchain database is no longer compatible with this version of the software (6 > 5).


This implements the agenda for voting on header commitments as defined in DCP0005 along with consensus tests to ensure its correctness. It consists of several commits that have been carefully crated to make the changes easier to reason about and review.

Each individual commit message more thoroughly describes its purpose.

The following is an overview of the changes:

  • Generate new version blocks and reject old version blocks after a super majority has been reached
    • New block version on mainnet is version 7
    • New block version on testnet is version 8
  • Generate block templates in accordance with the state of the vote
    • Set the merkle root to either the existing merkle root of the new combined merkle root
    • Set stake root to either the existing stake root or the new commitment root
    • Add new ErrCalcCommitmentRoot mining error code
    • Introduce a new function named calcBlockCommitmentRootV1 which handles creating the version 2 block filter and calculating the resulting commitment root for block templates
    • Remove the no longer used calcTxTreeMerkleRoot function
  • Modify block validation to enforce new consensus semantics in accordance with the state of the vote
    • Enforce merkle root block header field as either existing value or new combined merkle root
    • Enforce stake root block header field as either existing value or new commitment root
    • Add new ErrBadCommitmentRoot rule error code to uniquely identify the new consensus violation
    • Introduce new unexported struct to house header commitment data and modify relevant funcs to accept an instance of it
  • Introduce a convenience function for determining if the vote passed and is now active
    • Add tests for determining if the agenda is active for both mainnet and testnet
  • Introduce several new functions to the standalone module
    • CalcCombinedTxTreeMerkleRoot for calculating the combined merkle root
    • GenerateInclusionProof to generate inclusion proofs for header commitments
    • VerifyInclusionProof to verify inclusion proofs for header commitments
    • Add comprehensive tests for the new functions
    • Update the documentation for the new functions
  • Introduce a new package named blockcf2 for creating the filters
    • Provide the B and M constants which represent the predefined bin size and false positive rate for the block filters
    • Introduce PrevScripter interface and associated PrevScriptError
    • Implement Regular function for creating and returning the filter
    • Provide Key function to return the block-specific key for a filter from the merkle root of the associated block
    • Add a README.md for the new package
  • Introduce a new blockchain.CalcCommitmentRootV1 function for calculating the commitment root
    • Add tests to ensure the commitment root is calculated properly
  • Add a new blockchain.FetchUtxoViewParentTemplate method
  • Make UtxoViewpoint satisfy the PrevScripter interface so it may be used as a source previous scripts when creating filters
  • Introduce a new database bucket to house v2 block filters
  • Implement database migration code to retroactively create the new filters for all historical blocks
    • Bump the chain database version to 6
    • Introduce code to allow spent txout entries from the spend journal to be used as a source of previous scripts to significantly optimize the filter creation as compared to what would be required to reconstruct all the utxo views as of each block
    • Mark all blocks that failed validation under the current consensus rules as eligible for validation again
  • Create and store the new filters in the db when connecting blocks
  • Introduce exported function named FilterByBlockHash to retrieve the new block filters so they are available to be served
  • Export a new constant named HeaderCmtFilterIndex which indicates the header proof index for the filter header commitment
  • Implement getcfilterv2 and cfilterv2 wire messages to request and deliver version 2 committed filters
    • Bump the wire protocol version to 7
    • Export a new constant named CFilterV2Version
    • Bump the default user agent to /dcrwire:0.4.0/
    • Add comprehensive tests for the new messages
  • Update peer to provide listeners for the new getcfilterv2 and cfilterv2 messages
  • Update server to support the new CFilterV2Version protocol version
    • Respond to getcfilterv2 with the requested filter and associated inclusion proof
  • Implement getcfilterv2 JSON-RPC to request and deliver version 2 committed filter over RPC
    • Update rpcclient to support the new RPC and convert the result to concrete types
    • Update JSON-RPC API documentation
@davecgh davecgh added this to the 1.5.0 milestone Sep 23, 2019
server.go Outdated Show resolved Hide resolved
gcs/blockcf2/blockcf.go Outdated Show resolved Hide resolved
@davecgh

This comment has been minimized.

Copy link
Member Author

davecgh commented Sep 23, 2019

The following is some example code showing how this all fits together from a lightweight client perspective:

package main

import (
	"encoding/hex"
	"fmt"

	"github.com/decred/dcrd/blockchain/standalone"
	"github.com/decred/dcrd/chaincfg/chainhash"
	"github.com/decred/dcrd/gcs/v2"
	"github.com/decred/dcrd/gcs/v2/blockcf2"
	"github.com/decred/dcrd/wire"
)

// block382251Data returns the actual version 2 cfilter wire message and block
// header for mainnet block 382251.
func block382251Data() (*wire.MsgCFilterV2, *wire.BlockHeader, error) {
	// Ordinarily this would be requested over the wire via the `getcfilterv2`
	// message, however, it is hard coded here for the purposes of the example.
	//
	// This data is from main chain block 382251.
	hashStr := "000000000000000004756ceb7d94730d0686dd9947a46fd1e896df289e75d86c"
	blockHash, err := chainhash.NewHashFromStr(hashStr)
	if err != nil {
		return nil, nil, err
	}
	filterHex := "2a2ccbedf4ab5a488cc1eaf328102d82951095e99cc99cf150603573853" +
		"f9aa9bb11809a46f19f1efc322efe04504dc5a304e89c4303f28010951c8ab420ccc" +
		"0a7cb010a95684dcdf02af6d7d0812f36503e7d3e0ec1d29e82aa934b5fd69d21597" +
		"4a7915a49d885e389d0bf2f122670"
	filterBytes, err := hex.DecodeString(filterHex)
	if err != nil {
		return nil, nil, err
	}
	cfilter := wire.MsgCFilterV2{
		BlockHash:   *blockHash,
		Data:        filterBytes,
		ProofIndex:  0,
		ProofHashes: nil,
	}

	// Ordinarily this would already be known to the client and proven to
	// properly connect to the rest of the chain back to the genesis block,
	// however, it is hard coded here for the purposes of the example.
	headerHex := "0600000095c4d60268260ac4f0b888a8efdcc3f23c852d30379e0807000" +
		"0000000000000931e41516eb7b6cea3c7c2f0d79cc02ab643e1780f8d305038c2a34" +
		"b44e3cd5ffda62ac6f21d26e0e0d9946735faaccea1f27b5659d7debf3853f307d02" +
		"e72a201007e54003cb7d305000200df9f000021e01b18da4cde05030000002bd5050" +
		"09f2e0000ea0c895d78df86eb374153005872e410903900010000000000000000000" +
		"00000000000000000000006000000"
	headerBytes, err := hex.DecodeString(headerHex)
	if err != nil {
		return nil, nil, err
	}
	var header wire.BlockHeader
	if err := header.FromBytes(headerBytes); err != nil {
		return nil, nil, err
	}

	return &cfilter, &header, nil
}

func main() {
	// Obtain the filter and block header for mainnet block 382251.
	cfilter, header, err := block382251Data()
	if err != nil {
		panic(err)
	}

	// Parse the raw filter bytes.
	filter, err := gcs.FromBytesV2(blockcf2.B, blockcf2.M, cfilter.Data)
	if err != nil {
		panic(err)
	}

	// Verify the filter inclusion proof.
	//
	// This will be false for this example because the consensus vote has not
	// taken place and hence the stake root field will not commit to the filter
	// yet.
	filterHash := filter.Hash()
	verified := standalone.VerifyInclusionProof(&header.StakeRoot, &filterHash,
		cfilter.ProofIndex, cfilter.ProofHashes)
	fmt.Printf("filter commitment verified?: %v\n", verified)

	// Match a script from one of the ticket commitment outputs in the block.
	pkScript, err := hex.DecodeString("76a9142c9ea7e95d37d6713269bbfb52038c98948996e788ac")
	if err != nil {
		panic(err)
	}
	matches := filter.Match(blockcf2.Key(&header.MerkleRoot), pkScript)
	fmt.Printf("matches script %x?: %v", pkScript, matches)
}

Output:

filter commitment verified?: false
matches script 76a9142c9ea7e95d37d6713269bbfb52038c98948996e788ac?: true
Copy link
Member

matheusd left a comment

First pass code review, only found a few nits.

Commit message for "gcs/blockcf2: Implement v2 block filter creation" has blockc.

blockchain/agendas_test.go Outdated Show resolved Hide resolved
gcs/blockcf2/blockcf.go Outdated Show resolved Hide resolved
gcs/blockcf2/blockcf.go Show resolved Hide resolved
blockchain/chainio.go Outdated Show resolved Hide resolved
@davecgh davecgh force-pushed the davecgh:multi_implement_v2_cfilters branch from 17bcff3 to 4f54457 Oct 6, 2019
@jholdstock

This comment has been minimized.

Copy link
Member

jholdstock commented Oct 6, 2019

Upgrade took around 1 minute on testnet https://pastebin.com/7N70CEkd
And around 2 minutes on mainnet https://pastebin.com/1B1ciTVm

@davecgh davecgh force-pushed the davecgh:multi_implement_v2_cfilters branch from d577f50 to 6475daf Oct 7, 2019
@davecgh davecgh marked this pull request as ready for review Oct 7, 2019
@davecgh davecgh force-pushed the davecgh:multi_implement_v2_cfilters branch from 6475daf to caeebee Oct 7, 2019
@matheusd

This comment has been minimized.

Copy link
Member

matheusd commented Oct 7, 2019

Small gotcha for clients of the filters: for all the talk of combining merkleRoot with stakeRoot, block filter data before the activation is keyed only by the regular tree (i.e. the merkleRoot header field), not by both merkleRoot and stakeRoot.

This is explicitly called out in the dcp doc:

"The key set to the first 16 bytes of the MerkleRoot (in the exact order produced by the hash function) of the block covered by the filter"

But the interpretation is that before the activation of the DCP the key derivation does not include the stake root. So if you're "manually" calculating the key you need to keep that in mind: https://0bin.net/paste/rbzta44vbmgbdWZh#0a9atexOMbjpVBi-Yq/OJ6IxiZe91sAxqvG8uy0QlbP

This might save someone a few minutes of head scratching :)

@davecgh

This comment has been minimized.

Copy link
Member Author

davecgh commented Oct 7, 2019

Yes, that behavior is intentional. It is keyed based on the MerkleRoot of the block, whatever it happens to be at that point. As you note, it will be only the Merkle root of the regular transaction tree prior to the point the vote activates and the semantics change such that it combines the two.

Typically clients will need to download the block headers and verify they connect together properly all the back to the genesis block. Thus, they will simply be able use the MerkleRoot from the header exactly as it is without having to worry about the switchover point.

Copy link
Member

matheusd left a comment

🎉 🎉 🎉

davecgh added 11 commits Sep 21, 2019
This adds a new definition for the upcoming agenda vote to enable header
commitments a provide an initial commitment to a version 2 GCS filter as
defined by DCP0005.  It does not include any code to make decisions or
bump the versions.  It is only the definition.

Also, bump the chaincfg module to v2.3.0 so the new definitions are
available to consumers.
This implements the agenda for voting on combining the regular and stake
tree merkle roots into a single merkle root that is stored in the merkle
root field of the block header as defined in DCP0005 along with
consensus tests.

In particular, once the vote has passed and is active, the final merkle
root is the result of a merkle tree that itself has the individual
merkle roots of the two transaction trees as leaves.

It is also worth noting that this does not repurpose the stake root
field once the vote has passed as that will be done in a future commit.
This implies that it is important for both this commit and the
aforementioned future commit which changes the stake root field
semantics to be merged at the same time.

The following is an overview of the changes:

- Generate new version blocks and reject old version blocks after a
  super majority has been reached
  - New block version on mainnet is version 7
  - New block version on testnet is version 8
- Generate block templates with the correct merkle root set in
  accordance with the state of the vote
- Introduce a convenience function for determining if the vote passed
  and is now active
- Introduce a new function in the standalone module for calculating the
  combined merkle root
  - Add tests to ensure the combined merkle root calculation produces
    the correct results
- Modify block validation to enforce the merkle root field in accordance
  with the state of the vote
- Add tests for determining if the agenda is active for both mainnet and
  testnet
This adds two new functions to the blockchain/standalone module named
GenerateInclusionProof and VerifyInclusionProof which can be used to
generate and verify inclusion proofs for merkle trees, respectively.

It also updates the documentation and includes comprehensive tests.
This implements logic to create version 2 block filters as defined in
DCP0005.

In order to accomplish this a new package named blockcf2 is provided.

Since version 2 block filters require all previous output scripts
referenced as inputs by the block, the new package provides an interface
named PrevScripter to prevent tight coupling to any specific type.  This
provides the caller with the flexibility to provide the necessary
scripts and associated script versions from whatever machanism they deem
fit.

The primary mechanism for creating a block filter with the package is
the Regular function which accepts the block to create the filter for
along with the aforemented PrevScripter interface.

In addition, the package exports the B and M constants which represent
the predefined GCS bin size and false positive rate for the block
filters.  These constants are required both to create the filters and to
deserialize the filters for matching against them.

Finally, the package also provides a Key function which returns the
block-specifc key for the block filter from the merkle root of the
associated block.  The resulting key is required both to create a filter
and to match entries in it.

This only implements the required logic to generate the filters.  Code
to create, store, commit to, validate, and retrieve the filters will be
added in future commits.

A consensus vote is required to commit to the filters and reject blockc
that violate the commitment.  Code to selectively enable consensus
enforcement based on the result of an agenda vote will be added in a
future commit.

The following is a high level overview of the changes:

- Introduce a new package named blockcf2 for creating the filters
- Provide the B and M constants which represent the predefined bin size
  and false positive rate for the block filters
- Introduce PrevScripter interface and associated PrevScriptError
- Implement Regular functions for creating and returning the filter
- Provide Key function to return the block-specific key for a filter
  from the merkle root of the associated block
- Add a README.md for the new package
This modifies the chain logic to create and store version 2 block
filters for all new blocks and also adds code to migrate the database to
retroactively create and store the v2 filters for all historical blocks.

Since this requires a database upgrade and the next release of the
software will include a vote to change the consensus rules, this also
takes the opportunity to unmark all blocks previously marked as having
failed validation so they are eligible for validation again under what
will likely become new consensus rules.  This ensures clients that did
not update prior to new rules activating are able to automatically
recover under the new rules without having to download the entire chain
again.

The following is a high level overview of the changes:

- Introduce a new database bucket to house v2 block filters
- Make UtxoViewpoint satisfy the PrevScripter interface so it may be
  used as a source previous scripts when creating filters
- Create and store the new filters in the db when connecting blocks
- Introduce exported function named FilterByBlockHash to retrieve the
  new block filters so they are available to be served in the future
- Implement database migration code to retroactively create the new
  filters for all historical blocks
  - Bump the chain database version to 6
  - Introduce code to allow spent txout entries from the spend journal
    to be used as a source of previous scripts to significantly optimize
    the filter creation as compared to what would be required to
    reconstruct all the utxo views as of each block
  - Mark all blocks that failed validation under the current consensus
    rules as eligible for validation again
- Export a new constant named HeaderCmtFilterIndex which indicates the
  header proof index for the upcoming filter header commitment
This implements new getcfilterv2 and cfilterv2 messages which are used
to request and deliver version 2 committed gcs filters along with their
associated header commitment inclusion proof.

The getcfilterv2 message requests a version 2 gcs filter and proof for a
given block hash.  The cfilterv2 message is sent in response with the
requested filter and proof.

It should be noted that since all header commitments require consensus
changes, the proof can only be used once a consensus vote passes and the
header actually commits to the filter.

Finally, it also bumps the minor default user agent version for the wire
package to account for the updates.
This implements an agenda for voting to repurpose the stake root field
of the block header to house a commitment root that includes an initial
commitment to a version 2 block filter as defined in DCP0005 along with
consensus tests.

In particular, once the vote has passed and is active, the stake root
field of the block header will contain the merkle root of a merkle tree
that consists of the hash of the version 2 GCS block filter as the sole
leaf.

In order to accomplish validation efficiently and such that it provides
a well-defined path for adding future commitments, this introduces a new
unexported struct in blockchain to house header commitment data, named
headerCommitmentData, and modifies the relevant funcs to accept an
instance of it.

Next, it introduces a new function named CalcCommitmentRootV1 which
takes the filter hash to commit to and returns the resulting commitment
root.  It is certainly the case that this function is not strictly
necessary yet since the version 1 header commitment only consists of a
single item, and hence the root will be the same as the hash provided.
However, this approach is used in order to provide a clear path for
future commitment versions, help make it clear exactly what each version
commits to, and to provide for more consistent code the supports
multiple versions.

Finally, since the version 2 block filters require all previous output
scripts referenced as inputs by the block, and some of those scripts may
no longer be availabled in the pruned utxo set in the case the current
tip block of the main chain does not have enough votes, this introduces
a new blockchain method named FetchUtxoViewParentTemplate to load utxo
details from the point of view of just having connected the given block
template to the parent of the tip of the main chain.  It also ensures
the provided template connects to the parent as expected.

It is also worth noting that this makes use of the already introduced
header commitments agenda and associated changes to generate new version
blocks and therefore must be merged at the same time as the commits
which introduce those changes along with the other consensus changes
that the agenda entails.

The following is an overview of the changes:

- Generate block templates with the stake root value set to either the
  existing stake root or the new commitment root in accordance with the
  state of the vote
  - Add new ErrCalcCommitmentRoot mining error code
  - Introduce a new function named calcBlockCommitmentRootV1 which
    handles creating the version 2 block filter and calculating the
    resulting commitment root for block templates
  - Remove the no longer used mining calcTxTreeMerkleRoot function
- Modify block validation to enforce the commitment root field in
  accordance with the state of the vote
  - Add new ErrBadCommitmentRoot rule error code to uniquely identify
    the new consensus violation
  - Introduce new unexported struct to house header commitment data and
    modify relevant funcs to accept an instance of it
- Introduce a new function named CalcCommitmentRootV1 for calculating
  the commitment root
  - Add tests to ensure the commitment root is calculated properly
- Add a new blockchain method named FetchUtxoViewParentTemplate
This implements a new getcfilterv2 JSON-RPC and updates the JSON-RPC API
documentation accordingly.
This adds support for the getcfilterv2 RPC to rpcclient including
the asynchronous and blocking functions.

The functions return a new type introduced in the rpcclient, named
CFilterV2Result, which contains the concrete types of the result fields
instead of directly returning the raw JSON-RPC data.
@davecgh davecgh force-pushed the davecgh:multi_implement_v2_cfilters branch from caeebee to f3af437 Oct 12, 2019
@dnldd
dnldd approved these changes Oct 12, 2019
@dajohi
dajohi approved these changes Oct 13, 2019
Copy link
Member

dajohi left a comment

testnet miner tOK

@davecgh davecgh merged commit f3af437 into decred:master Oct 13, 2019
2 checks passed
2 checks passed
Go CI (1.12)
Details
Go CI (1.13)
Details
@davecgh davecgh deleted the davecgh:multi_implement_v2_cfilters branch Oct 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.