Block Template Validation Service

[joshmg]

Block Template Validation Service

GitHub Issue #8

Problem Description

There is a risk that blocks mined by miners could cause a chain split or be orphaned due to being incompatible between node implementations.

From a miner's perspective, even a small risk associated with these incompatibilities can have a large impact on profitability.
This incompatibility risk increases the incentive for miners and pools to all run the same implementation, which greatly reduces node-diversity between miners.

Incompatibilities in consensus, while generally very rare, are possible due to minute differences in the implementation details between nodes.
Generally, these differences are only challenged by uncommon edge-cases within the scripting and validation logic between the nodes.
Since the block template is only lacking the proper work before it is considered valid, nearly all of the possible incompatibilities between block candidates can be detected before any work is ever performed.

This block template validation service (TVS) proposal aims to reduce the risk of the miners creating an invalid block to near-zero by validating the template block against other implementations before any work is performed on the block template.

Value Proposition

This solution will provide the following benefits:

• reduce the risk of unintentionally mining a block that would cause a network split
• reduce the financial risk to miners of running different node implementations
• increase miner confidence in node-diversity
• alert developers of would-be incompatible blocks

Solution Overview

1. Create a service that is connected to the latest version of multiple node implementations ("validating nodes"):

• BCHD
• Bitcoin ABC
• Bitcoin Unlimited
• Bitcoin Verde
• Flowee The Hub

2. This service shall accept a standard block template from getblocktemplate (as specified by BIP-22, BIP-23, BIP-9, and BIP-145 (if necessary)).

3. Once a block template received, the service ensures each validating node has seen each transaction within the template block.

4. Then the service attempts to validate the template block for each implementation.

5. The service then responds to the requester if any node finds the template invalid.

6. A best effort attempt by the service will be made to determine with transaction(s) trigger the invalid state of the template block, so the requester may choose to omit them.

Additional future extensions to this proposal may include returning a modified block template that excludes only the incompatible transactions.

Notes:

The validating nodes may not be equipped to validate a template block.
This proposal will define a formal BIP to extend getblocktemplate to allow its proposal mode to allow a flag to ignore the proof of work validation for the block data.

Additionally, this solution will create a reference implementation and pull request for Bitcoin ABC that fulfills the above extension to getblocktemplate.
Providing an implementation for Bitcoin ABC (while others are omitted) is considered within scope of this issue due to its current majority marketshare.

If other implementations provide similar required functionality without directly using getblocktemplate then this issue may be later extended to create a compatibility shim(s) for those implementations.

Bitcoin Verde does not currently support proposal mode of getblocktemplate.
This issue will modify the currently equivalent functionality to match the getblocktemplate RPC API, including proposal mode.

Solution Milestones

1. Service Creation

a. Define the service API to validate a block template.

b. Invoke multiple RPC getblocktemplate:proposal calls to connected node(s) to validate the template block, and return the validation status.

2. Create BIP

a. Create a formal BIP to extend the existing functionality of getblocktemplate:proposal.

b. Create reference implementation for Bitcoin ABC.

3. Modify Bitcoin Verde

a. Modify Bitcoin Verde to fulfill the proposed BIP.

Estimated Relative Complexity

• Milestone 1 - 60 / 160 (37.5%)
• Milestone 2 - 60 / 160 (37.5%)
• Milestone 3 - 40 / 160 (25%)

Budget

This proposal does not have a minimum starting budget.

Completing this proposal will require approximately 160 hours.
At a rate of 0.5 BCH/hr, the total requested budget for this proposal is 80 BCH.

Funding Address

Funding this proposal may be sponsored by sending Bitcoin Cash to the following address:

18SSESYMBCsFkDGHwcvUFCoxwFcsCHgV73
(bitcoincash:qpges4u8lm9w6mekdkrn9cpy79lst2lukywwgpkf8r)

Authorization Signature:

The signature is signed with our primary donation address, 1VerdeMuXH1ApSZsQDkuHHZrwJaAtXTVn, which can be found on bitcoinverde.org.

The signature message consists of a Bitcoin Signed Message with the following format:

Issue-Number | Issue Title | Funding Address | Estimate Hours | Budget BCH

Notes:

1. The pre-image includes the concatenation symbol.

Pre-image:

8|Block Template Validation Service|18SSESYMBCsFkDGHwcvUFCoxwFcsCHgV73|160|80

Signature:

G8HaPOT1tLJE0JfcUnejdBKAuP+xzS5vu0ZZY85gEjS/WAQonhUI5kK94B1vb8cN4IFOSDDPKadmHzChy7xcgsg=

[cpacia]

This is a great idea.

Might be hard to deduce the cause of the invalid block from error strings as I imagine they might different from each implementation and may not even remain constant over time.

Hopefully the code for this will be open source a miner that doesn't want to rely on a third party can use it themselves.

[ftrader]

Very much like this proposal.
Had a look through, and not quite clear yet, but possibly subject to Solution Milestones "1. (a) Define ..." is how to interpret Solution Overview "(5) The service then responds to the requester if any node finds the template invalid."

The phrasing in the overview suggests a response only if a validation error occurs.
Presumably this isn't really intended like that but just a simplification for proposal purposes?
As I would still expect the TVS to respond if all was good, otherwise one cannot be sure...

I'd suggest as well that there should be a setting, possibly negotiated or announced, for the TVS to return a result set within a maximum agreed amount of time.
More concretely:
The result set might contain something like a Valid//Invalid//Indeterminate for each implementation it queried.
"Invalid" being accompanied by the node's error message / code.
"Indeterminate" included because a node may return unexpected information, or deterministically or nondeterministically crash without returning a result to the TVS.
In both cases no clear implication could be made on the validation of the template.
If a node fails to respond in time before the TVS has to provide its results, then its status would also be classed as Indeterminate.

Probably "before any work is performed on the block template" could be modified to "before any significant work is performed on the block template" as miners might start to work while the result is being sought, if the time to gather it is non-negligible. (?)
Then they could just abort if it came out negative in a way that they cared about.

[joshmg]

This is a great idea.

Thanks, Chris! I hope we can get BCHD involved, and I look forward to getting to collaborate with you guys.

Might be hard to deduce the cause of the invalid block from error strings as I imagine they might different from each implementation and may not even remain constant over time.

Totally agree. I tried to describe the difficulty of essentially falling back to heuristics for this by describing the feature as "best effort". Perhaps in later iterations we can form some kind of standard procedure for communicating this across nodes.

Hopefully the code for this will be open source a miner that doesn't want to rely on a third party can use it themselves.

100% yes. The word "service" is more technical than anything else; what I mean is it's a standalone application. We want to leave the decision up to the pools/miners to run their own or if they want to rely on a service they don't control. I see benefits of both, so it's intended to be able to do either.

[joshmg]

Very much like this proposal.

Thanks, freetrader. I'm happy to have your support on this.

Had a look through, and not quite clear yet, but possibly subject to Solution Milestones "1. (a) Define ..." is how to interpret Solution Overview "(5) The service then responds to the requester if any node finds the template invalid."

The phrasing in the overview suggests a response only if a validation error occurs.
Presumably this isn't really intended like that but just a simplification for proposal purposes?
As I would still expect the TVS to respond if all was good, otherwise one cannot be sure...

It's just a simplification. Basically, the service acts as a web service with a request/response mechanism. Every request will have a response. The specifics are going to be complicated to define of course, especially since the rate in which template blocks are updated/validated can be quite frequent, and validating a block is usually nontrivial. So the service may end up having a queue/webhook callback mechanic to it instead of a serialized response. I think that's ultimately going to be a decision we make once we have idea of load and throughput.

I'd suggest as well that there should be a setting, possibly negotiated or announced, for the TVS to return a result set within a maximum agreed amount of time.
More concretely:
The result set might contain something like a Valid//Invalid//Indeterminate for each implementation it queried.
"Invalid" being accompanied by the node's error message / code.
"Indeterminate" included because a node may return unexpected information, or deterministically or nondeterministically crash without returning a result to the TVS.
In both cases no clear implication could be made on the validation of the template.
If a node fails to respond in time before the TVS has to provide its results, then its status would also be classed as Indeterminate.

This is an excellent suggestion, and it aligns itself well with the above consideration regarding the nonfunctional requirements around throughput. I'd probably presume a node crashing due a template block would be treated similarly as "invalid", but calling out more specifically (via a mechanism like the "indeterminate" idea you suggested) would be more useful.

Probably "before any work is performed on the block template" could be modified to "before any significant work is performed on the block template" as miners might start to work while the result is being sought, if the time to gather it is non-negligible. (?)
Then they could just abort if it came out negative in a way that they cared about.

Absolutely. I expect the miner/pool to make this decision ultimately. It'd be equally plausible they'd just mine an empty block or whatever previous job they were on if still valid, too. The same decision making ability is up to the miners/pool as well in regards to what they do in the case the template would be considered invalid between the network. They could switch to another node and use its template instead, or they could again mine an empty block, or they can attempt to exclude any problematic transactions, etc. Lots of options, and I expect different miners will have different opinions here, and I look forward to hearing their perspectives and thoughts.

[molecular]

great idea to introduce the "funding wanted" label.

[joshmg]

This issue was funded in our flipstarter campaign and work it has begun: https://verde.flipstarter.cash

[joshmg]

Bitbalancer code is complete and we are in conversations with mining pools to deploy the bitbalancer in a limited capacity. The code and product may be found here: https://github.com/softwareverde/bitbalancer