[Consensus] Approval Processing Core -> Sealing Core

[durkmurder]

This PR implements next issues:

• Worker pool integration
• Engine implementation
• Sealing engine replacement

Main changes:

• Sealing engine was replaced with new event driven approval processing engine
• Matching engine is back again and performs pending receipts processing
• New distributor for hotstuff events. Performs delivery of finalization events to subscribers.
• Big cleanup of not needed modules, mempools, etc

[codecov-commenter]

Codecov Report

Merging #736 (70cd038) into master (4682be4) will decrease coverage by 0.25%.
The diff coverage is 57.46%.

@@            Coverage Diff             @@
##           master     #736      +/-   ##
==========================================
- Coverage   56.64%   56.38%   -0.26%     
==========================================
  Files         423      423              
  Lines       25072    24783     -289     
==========================================
- Hits        14203    13975     -228     
+ Misses       8963     8913      -50     
+ Partials     1906     1895      -11     

Flag	Coverage Δ
unittests	56.38% <57.46%> (-0.26%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...e/consensus/approvals/assignment_collector_tree.go	0.00% <0.00%> (-74.74%)	⬇️
...e/consensus/approvals/incorporated_result_seals.go	0.00% <0.00%> (ø)
engine/consensus/approvals/request_tracker.go	76.74% <ø> (ø)
model/flow/incorporated_result.go	61.11% <ø> (+25.21%)	⬆️
module/mempool/consensus/execution_tree.go	72.97% <ø> (ø)
engine/consensus/matching/core.go	52.43% <52.43%> (ø)
engine/consensus/sealing/engine.go	51.97% <59.37%> (+14.57%)	⬆️
engine/consensus/matching/engine.go	60.56% <60.56%> (ø)
engine/consensus/approvals/approval_collector.go	78.84% <61.53%> (-8.66%)	⬇️
module/mempool/stdmap/pending_receipts.go	75.00% <63.63%> (+0.75%)	⬆️
... and 18 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4682be4...70cd038. Read the comment docs.

[AlexHentschel]

🎉 This PR is finally ready to be merged 🥳 🍾

I provided some minor revisions in PR #774 targeting this branch. Mostly revisions of goDoc and and inline comments and a tiny bit of polishing. The only noteworthy algorithmic addition is the implementation of my comment below.

This really turned out nicely. Great work. Thanks

[AlexHentschel]

I am wondering how this code will behave, when requiredApprovalsForSealConstruction = 0. In this case, sealing should proceed, even if all verifiers are offline.

My theory on how the current implementation will behave is:

• the only code paths reaching ApprovalCollector.SealResult() are:

  • emergency sealing

    flow-go/engine/consensus/approvals/assignment_collector.go

    Lines 139 to 143 in 7e5f2ed

    	func (ac *AssignmentCollector) CheckEmergencySealing(finalizedBlockHeight uint64) error {
    	for _, collector := range ac.allCollectors() {
    	sealable := ac.emergencySealable(collector, finalizedBlockHeight)
    	if sealable {
    	err := collector.SealResult()

  • processing an approval

    flow-go/engine/consensus/approvals/approval_collector.go

    Lines 89 to 110 in 7e5f2ed

    	func (c *ApprovalCollector) ProcessApproval(approval *flow.ResultApproval) error {
    	chunkIndex := approval.Body.ChunkIndex
    	if chunkIndex >= uint64(len(c.chunkCollectors)) {
    	return engine.NewInvalidInputErrorf("approval collector chunk index out of range: %v", chunkIndex)
    	}
    	// there is no need to process approval if we have already enough info for sealing
    	if c.aggregatedSignatures.HasSignature(chunkIndex) {
    	return nil
    	}
    	collector := c.chunkCollectors[chunkIndex]
    	aggregatedSignature, collected := collector.ProcessApproval(approval)
    	if !collected {
    	return nil
    	}
    	approvedChunks := c.aggregatedSignatures.PutSignature(chunkIndex, aggregatedSignature)
    	if approvedChunks < c.numberOfChunks {
    	return nil // still missing approvals for some chunks
    	}
    	return c.SealResult()

• Hence, if all verifiers are offline, we will not generate seals until the result is emergency sealed, even if requiredApprovalsForSealConstruction = 0.

I think the better behaviour would be produce a candidate seal right away, as soon as we construct the ApprovalCollector

This is already implemented in PR #774