Fix the prepareNextSlot scheduler and add payload metrics

[g11tech]

Motivation
Nethermind reported that we were not issuign advance fcU for building payload and hence zero transactions on latest mainnet shadow fork. On debugging it was figured out that the scheduler was wrongly checking the fork on epoch while it should be checking on slot.

This PR fixes the fork check on the slot rather than epoch, as well as add various metrics for block payload production in order to aid debugging.

metrics added to block production stats

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: fb53299	Previous: 5435d01	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	1.7466 ms/op	2.5630 ms/op	0.68
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	64.495 us/op	70.827 us/op	0.91
BLS verify - blst-native	2.1678 ms/op	1.8909 ms/op	1.15
BLS verifyMultipleSignatures 3 - blst-native	4.4704 ms/op	4.1974 ms/op	1.07
BLS verifyMultipleSignatures 8 - blst-native	9.6612 ms/op	8.3552 ms/op	1.16
BLS verifyMultipleSignatures 32 - blst-native	35.306 ms/op	30.345 ms/op	1.16
BLS aggregatePubkeys 32 - blst-native	46.565 us/op	41.153 us/op	1.13
BLS aggregatePubkeys 128 - blst-native	182.21 us/op	165.83 us/op	1.10
getAttestationsForBlock	144.52 ms/op	168.58 ms/op	0.86
isKnown best case - 1 super set check	475.00 ns/op	460.00 ns/op	1.03
isKnown normal case - 2 super set checks	464.00 ns/op	462.00 ns/op	1.00
isKnown worse case - 16 super set checks	464.00 ns/op	444.00 ns/op	1.05
CheckpointStateCache - add get delete	8.6040 us/op	9.4670 us/op	0.91
validate gossip signedAggregateAndProof - struct	5.0357 ms/op	4.8067 ms/op	1.05
validate gossip attestation - struct	2.3621 ms/op	2.1655 ms/op	1.09
altair verifyImport mainnet_s3766816:31	4.8004 s/op	4.9402 s/op	0.97
pickEth1Vote - no votes	2.0049 ms/op	2.1622 ms/op	0.93
pickEth1Vote - max votes	19.080 ms/op	25.937 ms/op	0.74
pickEth1Vote - Eth1Data hashTreeRoot value x2048	12.091 ms/op	12.615 ms/op	0.96
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	19.423 ms/op	22.726 ms/op	0.85
pickEth1Vote - Eth1Data fastSerialize value x2048	1.3922 ms/op	1.6377 ms/op	0.85
pickEth1Vote - Eth1Data fastSerialize tree x2048	14.064 ms/op	17.517 ms/op	0.80
bytes32 toHexString	1.0180 us/op	1.1740 us/op	0.87
bytes32 Buffer.toString(hex)	756.00 ns/op	764.00 ns/op	0.99
bytes32 Buffer.toString(hex) from Uint8Array	993.00 ns/op	1.0970 us/op	0.91
bytes32 Buffer.toString(hex) + 0x	769.00 ns/op	762.00 ns/op	1.01
Object access 1 prop	0.34800 ns/op	0.39800 ns/op	0.87
Map access 1 prop	0.31100 ns/op	0.31600 ns/op	0.98
Object get x1000	10.965 ns/op	14.920 ns/op	0.73
Map get x1000	0.95400 ns/op	0.99100 ns/op	0.96
Object set x1000	68.126 ns/op	100.75 ns/op	0.68
Map set x1000	46.793 ns/op	76.815 ns/op	0.61
Return object 10000 times	0.43590 ns/op	0.40190 ns/op	1.08
Throw Error 10000 times	6.0419 us/op	6.8414 us/op	0.88
enrSubnets - fastDeserialize 64 bits	2.6440 us/op	2.9460 us/op	0.90
enrSubnets - ssz BitVector 64 bits	839.00 ns/op	800.00 ns/op	1.05
enrSubnets - fastDeserialize 4 bits	432.00 ns/op	409.00 ns/op	1.06
enrSubnets - ssz BitVector 4 bits	854.00 ns/op	770.00 ns/op	1.11
prioritizePeers score -10:0 att 32-0.1 sync 2-0	82.033 us/op	102.47 us/op	0.80
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	115.36 us/op	128.22 us/op	0.90
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	194.88 us/op	240.15 us/op	0.81
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	385.14 us/op	577.08 us/op	0.67
prioritizePeers score 0:0 att 64-1 sync 4-1	405.25 us/op	487.67 us/op	0.83
RateTracker 1000000 limit, 1 obj count per request	181.55 ns/op	183.14 ns/op	0.99
RateTracker 1000000 limit, 2 obj count per request	131.91 ns/op	143.73 ns/op	0.92
RateTracker 1000000 limit, 4 obj count per request	106.84 ns/op	119.36 ns/op	0.90
RateTracker 1000000 limit, 8 obj count per request	95.829 ns/op	115.09 ns/op	0.83
RateTracker with prune	3.8490 us/op	4.6330 us/op	0.83
array of 16000 items push then shift	51.587 us/op	4.8218 us/op	10.70
LinkedList of 16000 items push then shift	14.821 ns/op	27.271 ns/op	0.54
array of 16000 items push then pop	197.31 ns/op	253.21 ns/op	0.78
LinkedList of 16000 items push then pop	14.021 ns/op	22.599 ns/op	0.62
array of 24000 items push then shift	77.331 us/op	6.5019 us/op	11.89
LinkedList of 24000 items push then shift	17.210 ns/op	26.388 ns/op	0.65
array of 24000 items push then pop	192.82 ns/op	226.61 ns/op	0.85
LinkedList of 24000 items push then pop	14.700 ns/op	22.571 ns/op	0.65
intersect bitArray bitLen 8	10.870 ns/op	11.180 ns/op	0.97
intersect array and set length 8	132.13 ns/op	173.91 ns/op	0.76
intersect bitArray bitLen 128	57.299 ns/op	66.350 ns/op	0.86
intersect array and set length 128	1.7415 us/op	2.1827 us/op	0.80
Buffer.concat 32 items	1.8350 ns/op	2.2410 ns/op	0.82
pass gossip attestations to forkchoice per slot	4.1986 ms/op	7.0529 ms/op	0.60
computeDeltas	3.5272 ms/op	3.5610 ms/op	0.99
computeProposerBoostScoreFromBalances	803.67 us/op	793.63 us/op	1.01
altair processAttestation - 250000 vs - 7PWei normalcase	3.4236 ms/op	4.4866 ms/op	0.76
altair processAttestation - 250000 vs - 7PWei worstcase	5.9897 ms/op	7.1250 ms/op	0.84
altair processAttestation - setStatus - 1/6 committees join	174.50 us/op	230.29 us/op	0.76
altair processAttestation - setStatus - 1/3 committees join	337.14 us/op	406.30 us/op	0.83
altair processAttestation - setStatus - 1/2 committees join	478.09 us/op	629.30 us/op	0.76
altair processAttestation - setStatus - 2/3 committees join	626.44 us/op	822.02 us/op	0.76
altair processAttestation - setStatus - 4/5 committees join	884.41 us/op	1.1692 ms/op	0.76
altair processAttestation - setStatus - 100% committees join	1.0648 ms/op	1.3648 ms/op	0.78
altair processBlock - 250000 vs - 7PWei normalcase	23.686 ms/op	30.844 ms/op	0.77
altair processBlock - 250000 vs - 7PWei normalcase hashState	40.050 ms/op	52.619 ms/op	0.76
altair processBlock - 250000 vs - 7PWei worstcase	91.623 ms/op	98.559 ms/op	0.93
altair processBlock - 250000 vs - 7PWei worstcase hashState	96.835 ms/op	118.32 ms/op	0.82
phase0 processBlock - 250000 vs - 7PWei normalcase	3.5480 ms/op	6.9678 ms/op	0.51
phase0 processBlock - 250000 vs - 7PWei worstcase	51.119 ms/op	48.393 ms/op	1.06
altair processEth1Data - 250000 vs - 7PWei normalcase	654.28 us/op	1.4961 ms/op	0.44
Tree 40 250000 create	691.59 ms/op	846.83 ms/op	0.82
Tree 40 250000 get(125000)	236.46 ns/op	306.78 ns/op	0.77
Tree 40 250000 set(125000)	1.8948 us/op	2.3284 us/op	0.81
Tree 40 250000 toArray()	25.692 ms/op	33.503 ms/op	0.77
Tree 40 250000 iterate all - toArray() + loop	25.980 ms/op	34.553 ms/op	0.75
Tree 40 250000 iterate all - get(i)	111.14 ms/op	120.07 ms/op	0.93
MutableVector 250000 create	19.399 ms/op	25.008 ms/op	0.78
MutableVector 250000 get(125000)	13.889 ns/op	12.292 ns/op	1.13
MutableVector 250000 set(125000)	423.51 ns/op	594.98 ns/op	0.71
MutableVector 250000 toArray()	5.4350 ms/op	6.6787 ms/op	0.81
MutableVector 250000 iterate all - toArray() + loop	5.7028 ms/op	6.7357 ms/op	0.85
MutableVector 250000 iterate all - get(i)	2.6206 ms/op	2.9867 ms/op	0.88
Array 250000 create	5.4602 ms/op	5.9710 ms/op	0.91
Array 250000 clone - spread	2.3346 ms/op	2.6976 ms/op	0.87
Array 250000 get(125000)	1.2030 ns/op	1.1290 ns/op	1.07
Array 250000 set(125000)	1.2110 ns/op	1.1370 ns/op	1.07
Array 250000 iterate all - loop	151.01 us/op	135.01 us/op	1.12
effectiveBalanceIncrements clone Uint8Array 300000	35.713 us/op	230.99 us/op	0.15
effectiveBalanceIncrements clone MutableVector 300000	777.00 ns/op	699.00 ns/op	1.11
effectiveBalanceIncrements rw all Uint8Array 300000	247.34 us/op	268.33 us/op	0.92
effectiveBalanceIncrements rw all MutableVector 300000	130.92 ms/op	178.68 ms/op	0.73
phase0 afterProcessEpoch - 250000 vs - 7PWei	185.76 ms/op	177.55 ms/op	1.05
phase0 beforeProcessEpoch - 250000 vs - 7PWei	112.75 ms/op	165.28 ms/op	0.68
altair processEpoch - mainnet_e81889	639.59 ms/op	725.04 ms/op	0.88
mainnet_e81889 - altair beforeProcessEpoch	165.09 ms/op	219.42 ms/op	0.75
mainnet_e81889 - altair processJustificationAndFinalization	22.462 us/op	69.411 us/op	0.32
mainnet_e81889 - altair processInactivityUpdates	8.6679 ms/op	10.991 ms/op	0.79
mainnet_e81889 - altair processRewardsAndPenalties	158.85 ms/op	192.84 ms/op	0.82
mainnet_e81889 - altair processRegistryUpdates	2.8560 us/op	13.090 us/op	0.22
mainnet_e81889 - altair processSlashings	810.00 ns/op	3.7640 us/op	0.22
mainnet_e81889 - altair processEth1DataReset	811.00 ns/op	3.9760 us/op	0.20
mainnet_e81889 - altair processEffectiveBalanceUpdates	2.0797 ms/op	2.1981 ms/op	0.95
mainnet_e81889 - altair processSlashingsReset	6.1950 us/op	24.490 us/op	0.25
mainnet_e81889 - altair processRandaoMixesReset	5.5140 us/op	23.255 us/op	0.24
mainnet_e81889 - altair processHistoricalRootsUpdate	991.00 ns/op	5.4600 us/op	0.18
mainnet_e81889 - altair processParticipationFlagUpdates	3.7320 us/op	15.508 us/op	0.24
mainnet_e81889 - altair processSyncCommitteeUpdates	748.00 ns/op	3.5580 us/op	0.21
mainnet_e81889 - altair afterProcessEpoch	194.93 ms/op	177.46 ms/op	1.10
phase0 processEpoch - mainnet_e58758	614.56 ms/op	719.03 ms/op	0.85
mainnet_e58758 - phase0 beforeProcessEpoch	229.56 ms/op	307.99 ms/op	0.75
mainnet_e58758 - phase0 processJustificationAndFinalization	18.730 us/op	57.855 us/op	0.32
mainnet_e58758 - phase0 processRewardsAndPenalties	133.88 ms/op	162.80 ms/op	0.82
mainnet_e58758 - phase0 processRegistryUpdates	8.9800 us/op	34.432 us/op	0.26
mainnet_e58758 - phase0 processSlashings	685.00 ns/op	3.3890 us/op	0.20
mainnet_e58758 - phase0 processEth1DataReset	735.00 ns/op	3.0420 us/op	0.24
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.8599 ms/op	2.1701 ms/op	0.86
mainnet_e58758 - phase0 processSlashingsReset	4.5060 us/op	16.793 us/op	0.27
mainnet_e58758 - phase0 processRandaoMixesReset	5.5670 us/op	23.403 us/op	0.24
mainnet_e58758 - phase0 processHistoricalRootsUpdate	959.00 ns/op	4.3480 us/op	0.22
mainnet_e58758 - phase0 processParticipationRecordUpdates	4.0250 us/op	21.453 us/op	0.19
mainnet_e58758 - phase0 afterProcessEpoch	161.40 ms/op	151.78 ms/op	1.06
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.9680 ms/op	2.1710 ms/op	0.91
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	2.2398 ms/op	2.6063 ms/op	0.86
altair processInactivityUpdates - 250000 normalcase	44.320 ms/op	74.574 ms/op	0.59
altair processInactivityUpdates - 250000 worstcase	50.373 ms/op	76.265 ms/op	0.66
phase0 processRegistryUpdates - 250000 normalcase	8.0390 us/op	28.414 us/op	0.28
phase0 processRegistryUpdates - 250000 badcase_full_deposits	404.51 us/op	464.46 us/op	0.87
phase0 processRegistryUpdates - 250000 worstcase 0.5	221.46 ms/op	291.99 ms/op	0.76
altair processRewardsAndPenalties - 250000 normalcase	131.62 ms/op	170.54 ms/op	0.77
altair processRewardsAndPenalties - 250000 worstcase	139.74 ms/op	167.87 ms/op	0.83
phase0 getAttestationDeltas - 250000 normalcase	11.394 ms/op	12.356 ms/op	0.92
phase0 getAttestationDeltas - 250000 worstcase	11.433 ms/op	12.111 ms/op	0.94
phase0 processSlashings - 250000 worstcase	5.0754 ms/op	5.5841 ms/op	0.91
altair processSyncCommitteeUpdates - 250000	286.55 ms/op	288.96 ms/op	0.99
BeaconState.hashTreeRoot - No change	517.00 ns/op	593.00 ns/op	0.87
BeaconState.hashTreeRoot - 1 full validator	70.745 us/op	117.14 us/op	0.60
BeaconState.hashTreeRoot - 32 full validator	806.18 us/op	697.62 us/op	1.16
BeaconState.hashTreeRoot - 512 full validator	6.9804 ms/op	8.2677 ms/op	0.84
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	99.958 us/op	91.177 us/op	1.10
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.2416 ms/op	1.3389 ms/op	0.93
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	18.049 ms/op	20.138 ms/op	0.90
BeaconState.hashTreeRoot - 1 balances	66.634 us/op	68.198 us/op	0.98
BeaconState.hashTreeRoot - 32 balances	709.56 us/op	628.34 us/op	1.13
BeaconState.hashTreeRoot - 512 balances	7.0175 ms/op	6.8226 ms/op	1.03
BeaconState.hashTreeRoot - 250000 balances	100.04 ms/op	94.992 ms/op	1.05
aggregationBits - 2048 els - zipIndexesInBitList	24.510 us/op	25.685 us/op	0.95
regular array get 100000 times	74.431 us/op	53.113 us/op	1.40
wrappedArray get 100000 times	64.679 us/op	51.982 us/op	1.24
arrayWithProxy get 100000 times	27.766 ms/op	36.541 ms/op	0.76
ssz.Root.equals	500.00 ns/op	637.00 ns/op	0.78
byteArrayEquals	495.00 ns/op	503.00 ns/op	0.98
shuffle list - 16384 els	11.251 ms/op	10.809 ms/op	1.04
shuffle list - 250000 els	165.30 ms/op	157.34 ms/op	1.05
processSlot - 1 slots	12.842 us/op	14.320 us/op	0.90
processSlot - 32 slots	1.8764 ms/op	2.1587 ms/op	0.87
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	421.06 us/op	438.09 us/op	0.96
getCommitteeAssignments - req 1 vs - 250000 vc	5.3716 ms/op	4.6103 ms/op	1.17
getCommitteeAssignments - req 100 vs - 250000 vc	7.7959 ms/op	6.9144 ms/op	1.13
getCommitteeAssignments - req 1000 vs - 250000 vc	8.4492 ms/op	7.1730 ms/op	1.18
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	8.2000 ns/op	8.8800 ns/op	0.92
state getBlockRootAtSlot - 250000 vs - 7PWei	964.20 ns/op	1.1531 us/op	0.84
computeProposers - vc 250000	17.179 ms/op	17.227 ms/op	1.00
computeEpochShuffling - vc 250000	169.89 ms/op	152.09 ms/op	1.12
getNextSyncCommittee - vc 250000	286.43 ms/op	286.24 ms/op	1.00

by benchmarkbot/action