feat: support el_offline in eth/v1/node/syncing

[nazarhussain]

Motivation

Add support for the status of execution engine in syncing status endpoint.

Description

• Add support for the status of execution engine.

Closes #5542

Steps to test or reproduce

• Run all tests.

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: ea0762f	Previous: 96820d7	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	582.34 us/op	583.73 us/op	1.00
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	52.998 us/op	57.893 us/op	0.92
BLS verify - blst-native	1.3589 ms/op	1.2762 ms/op	1.06
BLS verifyMultipleSignatures 3 - blst-native	2.5598 ms/op	2.5874 ms/op	0.99
BLS verifyMultipleSignatures 8 - blst-native	5.5273 ms/op	5.5594 ms/op	0.99
BLS verifyMultipleSignatures 32 - blst-native	19.986 ms/op	20.313 ms/op	0.98
BLS aggregatePubkeys 32 - blst-native	26.878 us/op	27.002 us/op	1.00
BLS aggregatePubkeys 128 - blst-native	104.81 us/op	105.31 us/op	1.00
getAttestationsForBlock	58.468 ms/op	75.681 ms/op	0.77
isKnown best case - 1 super set check	264.00 ns/op	274.00 ns/op	0.96
isKnown normal case - 2 super set checks	262.00 ns/op	260.00 ns/op	1.01
isKnown worse case - 16 super set checks	268.00 ns/op	267.00 ns/op	1.00
CheckpointStateCache - add get delete	5.8830 us/op	5.5690 us/op	1.06
validate gossip signedAggregateAndProof - struct	2.8544 ms/op	2.8622 ms/op	1.00
validate gossip attestation - struct	1.3814 ms/op	1.3678 ms/op	1.01
pickEth1Vote - no votes	1.3659 ms/op	1.4016 ms/op	0.97
pickEth1Vote - max votes	11.204 ms/op	12.340 ms/op	0.91
pickEth1Vote - Eth1Data hashTreeRoot value x2048	9.7733 ms/op	9.8472 ms/op	0.99
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	16.370 ms/op	17.970 ms/op	0.91
pickEth1Vote - Eth1Data fastSerialize value x2048	772.62 us/op	806.67 us/op	0.96
pickEth1Vote - Eth1Data fastSerialize tree x2048	7.2120 ms/op	7.0576 ms/op	1.02
bytes32 toHexString	559.00 ns/op	644.00 ns/op	0.87
bytes32 Buffer.toString(hex)	417.00 ns/op	424.00 ns/op	0.98
bytes32 Buffer.toString(hex) from Uint8Array	591.00 ns/op	624.00 ns/op	0.95
bytes32 Buffer.toString(hex) + 0x	402.00 ns/op	427.00 ns/op	0.94
Object access 1 prop	0.19200 ns/op	0.19900 ns/op	0.96
Map access 1 prop	0.17200 ns/op	0.17400 ns/op	0.99
Object get x1000	6.7260 ns/op	6.7270 ns/op	1.00
Map get x1000	0.57700 ns/op	0.69400 ns/op	0.83
Object set x1000	64.382 ns/op	76.561 ns/op	0.84
Map set x1000	51.987 ns/op	58.446 ns/op	0.89
Return object 10000 times	0.25100 ns/op	0.28260 ns/op	0.89
Throw Error 10000 times	4.3482 us/op	4.4238 us/op	0.98
fastMsgIdFn sha256 / 200 bytes	3.6590 us/op	3.7860 us/op	0.97
fastMsgIdFn h32 xxhash / 200 bytes	304.00 ns/op	336.00 ns/op	0.90
fastMsgIdFn h64 xxhash / 200 bytes	454.00 ns/op	519.00 ns/op	0.87
fastMsgIdFn sha256 / 1000 bytes	12.108 us/op	12.247 us/op	0.99
fastMsgIdFn h32 xxhash / 1000 bytes	454.00 ns/op	483.00 ns/op	0.94
fastMsgIdFn h64 xxhash / 1000 bytes	525.00 ns/op	545.00 ns/op	0.96
fastMsgIdFn sha256 / 10000 bytes	107.17 us/op	108.72 us/op	0.99
fastMsgIdFn h32 xxhash / 10000 bytes	1.9940 us/op	2.0370 us/op	0.98
fastMsgIdFn h64 xxhash / 10000 bytes	1.4430 us/op	1.4360 us/op	1.00
enrSubnets - fastDeserialize 64 bits	1.4440 us/op	1.6330 us/op	0.88
enrSubnets - ssz BitVector 64 bits	558.00 ns/op	593.00 ns/op	0.94
enrSubnets - fastDeserialize 4 bits	216.00 ns/op	210.00 ns/op	1.03
enrSubnets - ssz BitVector 4 bits	575.00 ns/op	594.00 ns/op	0.97
prioritizePeers score -10:0 att 32-0.1 sync 2-0	119.93 us/op	127.40 us/op	0.94
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	153.32 us/op	167.44 us/op	0.92
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	190.33 us/op	206.35 us/op	0.92
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	351.91 us/op	368.00 us/op	0.96
prioritizePeers score 0:0 att 64-1 sync 4-1	431.12 us/op	478.01 us/op	0.90
array of 16000 items push then shift	1.6849 us/op	1.8060 us/op	0.93
LinkedList of 16000 items push then shift	9.4620 ns/op	10.032 ns/op	0.94
array of 16000 items push then pop	116.51 ns/op	126.57 ns/op	0.92
LinkedList of 16000 items push then pop	8.9890 ns/op	10.088 ns/op	0.89
array of 24000 items push then shift	2.4665 us/op	2.5024 us/op	0.99
LinkedList of 24000 items push then shift	9.1120 ns/op	9.8590 ns/op	0.92
array of 24000 items push then pop	89.042 ns/op	95.535 ns/op	0.93
LinkedList of 24000 items push then pop	9.0340 ns/op	9.2410 ns/op	0.98
intersect bitArray bitLen 8	13.580 ns/op	13.963 ns/op	0.97
intersect array and set length 8	87.171 ns/op	99.567 ns/op	0.88
intersect bitArray bitLen 128	45.232 ns/op	47.718 ns/op	0.95
intersect array and set length 128	1.2052 us/op	1.2643 us/op	0.95
Buffer.concat 32 items	2.9990 us/op	3.4540 us/op	0.87
Uint8Array.set 32 items	2.3460 us/op	2.5940 us/op	0.90
transfer serialized Status (84 B)	2.2150 us/op	2.2040 us/op	1.00
copy serialized Status (84 B)	1.7420 us/op	1.8150 us/op	0.96
transfer serialized SignedVoluntaryExit (112 B)	2.2160 us/op	2.1940 us/op	1.01
copy serialized SignedVoluntaryExit (112 B)	1.7260 us/op	1.8320 us/op	0.94
transfer serialized ProposerSlashing (416 B)	2.3030 us/op	2.6950 us/op	0.85
copy serialized ProposerSlashing (416 B)	2.6840 us/op	2.8330 us/op	0.95
transfer serialized Attestation (485 B)	2.5450 us/op	3.2010 us/op	0.80
copy serialized Attestation (485 B)	2.7700 us/op	3.1510 us/op	0.88
transfer serialized AttesterSlashing (33232 B)	2.8120 us/op	3.3370 us/op	0.84
copy serialized AttesterSlashing (33232 B)	6.4310 us/op	11.548 us/op	0.56
transfer serialized Small SignedBeaconBlock (128000 B)	2.9270 us/op	3.4460 us/op	0.85
copy serialized Small SignedBeaconBlock (128000 B)	14.288 us/op	26.172 us/op	0.55
transfer serialized Avg SignedBeaconBlock (200000 B)	3.2880 us/op	4.0140 us/op	0.82
copy serialized Avg SignedBeaconBlock (200000 B)	26.037 us/op	38.977 us/op	0.67
transfer serialized BlobsSidecar (524380 B)	3.6760 us/op	4.9730 us/op	0.74
copy serialized BlobsSidecar (524380 B)	87.189 us/op	117.28 us/op	0.74
transfer serialized Big SignedBeaconBlock (1000000 B)	3.8620 us/op	4.7770 us/op	0.81
copy serialized Big SignedBeaconBlock (1000000 B)	367.56 us/op	327.44 us/op	1.12
pass gossip attestations to forkchoice per slot	2.7622 ms/op	2.8886 ms/op	0.96
forkChoice updateHead vc 100000 bc 64 eq 0	2.1534 ms/op	2.1982 ms/op	0.98
forkChoice updateHead vc 600000 bc 64 eq 0	12.325 ms/op	15.525 ms/op	0.79
forkChoice updateHead vc 1000000 bc 64 eq 0	22.618 ms/op	22.030 ms/op	1.03
forkChoice updateHead vc 600000 bc 320 eq 0	17.319 ms/op	18.398 ms/op	0.94
forkChoice updateHead vc 600000 bc 1200 eq 0	87.995 ms/op	96.298 ms/op	0.91
forkChoice updateHead vc 600000 bc 64 eq 1000	21.530 ms/op	23.047 ms/op	0.93
forkChoice updateHead vc 600000 bc 64 eq 10000	23.088 ms/op	24.503 ms/op	0.94
forkChoice updateHead vc 600000 bc 64 eq 300000	35.468 ms/op	68.278 ms/op	0.52
computeDeltas	3.2092 ms/op	4.3505 ms/op	0.74
computeProposerBoostScoreFromBalances	1.8603 ms/op	1.8764 ms/op	0.99
altair processAttestation - 250000 vs - 7PWei normalcase	2.4554 ms/op	3.4743 ms/op	0.71
altair processAttestation - 250000 vs - 7PWei worstcase	3.4662 ms/op	5.1019 ms/op	0.68
altair processAttestation - setStatus - 1/6 committees join	147.66 us/op	158.03 us/op	0.93
altair processAttestation - setStatus - 1/3 committees join	279.91 us/op	299.99 us/op	0.93
altair processAttestation - setStatus - 1/2 committees join	382.52 us/op	405.82 us/op	0.94
altair processAttestation - setStatus - 2/3 committees join	467.86 us/op	496.95 us/op	0.94
altair processAttestation - setStatus - 4/5 committees join	668.46 us/op	700.23 us/op	0.95
altair processAttestation - setStatus - 100% committees join	772.55 us/op	816.27 us/op	0.95
altair processBlock - 250000 vs - 7PWei normalcase	19.088 ms/op	20.420 ms/op	0.93
altair processBlock - 250000 vs - 7PWei normalcase hashState	25.894 ms/op	29.318 ms/op	0.88
altair processBlock - 250000 vs - 7PWei worstcase	52.154 ms/op	57.429 ms/op	0.91
altair processBlock - 250000 vs - 7PWei worstcase hashState	72.248 ms/op	73.222 ms/op	0.99
phase0 processBlock - 250000 vs - 7PWei normalcase	2.1195 ms/op	2.5953 ms/op	0.82
phase0 processBlock - 250000 vs - 7PWei worstcase	29.216 ms/op	35.230 ms/op	0.83
altair processEth1Data - 250000 vs - 7PWei normalcase	438.35 us/op	518.58 us/op	0.85
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	7.5100 us/op	10.903 us/op	0.69
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	20.596 us/op	34.119 us/op	0.60
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	9.2630 us/op	11.826 us/op	0.78
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	6.8080 us/op	12.223 us/op	0.56
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	83.142 us/op	115.11 us/op	0.72
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	689.12 us/op	817.49 us/op	0.84
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	903.38 us/op	1.0734 ms/op	0.84
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	863.22 us/op	1.0256 ms/op	0.84
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	2.3425 ms/op	2.4016 ms/op	0.98
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	1.4685 ms/op	1.5264 ms/op	0.96
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	3.8525 ms/op	4.1674 ms/op	0.92
Tree 40 250000 create	307.63 ms/op	329.14 ms/op	0.93
Tree 40 250000 get(125000)	192.20 ns/op	199.48 ns/op	0.96
Tree 40 250000 set(125000)	1.0221 us/op	999.75 ns/op	1.02
Tree 40 250000 toArray()	23.617 ms/op	23.593 ms/op	1.00
Tree 40 250000 iterate all - toArray() + loop	23.896 ms/op	23.452 ms/op	1.02
Tree 40 250000 iterate all - get(i)	71.580 ms/op	79.619 ms/op	0.90
MutableVector 250000 create	9.2592 ms/op	14.075 ms/op	0.66
MutableVector 250000 get(125000)	6.3370 ns/op	6.6860 ns/op	0.95
MutableVector 250000 set(125000)	253.01 ns/op	246.02 ns/op	1.03
MutableVector 250000 toArray()	3.0090 ms/op	3.2736 ms/op	0.92
MutableVector 250000 iterate all - toArray() + loop	3.1369 ms/op	3.6863 ms/op	0.85
MutableVector 250000 iterate all - get(i)	1.5432 ms/op	1.5701 ms/op	0.98
Array 250000 create	2.6012 ms/op	2.7437 ms/op	0.95
Array 250000 clone - spread	1.1889 ms/op	1.1648 ms/op	1.02
Array 250000 get(125000)	0.59900 ns/op	0.57700 ns/op	1.04
Array 250000 set(125000)	0.66900 ns/op	0.66100 ns/op	1.01
Array 250000 iterate all - loop	112.47 us/op	84.473 us/op	1.33
effectiveBalanceIncrements clone Uint8Array 300000	31.349 us/op	40.900 us/op	0.77
effectiveBalanceIncrements clone MutableVector 300000	378.00 ns/op	345.00 ns/op	1.10
effectiveBalanceIncrements rw all Uint8Array 300000	169.86 us/op	165.95 us/op	1.02
effectiveBalanceIncrements rw all MutableVector 300000	87.473 ms/op	83.326 ms/op	1.05
phase0 afterProcessEpoch - 250000 vs - 7PWei	115.22 ms/op	121.18 ms/op	0.95
phase0 beforeProcessEpoch - 250000 vs - 7PWei	32.847 ms/op	43.997 ms/op	0.75
altair processEpoch - mainnet_e81889	339.52 ms/op	359.24 ms/op	0.95
mainnet_e81889 - altair beforeProcessEpoch	67.559 ms/op	70.733 ms/op	0.96
mainnet_e81889 - altair processJustificationAndFinalization	17.576 us/op	18.970 us/op	0.93
mainnet_e81889 - altair processInactivityUpdates	5.4854 ms/op	5.6098 ms/op	0.98
mainnet_e81889 - altair processRewardsAndPenalties	47.129 ms/op	56.410 ms/op	0.84
mainnet_e81889 - altair processRegistryUpdates	3.1110 us/op	3.4640 us/op	0.90
mainnet_e81889 - altair processSlashings	529.00 ns/op	486.00 ns/op	1.09
mainnet_e81889 - altair processEth1DataReset	592.00 ns/op	570.00 ns/op	1.04
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.2539 ms/op	1.4897 ms/op	0.84
mainnet_e81889 - altair processSlashingsReset	4.4110 us/op	7.7250 us/op	0.57
mainnet_e81889 - altair processRandaoMixesReset	4.7140 us/op	8.3060 us/op	0.57
mainnet_e81889 - altair processHistoricalRootsUpdate	1.0180 us/op	826.00 ns/op	1.23
mainnet_e81889 - altair processParticipationFlagUpdates	5.1950 us/op	2.9400 us/op	1.77
mainnet_e81889 - altair processSyncCommitteeUpdates	712.00 ns/op	742.00 ns/op	0.96
mainnet_e81889 - altair afterProcessEpoch	125.76 ms/op	128.65 ms/op	0.98
phase0 processEpoch - mainnet_e58758	364.12 ms/op	383.47 ms/op	0.95
mainnet_e58758 - phase0 beforeProcessEpoch	137.17 ms/op	122.77 ms/op	1.12
mainnet_e58758 - phase0 processJustificationAndFinalization	21.355 us/op	18.406 us/op	1.16
mainnet_e58758 - phase0 processRewardsAndPenalties	61.820 ms/op	70.512 ms/op	0.88
mainnet_e58758 - phase0 processRegistryUpdates	8.5030 us/op	10.539 us/op	0.81
mainnet_e58758 - phase0 processSlashings	551.00 ns/op	716.00 ns/op	0.77
mainnet_e58758 - phase0 processEth1DataReset	590.00 ns/op	574.00 ns/op	1.03
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.0242 ms/op	1.4671 ms/op	0.70
mainnet_e58758 - phase0 processSlashingsReset	3.5680 us/op	4.0500 us/op	0.88
mainnet_e58758 - phase0 processRandaoMixesReset	4.8990 us/op	4.9910 us/op	0.98
mainnet_e58758 - phase0 processHistoricalRootsUpdate	775.00 ns/op	1.0570 us/op	0.73
mainnet_e58758 - phase0 processParticipationRecordUpdates	4.5420 us/op	4.7610 us/op	0.95
mainnet_e58758 - phase0 afterProcessEpoch	97.830 ms/op	99.295 ms/op	0.99
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.2057 ms/op	1.9899 ms/op	0.61
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.5648 ms/op	1.9327 ms/op	0.81
altair processInactivityUpdates - 250000 normalcase	19.602 ms/op	27.574 ms/op	0.71
altair processInactivityUpdates - 250000 worstcase	25.885 ms/op	26.069 ms/op	0.99
phase0 processRegistryUpdates - 250000 normalcase	10.095 us/op	7.3860 us/op	1.37
phase0 processRegistryUpdates - 250000 badcase_full_deposits	262.12 us/op	358.82 us/op	0.73
phase0 processRegistryUpdates - 250000 worstcase 0.5	125.14 ms/op	136.70 ms/op	0.92
altair processRewardsAndPenalties - 250000 normalcase	69.557 ms/op	70.418 ms/op	0.99
altair processRewardsAndPenalties - 250000 worstcase	69.459 ms/op	75.895 ms/op	0.92
phase0 getAttestationDeltas - 250000 normalcase	6.8524 ms/op	6.8624 ms/op	1.00
phase0 getAttestationDeltas - 250000 worstcase	6.9679 ms/op	7.3051 ms/op	0.95
phase0 processSlashings - 250000 worstcase	3.3931 ms/op	3.4628 ms/op	0.98
altair processSyncCommitteeUpdates - 250000	188.53 ms/op	183.67 ms/op	1.03
BeaconState.hashTreeRoot - No change	341.00 ns/op	263.00 ns/op	1.30
BeaconState.hashTreeRoot - 1 full validator	53.477 us/op	60.209 us/op	0.89
BeaconState.hashTreeRoot - 32 full validator	568.99 us/op	605.06 us/op	0.94
BeaconState.hashTreeRoot - 512 full validator	5.1648 ms/op	5.1265 ms/op	1.01
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	62.489 us/op	64.525 us/op	0.97
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	963.59 us/op	980.71 us/op	0.98
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	11.839 ms/op	12.848 ms/op	0.92
BeaconState.hashTreeRoot - 1 balances	50.798 us/op	51.717 us/op	0.98
BeaconState.hashTreeRoot - 32 balances	454.63 us/op	495.73 us/op	0.92
BeaconState.hashTreeRoot - 512 balances	4.5845 ms/op	4.6539 ms/op	0.99
BeaconState.hashTreeRoot - 250000 balances	76.535 ms/op	77.420 ms/op	0.99
aggregationBits - 2048 els - zipIndexesInBitList	15.738 us/op	16.998 us/op	0.93
regular array get 100000 times	33.609 us/op	33.518 us/op	1.00
wrappedArray get 100000 times	33.641 us/op	33.484 us/op	1.00
arrayWithProxy get 100000 times	15.720 ms/op	16.100 ms/op	0.98
ssz.Root.equals	568.00 ns/op	560.00 ns/op	1.01
byteArrayEquals	564.00 ns/op	545.00 ns/op	1.03
shuffle list - 16384 els	7.0880 ms/op	7.0629 ms/op	1.00
shuffle list - 250000 els	102.13 ms/op	102.36 ms/op	1.00
processSlot - 1 slots	8.7880 us/op	9.9050 us/op	0.89
processSlot - 32 slots	1.4247 ms/op	1.6444 ms/op	0.87
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	36.384 ms/op	39.456 ms/op	0.92
getCommitteeAssignments - req 1 vs - 250000 vc	3.0129 ms/op	2.9502 ms/op	1.02
getCommitteeAssignments - req 100 vs - 250000 vc	4.5841 ms/op	4.1630 ms/op	1.10
getCommitteeAssignments - req 1000 vs - 250000 vc	4.6481 ms/op	4.5634 ms/op	1.02
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	5.2500 ns/op	4.7300 ns/op	1.11
state getBlockRootAtSlot - 250000 vs - 7PWei	1.0639 us/op	581.21 ns/op	1.83
computeProposers - vc 250000	11.293 ms/op	10.585 ms/op	1.07
computeEpochShuffling - vc 250000	106.09 ms/op	103.25 ms/op	1.03
getNextSyncCommittee - vc 250000	199.34 ms/op	176.92 ms/op	1.13
computeSigningRoot for AttestationData	13.733 us/op	14.575 us/op	0.94
hash AttestationData serialized data then Buffer.toString(base64)	2.6543 us/op	2.6368 us/op	1.01
toHexString serialized data	1.1532 us/op	1.1622 us/op	0.99
Buffer.toString(base64)	348.70 ns/op	344.95 ns/op	1.01

by benchmarkbot/action

[dapplion]

We should deeply review the implications of adding this call here, sync is delicate. Also, what's somewhat concerning is that this is a different code path than other executionEngine consumers. It's possible that:

• block processing attempts to call executionEngine and errors because it's offline
• this getSyncStatus calls eth_chainId and succeeds, considering el online
  How to resolve this inconsistency? It can happen both ways

[nazarhussain]

@dapplion I explicitly checked for eth_chainId for ECONNREFUSED which can happen only if server is offline, not in case server respond error.

As far I knew it's the same executionEngine used by the block processing. If there is a some error say 500 or 400 range that's concerns to block processing, it still means that server is online.

[g11tech]

I would like to simplfy the implementation :

1. For offline/online/auth state latch on to httpClient (or wrap it), so any/all calls to the engine can be used to update state
2. For syncing/synced track the status and update (as you have correctly done this)

So for 1. would like track on level of httpClient

[wemeetagain]

LGTM

[g11tech]

post merge review LGTM 👍

[wemeetagain]

🎉 This PR is included in v1.10.0 🎉