Implement remote key manager API

[dapplion]

Motivation

The Keymanager API now has Remote Key Manager which are set of endpoints for key management of external keys.

DAppNode requires this API to be implemented to safely manage its interactions with the local web3signer.

Description

• Adds required routes to the keymanager API impl and routes declarations
• Wire with validator instance
• Decide in which format to persist remote signer declarations
• Ensure imported remote signers are consumed after restarting the validator

Blocked by / related

• Keymanager server does not persist keystore password #4105
• Deprecate accounts CLI #4104

Closes #3886

[g11tech]

LGTM! 🚀

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: f2122cc	Previous: 0553523	Ratio
altair processAttestation - 250000 vs - 7PWei normalcase	3.5126 ms/op	3.3950 ms/op	1.03
altair processAttestation - 250000 vs - 7PWei worstcase	5.4621 ms/op	5.2224 ms/op	1.05
altair processAttestation - setStatus - 1/6 committees join	204.40 us/op	174.42 us/op	1.17
altair processAttestation - setStatus - 1/3 committees join	394.21 us/op	346.73 us/op	1.14
altair processAttestation - setStatus - 1/2 committees join	547.29 us/op	494.91 us/op	1.11
altair processAttestation - setStatus - 2/3 committees join	706.51 us/op	651.55 us/op	1.08
altair processAttestation - setStatus - 4/5 committees join	972.81 us/op	914.56 us/op	1.06
altair processAttestation - setStatus - 100% committees join	1.1610 ms/op	1.1111 ms/op	1.04
altair processBlock - 250000 vs - 7PWei normalcase	24.144 ms/op	22.687 ms/op	1.06
altair processBlock - 250000 vs - 7PWei normalcase hashState	33.847 ms/op	32.431 ms/op	1.04
altair processBlock - 250000 vs - 7PWei worstcase	76.747 ms/op	79.363 ms/op	0.97
altair processBlock - 250000 vs - 7PWei worstcase hashState	92.579 ms/op	95.502 ms/op	0.97
phase0 processBlock - 250000 vs - 7PWei normalcase	3.5415 ms/op	3.3161 ms/op	1.07
phase0 processBlock - 250000 vs - 7PWei worstcase	45.881 ms/op	51.755 ms/op	0.89
altair processEth1Data - 250000 vs - 7PWei normalcase	736.17 us/op	701.65 us/op	1.05
Tree 40 250000 create	778.66 ms/op	755.25 ms/op	1.03
Tree 40 250000 get(125000)	285.42 ns/op	239.47 ns/op	1.19
Tree 40 250000 set(125000)	2.3978 us/op	2.3731 us/op	1.01
Tree 40 250000 toArray()	31.777 ms/op	28.723 ms/op	1.11
Tree 40 250000 iterate all - toArray() + loop	31.735 ms/op	29.007 ms/op	1.09
Tree 40 250000 iterate all - get(i)	112.33 ms/op	121.69 ms/op	0.92
MutableVector 250000 create	15.025 ms/op	12.713 ms/op	1.18
MutableVector 250000 get(125000)	14.765 ns/op	10.851 ns/op	1.36
MutableVector 250000 set(125000)	620.98 ns/op	480.47 ns/op	1.29
MutableVector 250000 toArray()	6.9424 ms/op	5.3778 ms/op	1.29
MutableVector 250000 iterate all - toArray() + loop	7.2173 ms/op	6.4793 ms/op	1.11
MutableVector 250000 iterate all - get(i)	3.6676 ms/op	2.5926 ms/op	1.41
Array 250000 create	5.9874 ms/op	5.8754 ms/op	1.02
Array 250000 clone - spread	2.6571 ms/op	2.2909 ms/op	1.16
Array 250000 get(125000)	1.1700 ns/op	1.2540 ns/op	0.93
Array 250000 set(125000)	1.1400 ns/op	1.2410 ns/op	0.92
Array 250000 iterate all - loop	167.89 us/op	151.02 us/op	1.11
effectiveBalanceIncrements clone Uint8Array 300000	358.57 us/op	59.480 us/op	6.03
effectiveBalanceIncrements clone MutableVector 300000	722.00 ns/op	667.00 ns/op	1.08
effectiveBalanceIncrements rw all Uint8Array 300000	252.54 us/op	247.35 us/op	1.02
effectiveBalanceIncrements rw all MutableVector 300000	160.98 ms/op	138.74 ms/op	1.16
phase0 afterProcessEpoch - 250000 vs - 7PWei	180.49 ms/op	186.96 ms/op	0.97
phase0 beforeProcessEpoch - 250000 vs - 7PWei	68.183 ms/op	56.784 ms/op	1.20
altair processEpoch - mainnet_e81889	560.73 ms/op	533.56 ms/op	1.05
mainnet_e81889 - altair beforeProcessEpoch	136.93 ms/op	121.26 ms/op	1.13
mainnet_e81889 - altair processJustificationAndFinalization	19.862 us/op	17.060 us/op	1.16
mainnet_e81889 - altair processInactivityUpdates	11.493 ms/op	9.4223 ms/op	1.22
mainnet_e81889 - altair processRewardsAndPenalties	85.225 ms/op	112.68 ms/op	0.76
mainnet_e81889 - altair processRegistryUpdates	3.4790 us/op	2.5380 us/op	1.37
mainnet_e81889 - altair processSlashings	750.00 ns/op	650.00 ns/op	1.15
mainnet_e81889 - altair processEth1DataReset	804.00 ns/op	653.00 ns/op	1.23
mainnet_e81889 - altair processEffectiveBalanceUpdates	2.4411 ms/op	2.2166 ms/op	1.10
mainnet_e81889 - altair processSlashingsReset	5.5160 us/op	5.1710 us/op	1.07
mainnet_e81889 - altair processRandaoMixesReset	6.0230 us/op	4.9500 us/op	1.22
mainnet_e81889 - altair processHistoricalRootsUpdate	858.00 ns/op	638.00 ns/op	1.34
mainnet_e81889 - altair processParticipationFlagUpdates	2.6370 us/op	3.4230 us/op	0.77
mainnet_e81889 - altair processSyncCommitteeUpdates	637.00 ns/op	562.00 ns/op	1.13
mainnet_e81889 - altair afterProcessEpoch	193.54 ms/op	197.98 ms/op	0.98
phase0 processEpoch - mainnet_e58758	506.38 ms/op	473.13 ms/op	1.07
mainnet_e58758 - phase0 beforeProcessEpoch	192.27 ms/op	174.52 ms/op	1.10
mainnet_e58758 - phase0 processJustificationAndFinalization	19.936 us/op	17.787 us/op	1.12
mainnet_e58758 - phase0 processRewardsAndPenalties	69.196 ms/op	109.13 ms/op	0.63
mainnet_e58758 - phase0 processRegistryUpdates	12.434 us/op	7.9790 us/op	1.56
mainnet_e58758 - phase0 processSlashings	906.00 ns/op	619.00 ns/op	1.46
mainnet_e58758 - phase0 processEth1DataReset	848.00 ns/op	607.00 ns/op	1.40
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.9965 ms/op	1.9078 ms/op	1.05
mainnet_e58758 - phase0 processSlashingsReset	5.3790 us/op	4.0650 us/op	1.32
mainnet_e58758 - phase0 processRandaoMixesReset	6.1850 us/op	4.3260 us/op	1.43
mainnet_e58758 - phase0 processHistoricalRootsUpdate	764.00 ns/op	726.00 ns/op	1.05
mainnet_e58758 - phase0 processParticipationRecordUpdates	5.0470 us/op	3.6010 us/op	1.40
mainnet_e58758 - phase0 afterProcessEpoch	160.29 ms/op	163.70 ms/op	0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase	2.9357 ms/op	2.3805 ms/op	1.23
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	3.2058 ms/op	2.1641 ms/op	1.48
altair processInactivityUpdates - 250000 normalcase	25.160 ms/op	31.848 ms/op	0.79
altair processInactivityUpdates - 250000 worstcase	25.328 ms/op	26.520 ms/op	0.96
phase0 processRegistryUpdates - 250000 normalcase	8.1850 us/op	6.8180 us/op	1.20
phase0 processRegistryUpdates - 250000 badcase_full_deposits	465.87 us/op	367.86 us/op	1.27
phase0 processRegistryUpdates - 250000 worstcase 0.5	203.99 ms/op	190.84 ms/op	1.07
altair processRewardsAndPenalties - 250000 normalcase	112.50 ms/op	98.643 ms/op	1.14
altair processRewardsAndPenalties - 250000 worstcase	106.85 ms/op	91.652 ms/op	1.17
phase0 getAttestationDeltas - 250000 normalcase	12.317 ms/op	11.457 ms/op	1.08
phase0 getAttestationDeltas - 250000 worstcase	12.514 ms/op	11.824 ms/op	1.06
phase0 processSlashings - 250000 worstcase	5.3337 ms/op	5.0788 ms/op	1.05
altair processSyncCommitteeUpdates - 250000	280.54 ms/op	299.68 ms/op	0.94
BeaconState.hashTreeRoot - No change	574.00 ns/op	608.00 ns/op	0.94
BeaconState.hashTreeRoot - 1 full validator	58.397 us/op	74.145 us/op	0.79
BeaconState.hashTreeRoot - 32 full validator	623.16 us/op	656.30 us/op	0.95
BeaconState.hashTreeRoot - 512 full validator	6.0707 ms/op	7.4537 ms/op	0.81
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	78.886 us/op	91.492 us/op	0.86
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.0812 ms/op	1.3086 ms/op	0.83
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	14.914 ms/op	17.319 ms/op	0.86
BeaconState.hashTreeRoot - 1 balances	62.154 us/op	70.700 us/op	0.88
BeaconState.hashTreeRoot - 32 balances	611.35 us/op	665.31 us/op	0.92
BeaconState.hashTreeRoot - 512 balances	4.9569 ms/op	6.5141 ms/op	0.76
BeaconState.hashTreeRoot - 250000 balances	84.636 ms/op	98.522 ms/op	0.86
aggregationBits - 2048 els - zipIndexesInBitList	36.402 us/op	26.704 us/op	1.36
regular array get 100000 times	67.421 us/op	62.074 us/op	1.09
wrappedArray get 100000 times	67.390 us/op	60.897 us/op	1.11
arrayWithProxy get 100000 times	30.503 ms/op	28.749 ms/op	1.06
ssz.Root.equals	541.00 ns/op	573.00 ns/op	0.94
byteArrayEquals	531.00 ns/op	562.00 ns/op	0.94
shuffle list - 16384 els	11.105 ms/op	11.309 ms/op	0.98
shuffle list - 250000 els	163.68 ms/op	167.53 ms/op	0.98
processSlot - 1 slots	12.812 us/op	14.255 us/op	0.90
processSlot - 32 slots	1.8687 ms/op	2.0200 ms/op	0.93
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	390.08 us/op	384.73 us/op	1.01
getCommitteeAssignments - req 1 vs - 250000 vc	5.3319 ms/op	5.3995 ms/op	0.99
getCommitteeAssignments - req 100 vs - 250000 vc	7.3106 ms/op	7.7864 ms/op	0.94
getCommitteeAssignments - req 1000 vs - 250000 vc	7.7749 ms/op	8.5024 ms/op	0.91
computeProposers - vc 250000	18.489 ms/op	18.881 ms/op	0.98
computeEpochShuffling - vc 250000	165.88 ms/op	170.25 ms/op	0.97
getNextSyncCommittee - vc 250000	274.51 ms/op	289.30 ms/op	0.95
pass gossip attestations to forkchoice per slot	5.9823 ms/op	2.8760 ms/op	2.08
computeDeltas	3.1907 ms/op	3.3541 ms/op	0.95
computeProposerBoostScoreFromBalances	907.61 us/op	813.67 us/op	1.12
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	2.2587 ms/op	2.0554 ms/op	1.10
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	78.936 us/op	67.019 us/op	1.18
BLS verify - blst-native	1.8638 ms/op	2.1792 ms/op	0.86
BLS verifyMultipleSignatures 3 - blst-native	3.8101 ms/op	4.4821 ms/op	0.85
BLS verifyMultipleSignatures 8 - blst-native	8.2077 ms/op	9.6806 ms/op	0.85
BLS verifyMultipleSignatures 32 - blst-native	29.811 ms/op	35.205 ms/op	0.85
BLS aggregatePubkeys 32 - blst-native	39.549 us/op	46.823 us/op	0.84
BLS aggregatePubkeys 128 - blst-native	153.33 us/op	182.86 us/op	0.84
getAttestationsForBlock	60.864 ms/op	55.715 ms/op	1.09
isKnown best case - 1 super set check	439.00 ns/op	489.00 ns/op	0.90
isKnown normal case - 2 super set checks	434.00 ns/op	480.00 ns/op	0.90
isKnown worse case - 16 super set checks	432.00 ns/op	481.00 ns/op	0.90
CheckpointStateCache - add get delete	11.052 us/op	11.128 us/op	0.99
validate gossip signedAggregateAndProof - struct	4.2631 ms/op	5.0105 ms/op	0.85
validate gossip attestation - struct	2.0273 ms/op	2.3630 ms/op	0.86
altair verifyImport mainnet_s3766816:31	6.2938 s/op	6.9971 s/op	0.90
pickEth1Vote - no votes	2.1016 ms/op	2.1377 ms/op	0.98
pickEth1Vote - max votes	25.810 ms/op	21.977 ms/op	1.17
pickEth1Vote - Eth1Data hashTreeRoot value x2048	12.079 ms/op	13.400 ms/op	0.90
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	21.586 ms/op	21.728 ms/op	0.99
pickEth1Vote - Eth1Data fastSerialize value x2048	1.6364 ms/op	1.4670 ms/op	1.12
pickEth1Vote - Eth1Data fastSerialize tree x2048	18.623 ms/op	16.601 ms/op	1.12
bytes32 toHexString	1.1050 us/op	978.00 ns/op	1.13
bytes32 Buffer.toString(hex)	713.00 ns/op	773.00 ns/op	0.92
bytes32 Buffer.toString(hex) from Uint8Array	862.00 ns/op	1.0280 us/op	0.84
bytes32 Buffer.toString(hex) + 0x	739.00 ns/op	777.00 ns/op	0.95
Object access 1 prop	0.38000 ns/op	0.35800 ns/op	1.06
Map access 1 prop	0.30200 ns/op	0.33600 ns/op	0.90
Object get x1000	18.036 ns/op	10.853 ns/op	1.66
Map get x1000	0.97800 ns/op	0.95800 ns/op	1.02
Object set x1000	123.39 ns/op	70.545 ns/op	1.75
Map set x1000	73.969 ns/op	49.709 ns/op	1.49
Return object 10000 times	0.37390 ns/op	0.43930 ns/op	0.85
Throw Error 10000 times	5.8918 us/op	6.0291 us/op	0.98
enrSubnets - fastDeserialize 64 bits	2.7340 us/op	2.6910 us/op	1.02
enrSubnets - ssz BitVector 64 bits	786.00 ns/op	813.00 ns/op	0.97
enrSubnets - fastDeserialize 4 bits	404.00 ns/op	364.00 ns/op	1.11
enrSubnets - ssz BitVector 4 bits	801.00 ns/op	845.00 ns/op	0.95
prioritizePeers score -10:0 att 32-0.1 sync 2-0	95.018 us/op	84.730 us/op	1.12
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	135.72 us/op	127.41 us/op	1.07
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	225.34 us/op	223.99 us/op	1.01
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	387.34 us/op	334.79 us/op	1.16
prioritizePeers score 0:0 att 64-1 sync 4-1	459.49 us/op	407.23 us/op	1.13
RateTracker 1000000 limit, 1 obj count per request	191.23 ns/op	188.06 ns/op	1.02
RateTracker 1000000 limit, 2 obj count per request	143.41 ns/op	136.60 ns/op	1.05
RateTracker 1000000 limit, 4 obj count per request	121.67 ns/op	112.42 ns/op	1.08
RateTracker 1000000 limit, 8 obj count per request	109.28 ns/op	98.832 ns/op	1.11
RateTracker with prune	4.5170 us/op	3.8450 us/op	1.17
array of 16000 items push then shift	3.1794 us/op	51.575 us/op	0.06
LinkedList of 16000 items push then shift	25.448 ns/op	16.899 ns/op	1.51
array of 16000 items push then pop	250.85 ns/op	192.39 ns/op	1.30
LinkedList of 16000 items push then pop	20.951 ns/op	14.295 ns/op	1.47
array of 24000 items push then shift	4.5536 us/op	77.340 us/op	0.06
LinkedList of 24000 items push then shift	25.447 ns/op	20.454 ns/op	1.24
array of 24000 items push then pop	210.59 ns/op	173.46 ns/op	1.21
LinkedList of 24000 items push then pop	21.230 ns/op	15.794 ns/op	1.34
intersect bitArray bitLen 8	11.651 ns/op	10.747 ns/op	1.08
intersect array and set length 8	167.97 ns/op	133.54 ns/op	1.26
intersect bitArray bitLen 128	72.131 ns/op	57.851 ns/op	1.25
intersect array and set length 128	2.2653 us/op	1.8760 us/op	1.21

by benchmarkbot/action

[wemeetagain]

Lgtm