Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(slashing-protection): limit min-max surround epoch lookback to 4096 #5454

Merged
merged 1 commit into from May 3, 2023

Conversation

nflaig
Copy link
Member

@nflaig nflaig commented May 2, 2023

Motivation

Closes #5356

Description

Limits number of epochs in the past to check for surrounding attestations (min-span) to 4096.

DB size

Running with 2k keys, the DB size was initially 4GB and is now reduced to 100MB.

Metrics

These are the metrics when running a VC with 2k keys and an empty db.

There are still quite a lot of read/writes but it goes down very fast compared to initially and does no longer cause instability in VC.

image

Request times do not show any I/O lag, previously was running into timeouts (10 seconds).

image

For the first 1-2 minutes there is still a increased step times diff but it goes down very fast, after a few minutes instead of hours as before.

image

Metrics look good to me now, there are no big issues and also need to keep in mind that those metrics are collected running 2k validators which is not a normal setup. This also only happens if the db is empty, meaning for most setups only on initial installation.

* Number of epochs in the past to check for surrounding attestations.
*
* This value can be limited to a reasonable high amount as Lodestar does not solely rely on this strategy but also
* implements the minimal strategy which has been formally proven to be safe (https://github.com/michaelsproul/slashing-proofs).
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minimal strategy refers to conditions 4 and 5 from EIP-3076 and is implemented in Lodestar

// Refuse to sign any attestation with:
// - source.epoch < min(att.source_epoch for att in data.signed_attestations if att.pubkey == attester_pubkey), OR
// - target_epoch <= min(att.target_epoch for att in data.signed_attestations if att.pubkey == attester_pubkey)
// (spec v4, Slashing Protection Database Interchange Format)
const attestationLowerBound = await this.attestationLowerBound.get(pubKey);
if (attestationLowerBound) {
const {minSourceEpoch, minTargetEpoch} = attestationLowerBound;
if (attestation.sourceEpoch < minSourceEpoch) {
throw new InvalidAttestationError({
code: InvalidAttestationErrorCode.SOURCE_LESS_THAN_LOWER_BOUND,
sourceEpoch: attestation.sourceEpoch,
minSourceEpoch,
});
}
if (attestation.targetEpoch <= minTargetEpoch) {
throw new InvalidAttestationError({
code: InvalidAttestationErrorCode.TARGET_LESS_THAN_OR_EQ_LOWER_BOUND,
targetEpoch: attestation.targetEpoch,
minTargetEpoch,
});
}
}

* on first startup with an empty db. See https://github.com/ChainSafe/lodestar/issues/5356 for more details on the issue.
*
* The value 4096 has been chosen as it is the default used by slashers (https://lighthouse-book.sigmaprime.io/slasher.html#history-length)
* and is generally higher than the weak subjectivity period. However, it would still be risky if we just relied on min-max surround
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, the weak subjectivity period for goerli is 3686 epochs and for mainnet 3442 epochs.

@github-actions
Copy link
Contributor

github-actions bot commented May 2, 2023

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: baad925 Previous: d835c8c Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 988.17 us/op 571.40 us/op 1.73
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 68.269 us/op 45.329 us/op 1.51
BLS verify - blst-native 1.3133 ms/op 1.2293 ms/op 1.07
BLS verifyMultipleSignatures 3 - blst-native 2.6038 ms/op 2.4818 ms/op 1.05
BLS verifyMultipleSignatures 8 - blst-native 5.6050 ms/op 5.3084 ms/op 1.06
BLS verifyMultipleSignatures 32 - blst-native 20.388 ms/op 18.979 ms/op 1.07
BLS aggregatePubkeys 32 - blst-native 27.410 us/op 25.925 us/op 1.06
BLS aggregatePubkeys 128 - blst-native 106.36 us/op 101.56 us/op 1.05
getAttestationsForBlock 91.726 ms/op 56.021 ms/op 1.64
isKnown best case - 1 super set check 291.00 ns/op 256.00 ns/op 1.14
isKnown normal case - 2 super set checks 285.00 ns/op 255.00 ns/op 1.12
isKnown worse case - 16 super set checks 261.00 ns/op 252.00 ns/op 1.04
CheckpointStateCache - add get delete 6.8730 us/op 5.1170 us/op 1.34
validate gossip signedAggregateAndProof - struct 3.0591 ms/op 2.8275 ms/op 1.08
validate gossip attestation - struct 1.3926 ms/op 1.3541 ms/op 1.03
pickEth1Vote - no votes 1.4662 ms/op 1.2605 ms/op 1.16
pickEth1Vote - max votes 13.503 ms/op 11.477 ms/op 1.18
pickEth1Vote - Eth1Data hashTreeRoot value x2048 10.725 ms/op 9.2133 ms/op 1.16
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 18.762 ms/op 19.569 ms/op 0.96
pickEth1Vote - Eth1Data fastSerialize value x2048 877.37 us/op 676.64 us/op 1.30
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.6372 ms/op 8.9303 ms/op 0.74
bytes32 toHexString 782.00 ns/op 497.00 ns/op 1.57
bytes32 Buffer.toString(hex) 439.00 ns/op 364.00 ns/op 1.21
bytes32 Buffer.toString(hex) from Uint8Array 671.00 ns/op 568.00 ns/op 1.18
bytes32 Buffer.toString(hex) + 0x 460.00 ns/op 348.00 ns/op 1.32
Object access 1 prop 0.21500 ns/op 0.16200 ns/op 1.33
Map access 1 prop 0.17100 ns/op 0.15300 ns/op 1.12
Object get x1000 8.1320 ns/op 6.6350 ns/op 1.23
Map get x1000 0.74100 ns/op 0.61200 ns/op 1.21
Object set x1000 79.031 ns/op 53.324 ns/op 1.48
Map set x1000 57.825 ns/op 45.492 ns/op 1.27
Return object 10000 times 0.26350 ns/op 0.23600 ns/op 1.12
Throw Error 10000 times 4.5080 us/op 4.2968 us/op 1.05
fastMsgIdFn sha256 / 200 bytes 3.7260 us/op 3.5720 us/op 1.04
fastMsgIdFn h32 xxhash / 200 bytes 333.00 ns/op 299.00 ns/op 1.11
fastMsgIdFn h64 xxhash / 200 bytes 510.00 ns/op 444.00 ns/op 1.15
fastMsgIdFn sha256 / 1000 bytes 12.344 us/op 12.408 us/op 0.99
fastMsgIdFn h32 xxhash / 1000 bytes 484.00 ns/op 410.00 ns/op 1.18
fastMsgIdFn h64 xxhash / 1000 bytes 582.00 ns/op 467.00 ns/op 1.25
fastMsgIdFn sha256 / 10000 bytes 107.88 us/op 105.39 us/op 1.02
fastMsgIdFn h32 xxhash / 10000 bytes 2.1100 us/op 1.9170 us/op 1.10
fastMsgIdFn h64 xxhash / 10000 bytes 1.5310 us/op 1.3560 us/op 1.13
enrSubnets - fastDeserialize 64 bits 1.8210 us/op 1.3130 us/op 1.39
enrSubnets - ssz BitVector 64 bits 638.00 ns/op 502.00 ns/op 1.27
enrSubnets - fastDeserialize 4 bits 216.00 ns/op 168.00 ns/op 1.29
enrSubnets - ssz BitVector 4 bits 618.00 ns/op 515.00 ns/op 1.20
prioritizePeers score -10:0 att 32-0.1 sync 2-0 127.09 us/op 124.75 us/op 1.02
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 167.00 us/op 165.26 us/op 1.01
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 212.33 us/op 161.87 us/op 1.31
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 385.89 us/op 309.69 us/op 1.25
prioritizePeers score 0:0 att 64-1 sync 4-1 457.27 us/op 380.58 us/op 1.20
array of 16000 items push then shift 1.7346 us/op 1.6951 us/op 1.02
LinkedList of 16000 items push then shift 9.7080 ns/op 9.1390 ns/op 1.06
array of 16000 items push then pop 122.12 ns/op 89.125 ns/op 1.37
LinkedList of 16000 items push then pop 9.4020 ns/op 8.8940 ns/op 1.06
array of 24000 items push then shift 2.5473 us/op 2.4692 us/op 1.03
LinkedList of 24000 items push then shift 9.7400 ns/op 10.688 ns/op 0.91
array of 24000 items push then pop 94.791 ns/op 77.337 ns/op 1.23
LinkedList of 24000 items push then pop 9.2830 ns/op 8.9520 ns/op 1.04
intersect bitArray bitLen 8 14.341 ns/op 13.640 ns/op 1.05
intersect array and set length 8 106.60 ns/op 76.267 ns/op 1.40
intersect bitArray bitLen 128 47.759 ns/op 44.457 ns/op 1.07
intersect array and set length 128 1.4954 us/op 1.0625 us/op 1.41
Buffer.concat 32 items 3.8770 us/op 2.6940 us/op 1.44
Uint8Array.set 32 items 3.5300 us/op 3.0510 us/op 1.16
pass gossip attestations to forkchoice per slot 4.4927 ms/op 3.4857 ms/op 1.29
computeDeltas 4.3734 ms/op 3.6943 ms/op 1.18
computeProposerBoostScoreFromBalances 2.0133 ms/op 1.8645 ms/op 1.08
altair processAttestation - 250000 vs - 7PWei normalcase 3.5502 ms/op 2.5024 ms/op 1.42
altair processAttestation - 250000 vs - 7PWei worstcase 5.0540 ms/op 3.8765 ms/op 1.30
altair processAttestation - setStatus - 1/6 committees join 150.49 us/op 153.97 us/op 0.98
altair processAttestation - setStatus - 1/3 committees join 291.13 us/op 305.16 us/op 0.95
altair processAttestation - setStatus - 1/2 committees join 382.40 us/op 392.91 us/op 0.97
altair processAttestation - setStatus - 2/3 committees join 478.14 us/op 487.18 us/op 0.98
altair processAttestation - setStatus - 4/5 committees join 679.91 us/op 690.36 us/op 0.98
altair processAttestation - setStatus - 100% committees join 808.26 us/op 793.61 us/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase 22.725 ms/op 20.935 ms/op 1.09
altair processBlock - 250000 vs - 7PWei normalcase hashState 30.509 ms/op 31.357 ms/op 0.97
altair processBlock - 250000 vs - 7PWei worstcase 58.806 ms/op 61.625 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase hashState 77.072 ms/op 78.622 ms/op 0.98
phase0 processBlock - 250000 vs - 7PWei normalcase 2.8704 ms/op 2.1915 ms/op 1.31
phase0 processBlock - 250000 vs - 7PWei worstcase 35.296 ms/op 31.168 ms/op 1.13
altair processEth1Data - 250000 vs - 7PWei normalcase 757.02 us/op 490.93 us/op 1.54
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 13.337 us/op 9.5350 us/op 1.40
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 42.662 us/op 28.506 us/op 1.50
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 17.418 us/op 9.9190 us/op 1.76
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 15.833 us/op 9.3970 us/op 1.68
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 134.71 us/op 114.34 us/op 1.18
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 880.90 us/op 845.27 us/op 1.04
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.8966 ms/op 1.0843 ms/op 1.75
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.2432 ms/op 1.1244 ms/op 1.11
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 3.8820 ms/op 2.8698 ms/op 1.35
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.1390 ms/op 1.9502 ms/op 1.10
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 6.7779 ms/op 4.8816 ms/op 1.39
Tree 40 250000 create 881.84 ms/op 375.94 ms/op 2.35
Tree 40 250000 get(125000) 221.99 ns/op 203.73 ns/op 1.09
Tree 40 250000 set(125000) 2.7176 us/op 1.3145 us/op 2.07
Tree 40 250000 toArray() 29.878 ms/op 24.612 ms/op 1.21
Tree 40 250000 iterate all - toArray() + loop 29.092 ms/op 22.857 ms/op 1.27
Tree 40 250000 iterate all - get(i) 88.779 ms/op 81.869 ms/op 1.08
MutableVector 250000 create 19.313 ms/op 15.338 ms/op 1.26
MutableVector 250000 get(125000) 7.1260 ns/op 7.2690 ns/op 0.98
MutableVector 250000 set(125000) 536.83 ns/op 297.82 ns/op 1.80
MutableVector 250000 toArray() 4.4090 ms/op 3.9289 ms/op 1.12
MutableVector 250000 iterate all - toArray() + loop 4.1881 ms/op 3.5075 ms/op 1.19
MutableVector 250000 iterate all - get(i) 1.5806 ms/op 1.6052 ms/op 0.98
Array 250000 create 3.8735 ms/op 3.2673 ms/op 1.19
Array 250000 clone - spread 1.7099 ms/op 1.2842 ms/op 1.33
Array 250000 get(125000) 0.74700 ns/op 0.65000 ns/op 1.15
Array 250000 set(125000) 0.76800 ns/op 0.71500 ns/op 1.07
Array 250000 iterate all - loop 89.805 us/op 111.25 us/op 0.81
effectiveBalanceIncrements clone Uint8Array 300000 56.160 us/op 41.559 us/op 1.35
effectiveBalanceIncrements clone MutableVector 300000 505.00 ns/op 393.00 ns/op 1.28
effectiveBalanceIncrements rw all Uint8Array 300000 195.13 us/op 187.95 us/op 1.04
effectiveBalanceIncrements rw all MutableVector 300000 112.69 ms/op 92.542 ms/op 1.22
phase0 afterProcessEpoch - 250000 vs - 7PWei 126.95 ms/op 119.75 ms/op 1.06
phase0 beforeProcessEpoch - 250000 vs - 7PWei 47.468 ms/op 47.232 ms/op 1.01
altair processEpoch - mainnet_e81889 343.91 ms/op 351.68 ms/op 0.98
mainnet_e81889 - altair beforeProcessEpoch 72.895 ms/op 98.070 ms/op 0.74
mainnet_e81889 - altair processJustificationAndFinalization 18.488 us/op 19.424 us/op 0.95
mainnet_e81889 - altair processInactivityUpdates 7.2789 ms/op 7.7607 ms/op 0.94
mainnet_e81889 - altair processRewardsAndPenalties 77.790 ms/op 65.076 ms/op 1.20
mainnet_e81889 - altair processRegistryUpdates 3.0320 us/op 3.1920 us/op 0.95
mainnet_e81889 - altair processSlashings 490.00 ns/op 579.00 ns/op 0.85
mainnet_e81889 - altair processEth1DataReset 599.00 ns/op 618.00 ns/op 0.97
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3139 ms/op 1.5891 ms/op 0.83
mainnet_e81889 - altair processSlashingsReset 5.9710 us/op 9.3850 us/op 0.64
mainnet_e81889 - altair processRandaoMixesReset 12.367 us/op 4.9850 us/op 2.48
mainnet_e81889 - altair processHistoricalRootsUpdate 2.4940 us/op 675.00 ns/op 3.69
mainnet_e81889 - altair processParticipationFlagUpdates 6.7270 us/op 2.6420 us/op 2.55
mainnet_e81889 - altair processSyncCommitteeUpdates 1.4700 us/op 577.00 ns/op 2.55
mainnet_e81889 - altair afterProcessEpoch 146.67 ms/op 132.99 ms/op 1.10
phase0 processEpoch - mainnet_e58758 526.59 ms/op 407.93 ms/op 1.29
mainnet_e58758 - phase0 beforeProcessEpoch 181.64 ms/op 186.32 ms/op 0.97
mainnet_e58758 - phase0 processJustificationAndFinalization 27.218 us/op 24.382 us/op 1.12
mainnet_e58758 - phase0 processRewardsAndPenalties 71.913 ms/op 67.289 ms/op 1.07
mainnet_e58758 - phase0 processRegistryUpdates 12.236 us/op 8.7610 us/op 1.40
mainnet_e58758 - phase0 processSlashings 1.0470 us/op 577.00 ns/op 1.81
mainnet_e58758 - phase0 processEth1DataReset 982.00 ns/op 604.00 ns/op 1.63
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.3547 ms/op 1.3341 ms/op 1.02
mainnet_e58758 - phase0 processSlashingsReset 6.5520 us/op 3.5800 us/op 1.83
mainnet_e58758 - phase0 processRandaoMixesReset 10.129 us/op 4.9080 us/op 2.06
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.0660 us/op 677.00 ns/op 1.57
mainnet_e58758 - phase0 processParticipationRecordUpdates 7.9150 us/op 4.1640 us/op 1.90
mainnet_e58758 - phase0 afterProcessEpoch 103.88 ms/op 113.02 ms/op 0.92
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.0762 ms/op 1.3345 ms/op 1.56
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.5764 ms/op 2.9214 ms/op 0.88
altair processInactivityUpdates - 250000 normalcase 30.791 ms/op 26.146 ms/op 1.18
altair processInactivityUpdates - 250000 worstcase 32.327 ms/op 28.046 ms/op 1.15
phase0 processRegistryUpdates - 250000 normalcase 15.955 us/op 5.9500 us/op 2.68
phase0 processRegistryUpdates - 250000 badcase_full_deposits 399.48 us/op 280.20 us/op 1.43
phase0 processRegistryUpdates - 250000 worstcase 0.5 165.07 ms/op 141.47 ms/op 1.17
altair processRewardsAndPenalties - 250000 normalcase 78.621 ms/op 50.839 ms/op 1.55
altair processRewardsAndPenalties - 250000 worstcase 84.180 ms/op 65.531 ms/op 1.28
phase0 getAttestationDeltas - 250000 normalcase 10.742 ms/op 9.0786 ms/op 1.18
phase0 getAttestationDeltas - 250000 worstcase 10.897 ms/op 9.5845 ms/op 1.14
phase0 processSlashings - 250000 worstcase 4.5582 ms/op 3.9501 ms/op 1.15
altair processSyncCommitteeUpdates - 250000 211.46 ms/op 200.07 ms/op 1.06
BeaconState.hashTreeRoot - No change 293.00 ns/op 317.00 ns/op 0.92
BeaconState.hashTreeRoot - 1 full validator 55.322 us/op 61.446 us/op 0.90
BeaconState.hashTreeRoot - 32 full validator 574.27 us/op 523.29 us/op 1.10
BeaconState.hashTreeRoot - 512 full validator 7.1533 ms/op 6.4524 ms/op 1.11
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 69.431 us/op 69.562 us/op 1.00
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.0635 ms/op 1.0565 ms/op 1.01
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 15.382 ms/op 14.155 ms/op 1.09
BeaconState.hashTreeRoot - 1 balances 58.611 us/op 54.902 us/op 1.07
BeaconState.hashTreeRoot - 32 balances 524.33 us/op 547.36 us/op 0.96
BeaconState.hashTreeRoot - 512 balances 5.2540 ms/op 5.3989 ms/op 0.97
BeaconState.hashTreeRoot - 250000 balances 89.162 ms/op 81.249 ms/op 1.10
aggregationBits - 2048 els - zipIndexesInBitList 24.895 us/op 17.255 us/op 1.44
regular array get 100000 times 37.051 us/op 34.345 us/op 1.08
wrappedArray get 100000 times 36.335 us/op 44.533 us/op 0.82
arrayWithProxy get 100000 times 20.102 ms/op 15.953 ms/op 1.26
ssz.Root.equals 745.00 ns/op 565.00 ns/op 1.32
byteArrayEquals 753.00 ns/op 565.00 ns/op 1.33
shuffle list - 16384 els 8.2680 ms/op 7.2783 ms/op 1.14
shuffle list - 250000 els 122.67 ms/op 106.49 ms/op 1.15
processSlot - 1 slots 10.714 us/op 9.6920 us/op 1.11
processSlot - 32 slots 1.5530 ms/op 1.5472 ms/op 1.00
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 50.478 ms/op 37.733 ms/op 1.34
getCommitteeAssignments - req 1 vs - 250000 vc 3.0120 ms/op 3.0751 ms/op 0.98
getCommitteeAssignments - req 100 vs - 250000 vc 4.2820 ms/op 4.3772 ms/op 0.98
getCommitteeAssignments - req 1000 vs - 250000 vc 4.6619 ms/op 4.7543 ms/op 0.98
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.5400 ns/op 4.4300 ns/op 1.25
state getBlockRootAtSlot - 250000 vs - 7PWei 688.97 ns/op 914.22 ns/op 0.75
computeProposers - vc 250000 11.247 ms/op 12.439 ms/op 0.90
computeEpochShuffling - vc 250000 107.34 ms/op 111.59 ms/op 0.96
getNextSyncCommittee - vc 250000 186.14 ms/op 176.22 ms/op 1.06
computeSigningRoot for AttestationData 14.953 us/op 14.273 us/op 1.05
hash AttestationData serialized data then Buffer.toString(base64) 2.7116 us/op 2.6646 us/op 1.02
toHexString serialized data 1.2890 us/op 1.5501 us/op 0.83
Buffer.toString(base64) 365.76 ns/op 386.11 ns/op 0.95

by benchmarkbot/action

@nflaig nflaig force-pushed the nflaig/limit-max-epoch-lookback branch 2 times, most recently from 842dbea to f738b3c Compare May 2, 2023 09:13
@nflaig nflaig marked this pull request as ready for review May 2, 2023 09:14
@nflaig nflaig requested a review from a team as a code owner May 2, 2023 09:14
@nflaig nflaig force-pushed the nflaig/limit-max-epoch-lookback branch from f738b3c to d997504 Compare May 2, 2023 09:27
@dapplion dapplion merged commit 0ed1cd1 into unstable May 3, 2023
11 checks passed
@dapplion dapplion deleted the nflaig/limit-max-epoch-lookback branch May 3, 2023 09:02
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Validator db read/write ops too high on first startup with empty db
3 participants