Validator db read/write ops too high on first startup with empty db

[nflaig]

Describe the bug

Validator db read/write ops too high on first startup with empty db which causes I/O lag and instability in VC. After one or two hours the process stabilizes and read/write ops per second drastically decrease.

Expected behavior

VC should not be unstable for one or two hours if started with an empty db.

Steps to Reproduce

Run VC with a lot of keys (500+) and make sure validator-db is empty. If VC was active before and db was removed, be careful to not get slashed as this also removes the slashing protection db.

Metrics

[dapplion]

Can you paste the promquery code for the first metrics screen capture?

[nflaig]

I just changed the prom query to use the vc metrics, the rate interval is 1m, would probably have to add bucket labels to further investigate

• rate(validator_db_read_items_total[$rate_interval]) (top left)
• rate(validator_db_write_items_total[$rate_interval]) (top right)
• rate( validator_db_read_req_total [$rate_interval]) (bottom left)
• rate( validator_db_write_req_total [$rate_interval]) (bottol right)

[nflaig]

@dapplion I found the problem, the huge amount of read/writes it due to minMaxSurround.ts. this.maxEpochLookback is set to Infinity meaning unitlEpoch will always be 0. This results in a db read and a bulk write for each validator pubkey from current epoch until genesis.

I have to dig into how min-max surround works, likely need to use another value for maxEpochLookback.

[nflaig]

Based on Detecting slashing conditions, it seems like maxEpochLookback only needs to be the number of epochs in weak subjectivity period. More details in Scanning for surrounding votes section.

[nflaig]

Slashers default to a period of 4096 epochs, which means could be fine to only set maxEpochLookback to that period. They might still collect slashable attestations over a longer period, meaning limiting the max lookback might be riksy when using min-max approach.

Maybe could consider setting maxEpochLookback to a practically safe enough value to fix the I/O issue for now.

Alternative would be to implement a different slashing protection strategy like "minimal"/high-watermark strategy as suggested in Eth R&D discord.

Edit: It looks like this strategy is also already implemented on top of min-max surround

lodestar/packages/validator/src/slashingProtection/attestation/index.ts

Lines 99 to 103 in b60b43c

	// Refuse to sign any attestation with:
	// - source.epoch < min(att.source_epoch for att in data.signed_attestations if att.pubkey == attester_pubkey), OR
	// - target_epoch <= min(att.target_epoch for att in data.signed_attestations if att.pubkey == attester_pubkey)
	// (spec v4, Slashing Protection Database Interchange Format)
	const attestationLowerBound = await this.attestationLowerBound.get(pubKey);

[dapplion]

Setting maxEpochLookback to 4096 sounds sensible. We can do that plus using the minimal strategy. That has the benefit of producing way smaller exports right?

[nflaig]

That has the benefit of producing way smaller exports right?

No, the slashing export size just depends on data in AttestationByTargetRepository, min-max spans are not exported. Setting maxEpochLookback to 4096 would fix the I/O issue and also massively reduce the validator db size. Right now we are creating ~200k (mainnet epochs) min span records per pubkey.

Setting maxEpochLookback to 4096 sounds sensible.

I think a better value would be weak subjectivity period epochs but we don't have access to that on the VC side. All of these values are quite theoretical though, so might also be fine with 4096 and possibly allow to override via CLI.

[nflaig]

That has the benefit of producing way smaller exports right?

Actually, right now it even affects the size of the slashing protection but this is a bug which will be fixed with #5437, it loads keys from other buckets which are not valid pubkeys, for those there will not be any signed_blocks or signed_attestations but still adds records to slashing protection.

The empty records with invalid pubkeys look like this

{
  "pubkey": "0x47454e455349535f54494d45",
  "signed_blocks": [],
  "signed_attestations": []
},
{
  "pubkey": "0x47454e455349535f56414c494441544f52535f524f4f54",
  "signed_blocks": [],
  "signed_attestations": []
}