Bad validator performance in electra

[twoeths]

Describe the bug

This is validator performance in one of our node in electra

the main error there is "no_aggregate_inclusion"

Expected behavior

same attestation performance like pre-electra

Steps to reproduce

No response

Additional context

this should be fixed by #7543

Operating system

Linux

Lodestar version or commit hash

v1.27.1

[twoeths]

the miss ratio really affected, note this for references later

[twoeths]

it took a lot of time to investigate this issue. Need to add metrics to track reason of reject/ignore and add to Grafana dashboards

[twoeths]

another metric for later reference, that's the oppool size. If we see this it means we have issues somewhere

[twoeths]

and this Produced Aggregate Participants Histogram is the metric how @nflaig found this issue originally