Attestation inclusion is slow on Medalla

[benjaminion]

Description

As a staker, I want my attestations included quickly so that I make maximum reward.

beaconcha.in has implemented an Attestation Effectiveness metric based on this model. This surfaces the detailed analysis of rewards and penalties I did on Altona.

Where identifiable, Prysm and Lighthouse validators seem to be consistently in the 80-90% effective range, whereas Teku validators are consistently around 40-50%. This appears to be solely due to delays in getting attestations included, not due to missing attestations.

Example. Click Attestations to see the inclusion distance (0 distance means in the slot immediately after the block).

Acceptance Criteria

• Attestation effectiveness at least as high as Prysm and Lighthouse

[ajsutton]

#2637 is likely to help improve the attestation inclusion rate. Deployed to our validators and will watch to see the results.

[ajsutton]

#2637 did significantly improve things but #2661 really won't be helping this.

[ajsutton]

Last enhancement to put in was flood publishing which went in with #2717. Attestation effectiveness is now looking really good and any further ideas on improving it can be done in their own tickets.