Investigate why there are so few mesh peers per attestation subnet

[twoeths]

Describe the bug

This happens in both prater and mainnet: we have a lot of topic peers

but so few mesh peers

Expected behavior

• Given options.D = 8, we should have up to 8 mesh peers per subnet
• Track both mesh peers (already exist) and topic peers in the grafana dashboard

[twoeths]

when joining a topic, mesh is created with peers that have score >=0, so we should add gossip peer score metric.

[twoeths]

It shows that at some points we have so few peers at mesh score threshold (score >=0)

@wemeetagain what's the consequence of having so few mesh peers?

[twoeths]

Adrian at lighthouse mentioned that Most are positive and hit the max of 100. Rarely see negative scores at lighthouse, we need to investigate why peer score is so low in lodestar

[twoeths]

this could mitigate the issue #3705, should assess the situation after that's done

[twoeths]

#3781 helps us gain good gossip peer score, hence more mesh peers. This could help us mitigate the issue.

Also, we need to care about Job Wait Time. If it's >3s, peers potentially get penalized due to broken IWANT/IHAVE promises, this is from a node of 30 validators

[twoeths]

There are good number of mesh peers in mainnet nodes - v0.36.0

closing the issue