Increased Bandwidth Usage Since dencun Upgrade

[haoei]

Describe the bug
Since the recent upgrade of dencun, I have noticed a significant increase in bandwidth usage for the ssv-node program I am running. The bandwidth peaks at 70Mbit/s, with the majority being consumed by the 13001 p2p port.

To Reproduce
Use iftop -P to observe bandwidth usage. Wait for a while and check the total peak.

Expected behavior
I expect the bandwidth usage to be at the previous level, rather than the current 70Mbit/s.

Node (please complete the following information):

• Operator Version: v1.3.2
• BN Client: Prysm, Lighthouse
• Cloud service provider: Huawei Cloud

Additional context
I have already tried restarting the program and the system, but the issue persists.

[hwhe]

Yes, I found the same problem on my production environment.After the 3.13 upgrade, the P2P traffic is almost doubled, resulting in severe packet loss. The performance of ssv is also much lower than before.
I think this is a general question.need to be taken seriously. @y0sher

[lilkk-jerry]

same problem. any feedback?

[moshe-blox]

Thanks for letting us know, we're investigating this right now.

@haoei Can you please confirm that its solely the SSV node that is causing all of this traffic (70Mbit/s), and it's not a combined traffic of, for example, the SSV + CL + EL nodes together? This is at least an order of magnitude above what we've seen so far.

@hwhe @lilkk-jerry Can you please provide the network usage of one of your SSV nodes?

We have an upcoming upgrade planned that would reduce traffic tremendously.

With that said, we're now brainstorming what can be done in the short-term.

[hwhe]

@moshe-blox

you use iftop, can clearly see that more than 90% of the traffic is occupied by the ssv p2p port. very obvious。
This is the result after I sorted it by traffic。

ecs-8416:13001 <=> 18.237.5.4:13021 106MB 304Kb 2.27Mb 1.89Mb
ecs-8416:13001 <=> ns3216045.ip-141-95-202.eu:13001 109MB 314Kb 2.48Mb 1.88Mb
ecs-8416:13001 <=> ecs-119-8-172-250.compute.hwclouds-dns.com:13001 94.9MB 293Kb 2.08Mb 1.63Mb
ecs-8416:13001 <=> 18.237.5.4:13025 80.8MB 223Kb 2.00Mb 1.47Mb
ecs-8416:13001 <=> ecs-124-243-150-192.compute.hwclouds-dns.co:13001 86.3MB 169Kb 1.7578 1.43Mb
ecs-8416:13001 <=> 34.116.92.116:13000 81.5MB 111Kb 1.87Mb 1.42Mb
ecs-8416:13001 <=> 37.120.190.85:13001 28.4MB 88.0Kb 556Kb 515Kb
ecs-8416:13001 <=> ns31440147.ip-141-94-143.eu:13002 3001 3 79813.9MB 9.82Kb 517Kb 392b
ecs-8416:13001 <=> 103.106.58.1:13001 net.md:13000 :13001 18.3MB 203Kb 430Kb 339Kb
ecs-8416:13001 <=> 185-70-190-197.starnet.md:13000 15.3MB 15.6Kb 305Kb 321Kb
ecs-8416:13001 <=> mta27.greendeskservers.com:13001 1.75MB 15.6Kb 321Kb 265Kb
ecs-8416:13001 <=> 77.35.209.243.bcube.co.uk:12005 s-dns. m:30073 2.54MB 28.4Kb 447b 140Kb
ecs-8416:13001 <=> 75.98.207.226:13001 3Kb 161 7.99MB 12.2Kb 258Kb 175Kb
ecs-8416:13001 <=> 161.35.83.205:23101 co.uk:12005 7.83MB 80.8Kb 99.3Kb 169Kb
ecs-8416:13001 <=> 5.9.189.245:13001 5.20MB 10.8Kb 162Kb 162Kb
ecs-8416:13001 <=> 144.76.30.248:13001 6.43MB 16.7Kb 117Kb 129Kb
ecs-8416:13001 <=> 45.139.122.58:13001 s.com:30073 6.43MB 10.9Kb 139Kb 127Kb
ecs-8416:13001 <=> host.cloudsigma.net:13001 1.79MB 10.2Kb 165Kb 119Kb
ecs-8416:13001 <=> ns3179937.ip-51-210-209.eu:13001 4.85MB 13.2Kb 97.4Kb 89.4Kb
ecs-8416:13001 <=> static.180.71.21.65.clients.your-server.de:13001 4.86MB 12.6Kb 99.4Kb 88.1Kb
ecs-8416:13001 <=> static.91.114.181.135.clients.your-server.d:13001 758KB 4.93Kb 91.5Kb 65.3Kb
ecs-8416:13001 <=> static.207.100.181.135.clients.your-server.:13001 311KB 10.9Kb 86.7Kb 62.1Kb
ecs-8416:13001 <=> c-24-60-2-83.hsd1.ma.comcast.net:13001 34.3MB 0b 0b 33.985.1Kb 61.6Kb
ecs-8416:13001 <=> 34.146.254.37:13001 13001 3.18MB 0b 0b 22.9Kb
ecs-8416:ssh <=> 113.137.58.76:4795 586KB 10.6Kb 10.4Kb 11.7Kb
ecs-8416:13001 <=> 161.35.18.85:1300213-20.net:13003 567KB 0b 45.2Kb 11.4Kb
ecs-8416:13001 <=> 129.213.159.159:13001 0.net:13003 1.09MB 0b 11.4Kb 8.19Kb
ecs-8416:13001 <=> static.152.225.109.65.clients.your-server.d:13001 182KB 6.00Kb 2.40Kb 3.46Kb
ecs-8416:13001 <=> static.101.230.21.65.clients.your-server.de:13001 16.4KB 0b 13.1Kb 3.28Kb
ecs-8416:58690 <=> ecs-190-92-218-36.compute.hwclouds-dns.com:30073 71.8KB 0b 8.23Kb 3.19Kb
ecs-8416:13001 <=> 195.14.6.207:13001 139KB 2.09Kb 3.67Kb 3.02Kb

TX: cum: 369MB peak: 25.3Mb rates: 930Kb 7.92Mb 6.42Mb
RX: 386MB 22.2M96 8.8 1.04Mb 8.46Mb 6.65Mb
TOTAL: 766MB 54.1Mb 1.94Mb 16.4Mb 13.1Mb

[haoei]

@moshe-blox
EL and CL nodes are not running with SSV. They run on separate machines.