Identify what subsystem is causing the biggest difference between the 99th and 50th percentile for upload performance between the public and select networks

[iglesiasbrandon]

Background:

Slack channel: #strikeforce-upload-performance-variation

This issue is about following up on two of our OKRs:
Get Veeam Backup performance on par with Wasabi
Pass Veeam Ready

the edge team has been monitoring our speed for some time, and at this point believes we're "fast enough" usually, but the problem remains our 99th percentile performance. the edge team says that occasionally, upload (and download? but we're mostly focused on upload) performance slows enough that that variation tanks our overall score. the edge team feels that this variation is within the uplink envelope, and so has passed the torch on to us to try and figure out what's next.

so, this strikeforce's goal:

1. identify which subsystems are causing the most difference between our 50th and 99th percentile performance
2. fix those subsystems

we suspect that the problem is largely the storage node network, and there are different issues facing the public network and the select network, so we want to investigate both

a bunch of things we can do, but perhaps the best starting point for #1 is to:

• try a bunch of veeam ready size uploads
• take traces of each upload
• for each subsystem, keep track of the overall time distribution out of the trace
• find the subsystems that have the biggest difference between 50th and 99th percentile?
• repeat recursively

[profclems]

I will unassign from this issue since the effort has largely been on the team-delivery side

[storjrobot]

This issue has been mentioned on Storj Community Forum (official). There might be relevant details there:

https://forum.storj.io/t/updates-on-test-data/26034/40