Need to retune % of data going to unvetted nodes

[thepaul]

Description

Currently, in node selection, we select 5% of the nodes from the pool of unvetted nodes. This was meant to limit how much data went to unvetted nodes. However, (somewhat) recent changes have made it so that nodes can be vetted significantly faster than before, leaving the pool of vetted nodes much smaller. This, in turn, leads to unvetted nodes getting a larger share of data than they would have gotten when we first tuned the 5% value.

There are some pretty dramatic bandwidth screenshots illustrating this effect at https://forum.storj.io/t/unvetted-vetted-node-traffic/23086. These might not be reflective of the average node experience, though; a small number of unvetted nodes sharing a /24 network with many vetted nodes would see this bandwidth effect multiplied, and that may be what's happening there.

Possible Fix

We should lower the 5% parameter. How much we should lower it is not totally clear to me after researching the historical fraction of unvetted nodes, but we should probably lower it to be at most 2%.

Distinct last_nets with unvetted nodes currently make up about 2.8% of all distinct last_nets, so we probably want the new value lower than that, at least.

The parameter is overlay.node.new-node-fraction in satellite config.

[storjrobot]

This issue has been mentioned on Storj Community Forum (official). There might be relevant details there:

https://forum.storj.io/t/unvetted-vetted-node-traffic/23086/5

[thepaul]

Thinking more about the multiplicative effect where unvetted nodes share a last_net with vetted nodes. The last_net for the unvetted nodes effectively gets counted as a separate network from the last_net of the vetted nodes, which is why it can make such a difference.

We might need to do a bigger refactor of node selection code in order to mitigate that effect, so it doesn't become an incentive to keep lots of nodes unvetted.

[ReneSmeekes]

Suggestion: Create a satellite setting to set the % of normal node ingress an unvetted node should get. Calculate the number of pieces to go to unvetted nodes by: (nUnvettedNodes / nTotalNodes) * unvettedTraffic% * nPiecesToSelect
This won't usually be a whole number. In fact, in the future this will likely be below 1. To resolve that, round that number down and generate a random number between 0 and 1. If that number is below the remainder, add 1 to the rounded down number of unvetted nodes to select.

[storjrobot]

This issue has been mentioned on Storj Community Forum (official). There might be relevant details there:

https://forum.storj.io/t/outrageous-upload-from-some-nodes/23937/4

[thepaul]

We're currently evaluating whether we still even want to guarantee a percentage of data to new nodes, and what changes to the filter DSL would need to be made to avoid the traffic-multiplier effect.

Putting this back in Todo until we resolve those questions.

[shaupt131]

@shaupt131 to add this issue to arch review agenda to have a broader discussion and come to some decisions.

[shaupt131]

Decided during arch review to move forward with 1%.

[storjrobot]

This issue has been mentioned on Storj Community Forum (official). There might be relevant details there:

https://forum.storj.io/t/it-looks-like-a-ssd-is-much-faster-full-and-vetted-than-an-hdd/24818/11

[storj-gerrit]

Change satellite/overlay: change % of data going to new nodes from 5 to 1 mentions this issue.

[storj-gerrit]

Change satellite/overlay: change % of data going to new nodes from 5 to 1 mentions this issue.

[iglesiasbrandon]

moving to deployed.

[storjrobot]

This issue has been mentioned on Storj Community Forum (official). There might be relevant details there:

https://forum.storj.io/t/no-audit-traffic-and-no-repair-traffic/25425/43