perf(gateway): stable round robin

[kckeiks]

Why

Fixes #433.

Before, all tasks were attempting to rotate the provider list and update the cache with their updated version.
This resulted in this pattern for example:

Requesting from ip1
Requesting from ip1
Requesting from ip2
Requesting from ip1
Requesting from ip2
Requesting from ip3
Requesting from ip1

Requests to providers are not evenly distributed when receiving bursts. This PR adds time sharing so that each provider gets a time slot and thus a fair chance to provide. This results in a more stable implementation of round robin

Requesting from ip1
Requesting from ip1
Requesting from ip1
Requesting from ip2
Requesting from ip2
Requesting from ip3
Requesting from ip3

Demo

No regression. Performance is still the same. Running with tower-http=0.3.5 because of #436.

Before

Summary:
  Success rate:	1.0000
  Total:	22.7413 secs
  Slowest:	3.6432 secs
  Fastest:	0.0404 secs
  Average:	0.1134 secs
  Requests/sec:	439.7277

  Total data:	256.66 MiB
  Size/request:	26.28 KiB
  Size/sec:	11.29 MiB

Response time histogram:
  0.040 [1]    |
  0.401 [9949] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.761 [0]    |
  1.121 [0]    |
  1.481 [0]    |
  1.842 [0]    |
  2.202 [0]    |
  2.562 [0]    |
  2.923 [0]    |
  3.283 [0]    |
  3.643 [50]   |

Latency distribution:
  10% in 0.0719 secs
  25% in 0.0805 secs
  50% in 0.0925 secs
  75% in 0.1080 secs
  90% in 0.1281 secs
  95% in 0.1395 secs
  99% in 0.1600 secs

Details (average, fastest, slowest):
  DNS+dialup:	3.4494 secs, 2.9878 secs, 3.5651 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0002 secs

Status code distribution:
  [200] 10000 responses

After

Summary:
  Success rate:	1.0000
  Total:	22.7160 secs
  Slowest:	3.4743 secs
  Fastest:	0.0292 secs
  Average:	0.1132 secs
  Requests/sec:	440.2192

  Total data:	256.66 MiB
  Size/request:	26.28 KiB
  Size/sec:	11.30 MiB

Response time histogram:
  0.029 [1]    |
  0.374 [9949] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.718 [0]    |
  1.063 [0]    |
  1.407 [0]    |
  1.752 [0]    |
  2.096 [0]    |
  2.441 [0]    |
  2.785 [0]    |
  3.130 [0]    |
  3.474 [50]   |

Latency distribution:
  10% in 0.0761 secs
  25% in 0.0840 secs
  50% in 0.0922 secs
  75% in 0.1045 secs
  90% in 0.1265 secs
  95% in 0.1360 secs
  99% in 0.1485 secs

Details (average, fastest, slowest):
  DNS+dialup:	3.2702 secs, 0.9341 secs, 3.3973 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs

Status code distribution:
  [200] 10000 responses

Notes

Checklist

• I have made corresponding changes to the tests
• I have made corresponding changes to the documentation
• I have run the app using my feature and ensured that no functionality is broken

[b0xtch]

I wonder why req/s is very low if all its doing is routing requests.