Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

What affects download speeds? #276

Open
Popolechien opened this issue Oct 19, 2021 · 6 comments
Open

What affects download speeds? #276

Popolechien opened this issue Oct 19, 2021 · 6 comments
Labels
question Further information is requested

Comments

@Popolechien
Copy link
Contributor

This is sort of related to #266 I guess: there's a user that is located on the US East Coast that tried downloading an image from our servers. It is however pretty slow, eventhough they claim download speeds over 100+ Mbps
Download Speed
I've tried downloading one of their images and actually did get much better DL speeds. Any idea why/how this could be and how we could remediate this?
Capture d’écran 2021-10-19 à 10 36 10
@Popolechien Popolechien added the question Further information is requested label Oct 19, 2021
@rgaudin
Copy link
Member

rgaudin commented Oct 19, 2021

We've discussed that several times, but it might not have been captured in a ticket:

  • We store our images on Wasabi S3 servers.
  • Wasabi serves those files over HTTP/s
  • Wasabi is a storage solution, not a CDN.
  • We use Wasabi because it's much much cheaper than Amazon S3.
  • One of the key difference with Amazon is that Wasabi doesn't charge us for bandwidth when we download our images (egress transfer).
  • There is though a free egress policy. AFAIK, we've never exceeded our storage in egress (because we store a lot).
  • Wasabi doesn't communicate on the bandwidth available to users. We can thus assume it's shared across users of a region.
  • All our images are stored on us-west-1 datacenter.

Of course, the user's bandwidth and route congestion between user and the server play a big role but we've seen that frequently, users face slow download speeds, in the orders shown above: ~650KB/s.

The fact that it's roughly the same speed could indicate some sort of throttling on Wasabi side. We don't know if they reduce bandwidth after intensive use – IP? – (they don't mention it on their doc) nor if we've gone over the limits of that policy.

At this point, I'd suggest you contact them to ask why we're so frequently getting limited speeds.

@Popolechien
Copy link
Contributor Author

Will do, thanks.

@Popolechien
Copy link
Contributor Author

Ok Wasabi asked us to run the following tests:

https://wasabi-support.zendesk.com/hc/en-us/articles/115003033352-I-am-not-seeing-the-network-performance-I-m-expecting-what-should-I-do-?

@rgaudin
Copy link
Member

rgaudin commented Oct 20, 2021 •
edited

Thanks, how I understand it is, we should follow the following procedure when encountering issues:

  • calculate the expected duration of a transfer based on one's connection capabilities. Not much useful though.
  • use the Speedtest. Requires a password we can find in the console. If speedtest is OK, then there's a bucket specific issue -> contact Wasabi.
  • If the speedtest showed limited speed (the most probable), run the following tests and send output to Wasabi.

I suggest we do this the next time one of us is affected. Seems a bit much to ask from our clients… as this won't lead to an immediate resolution.

Sample runs from my 35/2Mbps connection (no issue ATM). Confirms #266 would benefit people from EU (or routing from EU).

Speedtest and route to eu-central-1 (⚠️ not what we're using!)

Screen Shot 2021-10-20 at 08 36 46

sudo mtr -T -P 443 -rw s3.eu-central-1.wasabisys.com
Start: 2021-10-20T09:00:55+0000
HOST: faku.local                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- yeleman-camel                           0.0%    10    1.6   1.9   1.6   2.1   0.1
  2.|-- 197.155.137.101                         0.0%    10    1.9   2.2   1.9   2.9   0.3
  3.|-- 213.154.92.106                          0.0%    10   26.8  86.8  26.8  99.5  21.4
        193.251.248.175
        81.52.188.117
  4.|-- 196.207.255.193                        40.0%    10   18.8  54.5  17.3  90.6  39.1
        hundredgige0-11-0-11.pastr4.paris.opentransit.net
  5.|-- 213.144.187.194                         0.0%    10   81.1  81.5  54.5  91.4  11.2
        be2868.ccr41.lon13.atlas.cogentco.com
        be2871.ccr42.lon13.atlas.cogentco.com
  6.|-- be3183.ccr41.par01.atlas.cogentco.com   0.0%    10   84.2  69.7  54.3  93.6  15.6
        e0-4.switch1.lis1.he.net
        be3184.ccr42.par01.atlas.cogentco.com
        be12488.ccr42.ams03.atlas.cogentco.com
  7.|-- be12266.ccr42.ams03.atlas.cogentco.com  0.0%    10   86.5  79.6  60.2  93.6  13.3
        be2278.rcr21.b038092-0.ams03.atlas.cogentco.com
        e0-6.switch1.mad2.he.net
        be2283.rcr21.b038092-0.ams03.atlas.cogentco.com
  8.|-- 149.11.201.210                          0.0%    10   88.8  90.2  88.3  95.6   2.4
        be2278.rcr21.b038092-0.ams03.atlas.cogentco.com
        100ge10-1.core1.par2.he.net
        be2283.rcr21.b038092-0.ams03.atlas.cogentco.com
  9.|-- 149.11.201.210                          0.0%    10   87.3  96.1  86.7 141.5  16.6
        130.117.252.16
        port-channel13.core2.ams1.he.net
 10.|-- 130.117.252.16                          0.0%     6   96.2  97.8  93.0 100.8   2.9
        wasabi-technologies-inc.port-channel3.switch1.ams4.he.net

Speedtest and route to us-west-1 (where our servers are)

Screen Shot 2021-10-20 at 08 39 26

sudo mtr -T -P 443 -rw s3.us-west-1.wasabisys.com
Start: 2021-10-20T08:32:23+0000
HOST: faku.local                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- yeleman-camel                           0.0%    10    1.8   1.8   1.6   2.4   0.2
  2.|-- 197.155.137.101                         0.0%    10    2.0   3.4   1.8  15.8   4.4
  3.|-- 213.154.92.106                          0.0%    10   24.5  79.4  24.5  98.2  28.6
        81.52.188.117
        193.251.248.175
  4.|-- 196.207.255.193                        70.0%    10   20.0  43.4  20.0  89.9  40.3
        hundredgige0-11-0-11.pastr4.paris.opentransit.net
  5.|-- 213.144.187.194                         0.0%    10   52.1  71.2  50.8  86.8  16.8
        be2868.ccr41.lon13.atlas.cogentco.com
        be2871.ccr42.lon13.atlas.cogentco.com
  6.|-- e0-4.switch1.lis1.he.net                0.0%    10   57.7 184.2  57.0 1072. 314.4
        be2099.ccr31.bos01.atlas.cogentco.com
        be2101.ccr32.bos01.atlas.cogentco.com
        be3184.ccr42.par01.atlas.cogentco.com
  7.|-- be12489.ccr42.lon13.atlas.cogentco.com  0.0%    10   87.1 108.9  60.3 158.8  40.3
        be12497.ccr41.lon13.atlas.cogentco.com
        be3600.ccr22.alb02.atlas.cogentco.com
        e0-6.switch1.mad2.he.net
        be3599.ccr21.alb02.atlas.cogentco.com
  8.|-- be2879.ccr22.cle04.atlas.cogentco.com   0.0%    10  165.3 124.0  66.7 168.4  44.7
        be2878.ccr21.cle04.atlas.cogentco.com
        100ge0-34.core1.bio1.he.net
        be2101.ccr32.bos01.atlas.cogentco.com
        be2099.ccr31.bos01.atlas.cogentco.com
  9.|-- 100ge0-30.core1.orf2.he.net             0.0%    10  159.2 156.3 149.0 163.7   4.5
        be3600.ccr22.alb02.atlas.cogentco.com
        be2718.ccr42.ord01.atlas.cogentco.com
        be3599.ccr21.alb02.atlas.cogentco.com
 10.|-- be2831.ccr21.mci01.atlas.cogentco.com   0.0%    10  175.5 167.3 154.7 176.6  10.1
        be2832.ccr22.mci01.atlas.cogentco.com
        be2878.ccr21.cle04.atlas.cogentco.com
        100ge15-1.core2.ash1.he.net
 11.|-- be2718.ccr42.ord01.atlas.cogentco.com  10.0%    10  164.4 171.7 158.1 189.8  13.3
        100ge1-2.core1.ash1.he.net
        be3036.ccr22.den01.atlas.cogentco.com
        be3035.ccr21.den01.atlas.cogentco.com
 12.|-- be3037.ccr21.slc01.atlas.cogentco.com   0.0%    10  205.5 188.1 174.1 205.6  14.2
        be2832.ccr22.mci01.atlas.cogentco.com
        be3038.ccr32.slc01.atlas.cogentco.com
        be2831.ccr21.mci01.atlas.cogentco.com
 13.|-- be3035.ccr21.den01.atlas.cogentco.com   0.0%    10  188.4 194.7 187.6 218.1  12.4
        be3036.ccr22.den01.atlas.cogentco.com
        be2029.ccr22.sea02.atlas.cogentco.com
        100ge14-2.core1.msp1.he.net
        be2042.ccr21.sea02.atlas.cogentco.com
 14.|-- be2670.ccr21.pdx01.atlas.cogentco.com   0.0%    10  223.6 224.0 206.2 240.0   9.4
        be3038.ccr32.slc01.atlas.cogentco.com
        100ge11-2.core1.sea1.he.net
        be3037.ccr21.slc01.atlas.cogentco.com
        be2671.ccr21.pdx01.atlas.cogentco.com
 15.|-- 100ge0-35.core1.pdx3.he.net             0.0%    10  234.0 231.5 215.6 249.8  10.1
        100ge15-1.core1.pdx1.he.net
        be2042.ccr21.sea02.atlas.cogentco.com
        be2216.ccr51.pdx02.atlas.cogentco.com
        be2029.ccr22.sea02.atlas.cogentco.com
 16.|-- 38.104.108.138                          0.0%    10  222.7 222.5 213.1 235.5   6.5
        be2671.ccr21.pdx01.atlas.cogentco.com
        wasabi-technologies-inc.e0-53.switch1.pdx3.he.net
        100ge0-36.core1.pdx2.he.net
 17.|-- 76.191.80.53                            0.0%    10  225.6 222.1 213.6 231.3   5.1
        100ge0-35.core1.pdx3.he.net
        be2216.ccr51.pdx02.atlas.cogentco.com

10MiB file retrieval test

for count in {1..5}; do dig +noall +answer s3.us-west-1.wasabisys.com | tail -1; echo -n `date`" -- ";curl -w 'bytesPerSecondDownload %{speed_download}\n' -so /dev/null http://s3.us-west-1.wasabisys.com/org-kiwix-hotspot-cardshop-download/10Mio.dat; done
us-west-1.wasabisys.com. 120	IN	A	76.191.80.53
Wed Oct 20 08:50:58 GMT 2021 -- bytesPerSecondDownload 397639.000
us-west-1.wasabisys.com. 120	IN	A	76.191.80.53
Wed Oct 20 08:51:25 GMT 2021 -- bytesPerSecondDownload 348040.000
us-west-1.wasabisys.com. 120	IN	A	76.191.80.53
Wed Oct 20 08:51:55 GMT 2021 -- bytesPerSecondDownload 490470.000
us-west-1.wasabisys.com. 120	IN	A	76.191.80.53
Wed Oct 20 08:52:17 GMT 2021 -- bytesPerSecondDownload 704688.000
us-west-1.wasabisys.com. 120	IN	A	76.191.80.53
Wed Oct 20 08:52:31 GMT 2021 -- bytesPerSecondDownload 515372.000

@kelson42
Copy link
Contributor

kelson42 commented Feb 1, 2023

My feedback, download a log of Zimfar, so save object-cache provider, bucket located in Europe (my guess), my download moves beetween 6MB/s and 13MB/s, so basically between 50 and 100Mb/s.

@rgaudin
Copy link
Member

rgaudin commented Feb 1, 2023

zimfarm logs bucket, as almost all of our buckets are in us-west-1 (portland)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rgaudin @kelson42 @Popolechien