Download speed for maps -- Bluehost vs Digital Ocean [OSM]

[georgejhunt]

Maybe Bluehost needs Cloudflare setup. Or maybe there is a bottleneck between Utah and California. I'm pretty clear remembering our discussion. Both Tim and Adam found timmoody.com was fast.

Observations:
wget http://timmoody.com/iiab-files/maps/osm_spanish_speaking_regions_z11-z14_2019.mbtiles --
osm_spanish_speakin 100%[===================>] 13.79G 1.50MB/s in 2h 51m
https://iiab.sfo2.digitaloceanspaces.com/osm-tiles/osm_spanish_speaking_regions_z11-z14_2019.mbtile
osm_spanish_speakin 100%[===================>] 13.79G 8.12MB/s in 28m 36s

[holta]

FYI over the past 4.0 hours, I tested Bluehost.com downloads from http://timmoody.com/iiab-files/maps/ (in or near Houston, apparently) to the US Northeast:

No matter how many http (wget) downloads were running simultaneously, the results were the same...

1. More than half of the time, each http connection downloaded (right around) 200 to 300 kByte/s
2. Almost half of the time, each http connection downloaded (about) 20X faster, e.g. around 2 to 8 MByte/s

In short there's some kind of a Bluehost-imposed auto-throttle that kicked in erratically and frequently, regardless how many http (wget) downloads were running simultaneously. When this auto-throttle kicked in, Bluehost capped each individual http download to about 250kByte/s = 2Mbit/s or less.

In short, this is likely equivalent or perhaps now worse than (free, global library) Archive.org
And certainly no longer helpful to those wanting OSM "continents" installed efficiently + quickly to their IIAB's :/

Recap:

• The good news is that almost exactly half of the ~220 GByte total is now downloaded, with 12 of the 18 downloads now complete.
• The bad news is that the 6 largest of the 18 total wget downloads are continuing, and will likely take many hours (13 more hours if the current wget projection is to be believed).

So I'm working on other viable option(s) and will call you @georgejhunt in coming days to fine tune.

Refs:

• Extremely slow downloads of IIAB Maps from archive.org (often 1-2 Mbit/s) - would torrents solve this - or is there a better way? [OSM] #2553
• PR Branch Maps7.3 -- includes 2020 OSM data and 2020 satellite images #3165
• PR Maps7.3--Uses 2020 OSM data and 2020 satellite imagery maps#45
• PR Programs used in the generation of 2020 version maptools#4

[holta]

13 more hours if the current wget projection is to be believed

Not Good: Projected remaining time to download 81 GB http://timmoody.com/iiab-files/maps/osm_planet_z11-z14_2020.mbtiles (i.e. from Bluehost) increased to 30+ hours — and that's on top of the 10h it's already been downloading!

Average bandwidth from Bluehost being ~4.5mbit/s and extremely erratic, which is better than nothing sure, but still incredibly disappointing overall. Bluehost used to be rather fast and reliable here, at least for many of us in 2020 (certainly I and few others tried from many locations back then) but it seems no longer.

As I just explained on a quick call with George — various new S3-compatible options appear to be much better options at this point. No option is perfect but it's looking good. Hopefully we'll all have better news to share here within a week or so!

(Especially given that downloading from Archive.org remains incredibly slow for most who try outside of the SF Bay area — unless one Torrents from their seeds etc...)

[holta]

13 more hours if the current wget projection is to be believed

Projected remaining time to download 81 GB http://timmoody.com/iiab-files/maps/osm_planet_z11-z14_2020.mbtiles (i.e. from Bluehost) increased to 30+ hours

Now 30h more hours are expected according to wget, to download the above single file :/

• As a result of Bluehost.com's download speed having dropped to ~2.1 Mbit/s = 260-to-270 kByte/s quite consistently over recent hours...

• Overall average download speed (over the entire 22h so far) has dropped below 4 Mbit/s. Reason: Friday morning's high-speed spurts [most occurred between 20 and 22 hours ago, i.e. occasional blasts of around ~40 Mbit/s = 2-to-8 MByte/s] have not recurred much at all since then :/

• FWIW stopping wget and restarting it a minute later (using "wget -c ") does not improve the situation unfortunately.

[holta]

Conclusion:

The 81 GB file download finally completely, over ~44 hours total.
(The overall average speed from Bluehost.com was about 548 kByte/s = 4.4 Mbit/s.)

Which means Good Riddance to Bluehost — thankfully we have new S3-compatible options that are almost 100 faster.

[tim-moody]

thankfully we have new S3-compatible options that are almost 100 faster

but for some reason you don't wish to tell us where it is

[holta]

I've asked George to evaluate it before making it public.

I can share with others privately who are also willing to test.

[georgejhunt]

@adam. Can you use rclone md5sum (see https://rclone.org/commands/rclone_md5sum/) to verify the correctness of your uploads?

…

On Mon, Apr 18, 2022 at 2:01 PM A Holt ***@***.***> wrote: I've asked George to evaluate it before making it public. I can share with others privately who are also willing to test. — Reply to this email directly, view it on GitHub <#3111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOTQHHUQNUJUDG5UPB37ULVFXESFANCNFSM5M65X3JQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[holta]

Can you use rclone md5sum (see
https://rclone.org/commands/rclone_md5sum/) to verify the correctness of
your uploads?

It looks like this rclone already did that automatically — as interestingly this command responded instantly:

# rclone md5sum anon:iiab-maps

b9774e2cf2cbcc23849819866e43bb68  map-catalog.json
76a2dd18c1a8d5eb22cc470c0e78c6a5  osm_central_america_z11-z14_2020.mbtiles
6b49b304f251c752e641a450e0c368ec  osm_middle_east_z11-z14_2020.mbtiles
129b63c3541127f04ff98d5e95948b17  osm_europe_z11-z14_2020.mbtiles
d93465cd354117f5c7bdc22824370273  osm-planet_z0-z10_2020.mbtiles
58062a1fe62ae41a218d38833fc94abf  osm_africa_z11-z14_2020.mbtiles
e3f1c6b1240c36f78a1010ccec4dd5b9  osm_planet_z11-z14_2020.mbtiles
8372b415a077b9082d2c104ece479256  osm_planet_z11-z14_2020.mbtiles.md5
957dad8109b5249c16c7466340b5f242  osm_san_jose_z11-z14_2020.mbtiles
39d5f30ea015609521ac9c8a44620715  osm_north_asia_z11-z14_2020.mbtiles
c2263da0d7b568ca4f475617f9e8df34  osm_oceania_z11-z14_2020.mbtiles
1ba11411a0d2187666d7a5c1a696a465  osm_north_america_z11-z14_2020.mbtiles
92437c8e6ad04c428141f7be78b082c0  planet_z0-z6_2020.mbtiles
15a55dd36e3c8e7b55486404d8c7cc4d  satellite_z0-z6_2020.mbtiles
7780b4bebf4463a6cf07e6984b099f5d  osm_south_america_z11-z14_2020.mbtiles
cb9f1242b121677a73bc62c050749363  osm_south_asia_z11-z14_2020.mbtiles
fa6462c6453182dc5b1c846157c0ff03  osm_spanish_speaking_regions_z11-z14_2020.mbtiles
afa23afc08ea9f1207e39fd654c612de  satellite_z0-z9_2020.mbtiles

I also verified that all 18 above MD5 hashes correspond to what I downloaded from http://timmoody.com/iiab-files/maps/ (i.e. by manually running those hashes, which took many minutes).

@georgejhunt can you additionally verify the above MD5 hashes against the original files you produced? (Or is that not necessary?)

[holta]

Here are the exact sizes in bytes just for the record:

# rclone ls anon:iiab-maps

      12409 map-catalog.json
 2108256256 osm-planet_z0-z10_2020.mbtiles
15891234816 osm_africa_z11-z14_2020.mbtiles
 3551305728 osm_central_america_z11-z14_2020.mbtiles
30514585600 osm_europe_z11-z14_2020.mbtiles
 9039286272 osm_middle_east_z11-z14_2020.mbtiles
24112046080 osm_north_america_z11-z14_2020.mbtiles
19161931776 osm_north_asia_z11-z14_2020.mbtiles
 6260080640 osm_oceania_z11-z14_2020.mbtiles
86868115456 osm_planet_z11-z14_2020.mbtiles
         66 osm_planet_z11-z14_2020.mbtiles.md5
   30449664 osm_san_jose_z11-z14_2020.mbtiles
11512279040 osm_south_america_z11-z14_2020.mbtiles
12899446784 osm_south_asia_z11-z14_2020.mbtiles
13473050624 osm_spanish_speaking_regions_z11-z14_2020.mbtiles
   50049024 planet_z0-z6_2020.mbtiles
   26144768 satellite_z0-z6_2020.mbtiles
 1273274368 satellite_z0-z9_2020.mbtiles

[holta]

Total size for these 18 files is...

# rclone size anon:iiab-maps

Total objects: 18 (18)
Total size: 220.511 GiB (236771549371 Byte)

[georgejhunt]

md5sum values for the 18 files at contaba match the original md6sums before upload to timmoody.com

[holta]

Related:

• PR change default map source to contabo iiab-admin-console#483

[holta]

Good enough for now.