Host files in multiple S3 regions

[leezu]

Downloading files from S3 hosted in the US can be painfully slow from Hong Kong (and likely mainland). @szha can we easily host them in a second region?
In my experience the bandwidth to Europe is good, in case that the Hong Kong region is not yet ready.

[szha]

I always keep the global s3 in sync with the China s3 mirror

[szha]

the global s3 is backed with s3-accelerate, which already utilizes all cloudfront edge locations.

[szha]

Try export MXNET_GLUON_REPO=https://apache-mxnet.s3.cn-north-1.amazonaws.com.cn/

[leezu]

Thanks, it's great you already thought about this issue and that there is already a solution!
What do you think about putting that behind Cloudfront. If I understand correctly we may use a single link then and the correct region will be selected automatically based on the source of the request.

[szha]

Due to regulation in China, it requires a business entity and registration with the government to set up CDN in China.

[szha]

Also, China AWS accounts are separate from global AWS accounts, so we would at least need to cloudfront end points. Then we would need DNS magic to point to the right endpoint, so I haven't gone through that route yet.

[leezu]

Btw, the China S3 mirror also slow in Asia-Pacific region outside of mainland (eg. HK).

[szha]

@leezu how's the speed with the global repo? Tokyo/Singapore edges might be fast.

[leezu]

That works. I had some speed issues but in the end a VPN is at fault. Sorry about the noise.

[eric-haibin-lin]

Is this issue resolved?

[leezu]

I think the current S3 Accelerate approach is sufficient. AWS recently added CloudFront endpoints in Hong Kong and mainland China.
S3 Accelerate doesn't make use of the endpoints in mainland China though, but maybe they will in the future.

Do we have any reports of trouble downloading in China? I'll close this for now, but let's reopen once we have more evidence of the issue.

[szha]

@leezu they won't. China is a separate region entirely and there's no data exchange with the global region except on the public network.

[leezu]

Cloudflare does, so I think Cloudfront will need to do eventually as well. Cloudflare advertises this feature heavily https://www.cloudflare.com/network/china/