fastscp uses the Internet routes from Cloudflare to quickly transfer data between two hosts. It requires at least one host to have a public IP address.
- set up an HTTP server on source or dest
- use Cloudflare to proxy the HTTP traffic so that traffic goes from src -> cf_edge -> cf_internal -> cf_edge -> dest, this often maximizes bandwidth due to the Cloudflare inter-connections
- download the files from the proxied HTTP server
fastscp uses jq, parallel and python3, on Ubuntu you can install using
sudo apt-get install -yqq jq parallel wget
sudo curl -s https://raw.githubusercontent.com/1a1a11a/fastscp/main/fastscp.sh -o /usr/local/bin/fastscp && sudo chmod +x /usr/local/bin/fastscp
fastscp data ${USER}@${HOST}:/PATH/
fastscp is not designed for production. It is a tool that I use to speed up data transfer from CMU and Cloudlab. Most of my data are not private, so privacy is not a design consideration. Moreover, when open-sourcing the tool, to make it easy to use, I created a shared account on Cloudflare, which means anyone is able to see your data during the transfer.
Do the following if you need better privacy and security
- use your own Cloudflare account, put your zone information in
api.sh - use the
authHTTPServer.pyinstead of unencrypted HTTP web server - use HTTPS instead of HTTP at source for encryption
- use a white list of IPs in the Python web HTTP server
- make port selection automatically
- support compression
- support parallel downloads
- support machines behind NAT
- add stat reporting function