A wget utility in go that accepts a list of files to download and only downloads the missing ones
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
.travis.yml
LICENSE
README.md
gwget.go

README.md

Wget based on file list

Documentation Go Report Card Build Status

The tool can (for example) mirror a media folder to a staging or developer server from the current live, over standard http requests, when scp is way too slow, creating tarball is not an option because of space requirements, VPN makes it hard to run tar | tar or rsync.

Usage

source$ cd ~/web/media
source$ find . -type f | grep -v '.thumb' | grep -v product/cache | sed 's#^./##' > ~/media.txt

scp media.txt onto the target server with a gwget binary

target$ cd ~/web/media
target$ ~/gwget ~/media.txt https://www.somesite.com/media/