Automated LFTP Sync from SeedBox to Home

This tutorial will explain how to use an automated LFTP script that runs every few minutes (or of your choosing) matching a remote directory with your home. This script only works one way, so if you remove the file on your server, it will not be removed from your home directory. It will also work with Windows, Mac, and Linux.

I prefer LFTP because, not only is it a fully automated daemon, it also maximizes my home pipeline. LFTP supports parallel downloads of the same file while also downloading others as well. Only one instance of this script will run, if it is currently transferring, it creates a lock and will not run again until the current operation has completed.

Prerequisites - Install LFTP @ Home

MacOS

Install Homebrew

After the above is installed execute the following within terminal to install LFTP

brew update
brew install lftp

Windows

Install Cygwin

Make sure to select the following addons during the installation

LFTP
bash
cygrunsrv
cron

Create a file called synctorrents.sh, replace all < > with your values. The only code you need to modify is within the top 6 lines.

Important note: bash takes EOL (End of line) quite seriously. You will need to use UNIX/LF style end of lines for bash scripts. See this FAQ for info Text editing - Over FTP or SFTP

In the script these are the variables you will need to customise to meet your requirements.

#!/bin/bash
login="username"
pass="password"
host="server.feralhosting.com"
remote_dir='~/folder/you/want/to/copy'
local_dir="$HOME/lftp/"

Here is the basic script:

Do this command in you Cygwin terminal to download the script.

wget -qO ~/autolftp-ftp.sh http://git.io/vf0wo

Here is the script to manually copy and paste:

#!/bin/bash
login="username"
pass="password"
host="server.feralhosting.com"
remote_dir='~/folder/you/want/to/copy'
local_dir="$HOME/lftp/"

base_name="$(basename "$0")"
lock_file="/tmp/$base_name.lock"
trap "rm -f $lock_file; exit 0" SIGINT SIGTERM
if [ -e "$lock_file" ]
then
    echo "$base_name is running already."
    exit
else
    touch "$lock_file"
    lftp -u $login,$pass $host << EOF
    set ftp:ssl-allow no
    set mirror:use-pget-n 5
    mirror -c -P5 --log="/var/log/$base_name.log" "$remote_dir" "$local_dir"
    quit
EOF
    rm -f "$lock_file"
    trap - SIGINT SIGTERM
    exit
fi

Here is an edited version for use with sftp:

Do this command in you Cygwin terminal to download the script.

wget -qO ~/autolftp-sftp.sh http://git.io/vfGVm

Here is the script to manually copy and paste:

#!/bin/bash
login="username"
pass="password"
host="server.feralhosting.com"
remote_dir='~/folder/you/want/to/copy'
local_dir="$HOME/lftp/"

base_name="$(basename "$0")"
lock_file="/tmp/$base_name.lock"
trap "rm -f $lock_file; exit 0" SIGINT SIGTERM
if [ -e "$lock_file" ]
then
    echo "$base_name is running already."
    exit
else
    touch "$lock_file"
    lftp -p 22 -u "$login","$pass" sftp://"$host" << EOF
    set sftp:auto-confirm yes
    set mirror:use-pget-n 5
    mirror -c -P5 --log="/var/log/$base_name.log" "$remote_dir" "$local_dir"
    quit
EOF
    rm -f "$lock_file"
    trap - SIGINT SIGTERM
    exit
fi

Make the script executable:

chmod 700 synctorrents.sh

The important parameters for lftp are:

set mirror:use-pget-n 5

This makes lftp try to split up files in 5 pieces for parallel downloading. Likewise,

-P5

Means it will download at most 5 files in parallel (for a total 25 connections). Those 2 combined work wonders. In my case, I always end up downloading the files at the limit of my connection, but feel free to play with them and find what works best for you.

-c

Just tells it to try and resume an interrupted download if it' s the case.

Creating a crontab.

WINDOWS USERS: Instead of crontab you may find it easier to use the Task Scheduler to run the command at the interval of your choosing.

Here is a link to a simple guide on using task scheduler

Using Cygwin:

C:\cygwin\bin\sh.exe -l /home/user/synctorrents.sh >> /your/home/directory/sync_cron.log 2>&1

The following instructions create a file called crontab and point it to the script you previously made called synctorrents.sh. This is what automates the process every few minutes.

touch crontab

nano crontab

*/5 * * * * /home/user/synctorrents.sh >> /your/home/directory/sync_cron.log 2>&1

syncs every 5 min, change this to your choosing

Save and exit:

crontab /crontab

points crontab to the file you just made and edited

crontab -l

lists crontab, confirm to make sure it is linked correctly

I must give credit to LordHades who created this amazing script.