How to Setup rclone
Manage transfer of media files to an Internet Remote Storage Service. Eg Google Drive, DropBox, Etc.
- Configure a Remote Storage Service
- How to Test rclone sync
- Sample rclone- scripts
- Change sync to Something Else
- How to Automate using crontab
- Installation (optional if not already installed)
NOTICE : Rclone is now the current pi-timolo default installed by pi-timolo-install.sh and is at /usr/bin/rclone. If a previous /usr/local/bin/gdrive File Exists, it Will Remain. If rclone is Not already installed upgrade pi-timolo per pi-timolo-install.sh script. You can also install rclone separately per these Rclone Install Instructions. If you have another project where you want rclone support see my rclone4pi GitHub repository
Rclone is a command line program to sync files and directories to and from many remote storage services. This utility can be used to update pi-timolo media folders/files to one or more remote services. For more information about rclone See https://rclone.org and/or https://github.com/ncw/rclone
How to Configure a Remote Storage Service
To configure one rclone remote storage service on a raspberry pi computer follow the steps below. If a service is already installed you will see an interactive configuration menu that allows you to change/remove existing service(s) or add a new one. The example below is for setting up a google drive. For other remote services see listings for other services
Example Steps for Google Drive
Open putty SSH login session to Raspberry Pi and execute command below, then follow rclone config interactive prompts. You will be required to have a login account on the remote storage service.
Select new, At name> prompt, Enter your choice of service name eg gdmedia
At Storage> prompt enter appropriate menu number selection for google drive
Follow prompts and accept defaults where appropriate.
At Use auto config? select n
rclone will display a long url at link: prompt.
On RPI SSH session, click and hold left mouse button and highlight https url after link: (do not press enter key)
On computer web browser that is logged into service, right click, paste and go on browser url bar.
On computer web browser service security web page, Select user account if appropriate. On service security web page confirm access.
On confirmation page a Long verification code will be displayed. Use mouse left button to highlight code then right click to copy verification code.
On RPI SSH session at prompt Enter verification code> right click, paste verification code then Enter to accept
At team drive? prompt Select n if the drive is not part of a team
Review and accept settings when prompted.
Install Details for Other Remote Storage Services
For a Full listing Select Storage Systems pull down menu at top of rclone.org Home Page.
DropBox Setup Details
Note: When setting up rclone for DropBox on a headless RPI, you need to install rclone on a windows computer by downloading the appropriate zip file from https://rclone.org then unzip file. At the rclone unzipped folder press shift and mouse right click at same time then select Open command window here. This should open a command window in the unzipped rclone folder. Make sure you are logged into DropBox from default web browser. On command prompt window execute
rclone authorize "dropbox"
A DropBox web page will open requesting to Allow Rclone Access. After you Allow Access on the web browser DropBox page, the command prompt window will have a long security string. Clicked on the top left corner icon of command window then select Edit, Mark. Highlight the two lines of the security string and Enter to copy. Paste result into a text editor and combine both lines of output into one line. Copy the security string and paste into RPI SSH session at the rclone prompt. Respond to rclone prompts and you should have access to your DropBox drive from the RPI. T
How to Test rclone
To test connection to remote service. Execute the following command where gdmedia: is the name you gave your remote storage service. After a short delay this will list the files and folders in the root of your remote drive. Note Remote Paths are specified as remote:path where remote: is name you gave remote storage service.
List configured remote storage services
list All files on remote storage service. Change gdmedia to name you gave remote service
rclone ls gdmedia:
rclone lsd gdmedia:
To get command help, type rclone with no parameters or rclone --help
Example sync From RPI Folder media/motion To remote service name gdmedia: and media/motion folder
rclone sync -v /home/pi/pi-timolo/media/motion gdmedia:media/motion
- IMPORTANT: If the media/motion folder has a large number of files and you just want recent status then edit pi-timolo config.py motionRecentMax= variable to required recent entries. Then rclone the media/recent/motion folder or whatever foldermotionRecentDir = variable is set to.
Sample rclone Scripts
Several sample rclone- sync scripts are stored in the *rclone-samples folder. These are provided with the filename prefix rclone- and .sh extension. menubox.sh has the RCLONE menu pick for managing rclone script files. You can Edit, Run or use nano to Save one of these bash script files to another name an/or folder.
HINT: If you select EDIT menu pick for one of the rclone-samples scripts and want to save to another location or filename then open a file WriteOut ctrl-O then change the directory path and/or filename and save to new location. If you then use RUN menu pick, the saved script it will be changed executable if required and run.
IMPORTANT: The rclone- sample files in the rclone-samples folder will be overwritten during an upgrade so it is highly recommended that you copy a sample script to another name and/or folder or into the pi-timolo main folder so changes will not be lost.
Edit variables for your needs. Refer to rclone --help for command parameters.
- HINT: If you are syncing from several RPI's to the same remote storage service. You should change the variable remoteSyncDir to be unique for each Raspberry Pi computer syncing to the same remote storage service. This will avoid sync conflicts. It is also recommended that you clean remote Trash occasionally per example command below.
To test cleanup of trash. This was tested with google drive. DropBox does not work.
rclone cleanup gdmedia:
To copy a sample sync script
cd ~/pi-timolo cp rclone-sync.sh rclone-mysync.sh nano rclone-mysync.sh
Edit the variables as desired then ctrl-x y to save and exit nano
Change sync to Something Else
Sync makes the destination folder identical to the source folder. Files that are not in the source folder are sent to the destination remote name trash. You should be able to find them there. Rclone can do many things including copy. Type rclone --help for all the commands. You can copy one of the rclone sample batch files and replace the sync command with the copy or another command per variable rcloneParam. Copy new files and won't delete files on the destination that do Not exist on the source. Use with caution since this can generate more files and take more storage space on the remote storage name.
cd ~/pi-timolo cp rclone-samples/rclone-mo-sync-lockfile.sh ./ nano rclone-mo-sync-lockfile.sh
Use menubox.sh RCLONE, EDIT pick menu or nano directly and edit script variable rcloneParam=. Change sync to copy or move per comments, then ctrl-x y to save and exit. See rclone --help for other possible options
lockFileCheck=false # true= Checks for pi-timolo.sync file. false = No Check (case sensitive) rcloneName="gdmedia" # Name of Remote Storage Service syncRoot="/home/pi/pi-timolo" # Root Folder to Start localDir="media/motion" # Source Folder on Local remoteDir="mycam/motion" # Destination Folder on Remote rcloneParam="sync" # rclone option to perform Eg sync, copy, move # IMPORTANT: sync will make remoteDir identical to localDir # so remoteDir Files that do not exist on localDir will be Deleted.
Test run the scripts to make sure they run OK before adding as a crontab entry.
This script should copy only the differences and skip over anything that is already existing.
Use menubox.sh to customize sync script Files
You can also use menubox.sh to select and edit an existing rclone- sync script and use nano to customize and save it to a new filename. You can also test execution of the script from menubox.sh
How to Automate rclone
To avoid multiple instances of rclone running, a small batch file called rclone-sync.sh will only allow one instance to run. Run commands below to edit the rclone-sync.sh script variables.
cd ~/pi-timolo nano rclone-sync.sh
Edit variables for source, destination, remote storage name as required ctrl-x y to save and exit nano editor
Test script per commands below. After a short delay you should see sync progress.
The best way to automate rclone is to add a crontab entry to run sync script(s) at regular intervals. See example below
sudo crontab -e
Example entries to crontab per below. If using nano ctrl-x y to save changes and exit.
*/6 * * * * su pi -c "/home/pi/pi-timolo/rclone-mo-sync-recent.sh" > /dev/null 2>&1 */10 * * * * su pi -c "/home/pi/pi-timolo/rclone-mo-sync-lockfile.sh" > /dev/null 2>&1 0 5 * * MON su pi -c "/home/pi/pi-timolo/rclone-cleanup.sh" > /dev/null 2>&1 */10 * * * * su pi -c "/home/pi/pi-timolo/watch-app.sh" > /dev/null 2>&1
- First entry runs pi-timolo/rclone-mo-sync-recent.sh every six minutes and make the remote folder identical to the source folder.
- Second entry runs rclone-mo-sync-lockfile.sh at 10 minutes. Checks is pi-timolo.sync file exists indicating files ready to sync.
- Third entry runs rclone-cleanup.sh once a week at 5 am on Monday.
You can change the frequency to what every you like. Just make sure you leave enough time to upload files.
How to Install rclone (if Not already installed by pi-timolo-install.sh)
Rclone is installed with the pi-timolo-install.sh script at /usr/bin/rclone. If for some reason you want to run rclone-install.sh separately use the Quick Install curl script or manual install per below.
Quick Install (if not already installed by pi-timolo-install.sh)
Change to the folder where you want rclone-samples subfolder installed. This will normally be the /home/pi/pi-timolo folder but if you can install to another folder if required by changing cd to folder of your choice.
Step 1 Highlight curl command in code box below using mouse left button. Right click mouse in highlighted area and Copy.
Step 2 On RPI putty SSH or terminal session right click, select paste then Enter to download and run script.
curl -L https://raw.github.com/pageauc/pi-timolo/master/source/rclone-install.sh | bash
This will download and install latest rclone version to /usr/bin/rclone and upgrade rclone-samples folder scripts to latest GitHub versions
You can also download it separately using wget and manually install. This allows you to inspect script before executing.
wget -O rclone-install.sh https://raw.github.com/pageauc/pi-timolo/master/source/rclone-install.sh chmod +x rclone-install.sh ./rclone-install.sh
To upgrade/repair Rclone you can simply rerun the Quick Install curl script above, Run the pi-timolo-install.sh curl command or manual install. You can also upgrade pi-timolo from menubox.sh UPGRADE Main Menu pick.