Manage rclone Remote Storage File Transfer

Manage transfer of media files to an Internet Remote Storage Service. Eg Google Drive, DropBox, Etc.

• Introduction
• Configure a Remote Storage Service
• How to Test rclone sync
• Sample rclone- scripts
• Change sync to Something Else
• How to Automate using crontab

NOTICE : Rclone is the speed-camera default installed by speed-install.sh and is at /usr/bin/rclone. If you have another project where you want rclone support see my rclone4pi GitHub repository

Introduction

Rclone is a command line program to sync files and directories to and from many remote storage services. This utility can be used to update speed-camera media folders/files to one or more remote services. For more information about rclone See https://rclone.org and/or https://github.com/ncw/rclone

How to Configure a Remote Storage Service

To configure one rclone remote storage service on a raspberry pi computer follow the steps below. If a service is already installed you will see an interactive configuration menu that allows you to change/remove existing service(s) or add a new one. The example below is for setting up a google drive. For other remote services see listings for other services

Example Steps for Google Drive

• Open putty SSH login session to Raspberry Pi and execute command below, then follow rclone config interactive prompts. You will be required to have a login account on the remote storage service.

  rclone config
  

• Select new, At name> prompt, Enter your choice of service name eg gdmedia

• At Storage> prompt enter number for google drive

• Follow prompts and accept defaults where appropriate.

• At Use auto config? select n

• rclone will display a long url at link: prompt.

• On RPI SSH session, click and hold left mouse button and highlight https url after link: (do not press enter key)

• On computer web browser that is logged into service, right click, paste and go on browser url bar.

• On computer web browser service security web page, Select user account if appropriate. On service security web page confirm access.

• On confirmation page a Long verification code will be displayed. Use mouse left button to highlight code then right click to copy verification code.

• On RPI SSH session at prompt Enter verification code> right click, paste verification code then Enter to accept

• At team drive? prompt Select n if the drive is not part of a team

• Review and accept settings when prompted.

Install Details for Other Remote Storage Services

• Google Drive
• Drop Box
• Amazon Drive
• Amazon S3
• Microsoft OneDrive
• Backblaze B2
• Box

For a Full listing Select Storage Systems pull down menu at top of rclone.org Home Page.

DropBox Setup Details

Note: When setting up rclone for DropBox on a headless RPI, you need to install rclone on a windows computer by downloading the appropriate zip file from https://rclone.org then unzip file. At the rclone unzipped folder press shift and mouse right click at same time then select Open command window here. This should open a command window in the unzipped rclone folder. Make sure you are logged into DropBox from default web browser. On command prompt window execute

rclone authorize "dropbox"

A DropBox web page will open requesting to Allow Rclone Access. After you Allow Access on the web browser DropBox page, the command prompt window will have a long security string. Clicked on the top left corner icon of command window then select Edit, Mark. Highlight the two lines of the security string and Enter to copy. Paste result into a text editor and combine both lines of output into one line. Copy the security string and paste into RPI SSH session at the rclone prompt. Respond to rclone prompts and you should have access to your DropBox drive from the RPI. T

How to Test rclone

To test connection to remote service. Execute the following command where gdmedia: is the name you gave your remote storage service. After a short delay this will list the files and folders in the root of your remote drive. Note Remote Paths are specified as remote:path where remote: is name you gave remote storage service.

List configured remote storage services

rclone listremotes

list All files on remote storage service. Change gdmedia to name you gave remote service

rclone ls gdmedia:

list directories

rclone lsd gdmedia:

To get command help, type rclone with no parameters or rclone --help

Example sync From RPI Folder media/motion To remote service name gdmedia: and media/motion folder

rclone sync -v /home/pi/speed-camera/media/images gdmedia:speed/images

• IMPORTANT: If the media/motion folder has a large number of files and you just want recent status then edit speed-camera config.py motionRecentMax= variable to required recent entries. Then rclone the media/recent/motion folder or whatever foldermotionRecentDir = variable is set to.

Sample rclone Scripts

Several sample rclone- sync scripts are stored in the *rclone-samples folder. These are provided with the filename prefix rclone- and .sh extension. menubox.sh has the RCLONE menu pick for managing rclone script files. You can Edit, Run or use nano to Save one of these bash script files to another name an/or folder.

• HINT: If you select EDIT menu pick for one of the rclone-samples scripts and want to save to another location or filename then open a file WriteOut ctrl-O then change the directory path and/or filename and save to new location. If you then use RUN menu pick, the saved script it will be changed executable if required and run.

• IMPORTANT: The rclone- sample files in the rclone-samples folder will be overwritten during an upgrade so it is highly recommended that you copy a sample script to another name and/or folder or into the speed-camera main folder so changes will not be lost.

Edit variables for your needs. Refer to rclone --help for command parameters.

rclone --help

• HINT: If you are syncing from several RPI's to the same remote storage service. You should change the variable remoteSyncDir to be unique for each Raspberry Pi computer syncing to the same remote storage service. This will avoid sync conflicts. It is also recommended that you clean remote Trash occasionally per example command below.

To test cleanup of trash. This was tested with google drive. DropBox does not work.

rclone cleanup gdmedia:

To copy a sample sync script

cd ~/speed-camera
cp rclone-sync.sh rclone-mysync.sh
nano rclone-mysync.sh

Edit the variables as desired then ctrl-x y to save and exit nano

Change sync to Something Else

Sync makes the destination folder identical to the source folder. Files that are not in the source folder are sent to the destination remote name trash. You should be able to find them there. Rclone can do many things including copy. Type rclone --help for all the commands. You can copy one of the rclone sample batch files and replace the sync command with the copy or another command per variable rcloneParam. Copy new files and won't delete files on the destination that do Not exist on the source. Use with caution since this can generate more files and take more storage space on the remote storage name.

cd ~/speed-camera
cp rclone-samples/rclone-security-sync ./
nano rclone-security-sync.sh

Use menubox.sh RCLONE, EDIT pick menu or nano directly and edit script variable rcloneParam=. Change sync to copy or move per comments, then ctrl-x y to save and exit. See rclone --help for other possible options

lockFileCheck=false             # true= This is NOT USED (for pi-timolo only)
rcloneName="gdmedia"            # Name of Remote Storage Service
syncRoot="/home/pi/speed-camera"   # Root Folder to Start
localDir="media/images"         # Source Folder on Local
remoteDir="speedcam/images"     # Destination Folder on Remote
rcloneParam="sync"       # rclone option to perform  Eg  sync, copy, move
                         # IMPORTANT: sync will make remoteDir identical to localDir
                         # so remoteDir Files that do not exist on localDir will be Deleted.

Test changes

Test run the scripts to make sure they run OK before adding as a crontab entry.

./rclone-mo-copy.sh

This script should copy only the differences and skip over anything that is already existing.

Use menubox.sh to customize sync script Files

You can also use menubox.sh to select and edit an existing rclone- sync script and use nano to customize and save it to a new filename. You can also test execution of the script from menubox.sh

How to Automate rclone

To avoid multiple instances of rclone running, a small batch file called rclone-sync.sh will only allow one instance to run. Run commands below to edit the rclone-sync.sh script variables.

cd ~/speed-camera
nano rclone-sync.sh

Edit variables for source, destination, remote storage name as required ctrl-x y to save and exit nano editor

Test script per commands below. After a short delay you should see sync progress.

./rclone-security-sync-recent.sh

The best way to automate rclone is to add a crontab entry to run sync script(s) at regular intervals. See example below

sudo crontab -e

Example entries to crontab per below. If using nano ctrl-x y to save changes and exit.

*/6 * * * * su pi -c "/home/pi/speed-camera/rclone-security-sync-recent.sh >/dev/null 2>&1"
0 5 * * MON su pi -c "/home/pi/speed-camera/rclone-cleanup.sh >/dev/null 2>&1"

• First entry runs speed-camera/rclone-security-sync-recent.sh every six minutes and make the remote folder identical to the source folder.
• Second entry runs rclone-cleanup.sh once a week at 5 am on Monday.

You can change the frequency to what every you like. Just make sure you leave enough time to upload files.