Skip to content

Commit

Permalink
Formatted
Browse files Browse the repository at this point in the history
  • Loading branch information
@monaghaa
monaghaa committed Jun 8, 2020
1 parent 1e50375 commit aa0ca5c
Showing 1 changed file with 69 additions and 91 deletions.
160 changes: 69 additions & 91 deletions docs/compute/rclone.md
View file
Original file line number Diff line number Diff line change
@@ -1,28 +1,16 @@
### Use rclone to back up directory on macOS to PetaLibrary
## Use rclone to back up directory on macOS to PetaLibrary

This is a guide to making a backup of a local directory on a macOS
system to a PetaLibrary allocation. rclone is available for many
architectures, and while the exact commands may differ, the general
steps to follow are the same.
This is a guide to making a backup of a local directory on a macOS system to a PetaLibrary allocation. rclone is available for many architectures, and while the exact commands may differ, the general steps to follow are the same.

### Generate public/private keypair on login.rc.colorado.edu


# Generate public/private keypair on login.rc.colorado.edu

ssh to login.rc.colorado.edu and generate an ssh keypair. You don't
need to complete this step on a login node if you have ssh-keygen
installed on your system. Be sure to generate a key in PEM format.
Here is the command to generate a key in PEM format with a unique
name:

In this step you will login to an RC login node (`ssh login.rc.colorado.edu`) and generate an ssh keypair. You don't need to complete this step on a login node if you have ssh-keygen installed on your system. Be sure to generate a key in PEM format. Here is the command to generate a key in PEM format with a uniquename, after you have logged in:
```
$ ssh-keygen -f ~/.ssh/rclone_ssh_key -m PEM
```
You will be prompted to enter a passphrase to protect the key. For automated backups, you will either want a key with no passphrase, or to set a passphrase and use something like ssh-agent to hold the key in memory. The easiest option is to not set a passphrase. Here is sample output from running ssh-ksygen:

You will be prompted to enter a passphrase to protect the key. For
automated backups, you will either want a key with no passphrase,
or to set a passphrase and use something like ssh-agent to hold the
key in memory. The easiest option is to not set a passphrase. Here
is sample output from running ssh-ksygen:

```
$ ssh-keygen -f ~/.ssh/rclone_ssh_key -m PEM
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Expand All @@ -32,6 +20,7 @@ Your public key has been saved in /home/jesse/.ssh/rclone_ssh_key.pub.
The key fingerprint is:
SHA256:ftryJYQ2LRnNQ/fSzm+iB0Tg8/LkePqVyeDky2z3JvA jesse@login10
The key's randomart image is:
+---[RSA 2048]----+
| o.. |
| = ..o |
Expand All @@ -43,79 +32,57 @@ The key's randomart image is:
| .+ B.+Eo.|
| .o+o*o +.|
+----[SHA256]-----+
```

Your home directory should now have the files ~/.ssh/rclone_ssh_key
and ~/.ssh/rclone_ssh_key.pub. rclone_ssh_key is your private key,
be careful with it as anyone with this key can access resources
that trust the public key.

Your home directory should now have the files `~/.ssh/rclone_ssh_key` and `~/.ssh/rclone_ssh_key.pub`. The `rclone_ssh_key` is your private key, be careful with it as anyone with this key can access resources that trust the public key.

### Copy public key to your authorized_keys file

# Copy public key to your authorized_keys file

Copy the newly generated public key rclone_ssh_key.pub to your
authorized_keys file:

Copy the newly generated public key rclone_ssh_key.pub to your authorized_keys file:
```
$ cat ~/.ssh/rclone_ssh_key.pub >> ~/.ssh/authorized_keys
```
This will append the contents of `rclone_ssh_key.pub` to the file authorized_keys (authorized_keys will be created if it does not exist). Now anyone with your private key (`rclone_ssh_key`) can login to hosts that mount home directories from RC core storage.

This will append the contents of rclone_ssh_key.pub to the file
authorized_keys (authorized_keys will be created if it does not
exist). Now anyone with your private key (rclone_ssh_key) can log
in to hosts that mount home directories from RC core storage.

### Copy private key to the backup source host

This next step can't be documented exactly as every client system will be different, but use a secure method (such as scp or sftp) to copy the private key in `~/.ssh/rclone_ssh_key` to the system that you want to back up to PetaLibrary (e.g., your laptop or lab server).

# Copy private key to the backup source host
* WINDOWS: Windows host users, make sure you run your copy client (e.g. WSL2, WinSCP, ect) as Admin. Change directories to rclone file location and run:

I can't document this step as every client system will be different,
but use a secure method (such as scp or sftp) to copy the private
key in ~/.ssh/rclone_ssh_key to the system that you want to back
up to PetaLibrary.
```$ .\rclone.exe config```
...then select n) New Remote

--WINDOWS: Windows host users, make sure you run your copy client (e.g. WSL2, WinSCP, ect) as Admin
cd to rclone file location and run
$ .\rclone.exe config
Then select n) New Remote


# Install rclone on the backup source host
### Install rclone on the backup source host

rclone is available download here for a variety of architectures:

--WINDOWS: Windows hosts users, download and unzip the rclone folder to desired location. The folder contains the rclone.exe. To run config from the commandline (e.g. WSL2, Powershell, ect)


https://rclone.org/downloads/

* WINDOWS: Windows hosts users, download and unzip the rclone folder to desired location. The folder contains the rclone.exe. Run config from the commandline (e.g. WSL2, Powershell, ect). This is covered in the next section.


# Configure rclone

rclone is a command line program that requires you to configure
endpoints. Once your endpoints are configured, you can copy data
to/from your local system to configured endpoints. We will configure
an sftp endpoint in rclone that points to the data transfer node
(DTN) hosts.

### Configure rclone

rclone is a command line program that requires you to configure endpoints. Once your endpoints are configured, you can copy data to/from your local system to configured endpoints. We will configure an sftp endpoint in rclone that points to the data transfer node (DTN) hosts.

I will use rclone to create an sftp endpoint with the following
settings:

In this example we use rclone to create an sftp endpoint with the following settings:
```
name: cu_rc_dtn
type: sftp
host: dtn-data.rc.int.colorado.edu
key_file = /Users/jesse/.ssh/rclone_ssh_key
user = jesse
```
rclone is interactive and will prompt you for all of the above information. Here is the ouput of my interactive session when creating an endpoint with the above settings:

```$ rclone config```

rclone is interactive and will prompt you for all of the above
information. Here is the ouput of my interactive session when
creating an endpoint with the above settings:
* WINDOWS: Windows host users, cd to rclone file location and run
```$ .\rclone.exe config```

% /tmp/rclone config
--WINDOWS: Windows host users, cd to rclone file location and run
$ .\rclone.exe config

```
2020/05/18 15:00:00 NOTICE: Config file "/Users/jesse/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
Expand Down Expand Up @@ -288,49 +255,60 @@ c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>
```


### Verify rclone config

# Verify rclone config

You can verify your settings by running 'rclone config show'. Mine
looked like this after running through the initial configuration:
You can verify your settings by running 'rclone config show'. The results from the example above looked like this after running through the initial configuration:

```
$ rclone config show
--WINDOWS: Windows host users
$ .\rclone.exe config
```
* WINDOWS: Windows host users
```$ .\rclone.exe config```

```
[cu_rc_dtn]
type = sftp
host = dtn-data.rc.int.colorado.edu
key_file = /Users/jesse/.ssh/rclone_ssh_key
user = jesse
md5sum_command = none
sha1sum_command = none
```


### Test rclone

# Test rclone

The syntax for using rclone to synchronize a local directory
'/tmp/local_backup_dir' to a directory 'pl_backup_dir' in a PetaLibrary
allocation named "pl_allocation", the command would be:

rclone sync /tmp/local_backup_dir cu_rc_dtn:/pl/active/pl_allocation/pl_backup_dir
_Example_: The syntax for using rclone to synchronize a local directory `/tmp/local_backup_dir` to a directory `pl_backup_dir` in a PetaLibrary allocation named `pl_allocation`, the command (executed from your laptop or lab server) would be:

This should synchronize the data in the two directories. The source
directory will not be modified, and pl_backup_dir will have files
added/removed to match the contents of local_backup_dir.
```$ rclone sync /tmp/local_backup_dir cu_rc_dtn:/pl/active/pl_allocation/pl_backup_dir```

This should synchronize the data in the two directories. The source directory will not be modified, and `pl_backup_dir` will have files added/removed to match the contents of `local_backup_dir`.

### Run rclone on a schedule
On unix/macOS hosts, you can set up a cron job to run the rclone sync job regularly. To create a crontab entry type:

# Run rclone on a schedule
On unix/macOS hosts, you can set up a cron job to run the rclone
sync job regularly. A sample cron job looks like:
```$ crontab -e```

And then add the line for the automated job you want to run, e.g.:
```
0 4 * * 1 rclone sync /tmp/local_backup_dir cu_rc_dtn:/pl/active/pl_allocation/pl_backup_dir

This would run the rclone sync command every Monday at 4am. Windows
systems can set up scheduled tasks to run rclone automatically.

--WINDOWS: Windows host users, if you do not have WSL2 installed use taskschd.msc (Task Scheduler) as Admin and run Action=>Create Basic Task
```
This would run the rclone sync command every Monday at 4am. The syntax for a crontab entry is:

```
* * * * * command
* - minute (0-59)
* - hour (0-23)
* - day of the month (1-31)
* - month (1-12)
* - day of the week (0-6, 0 is Sunday)
command - command to execute
(from left-to-right)
```

Windows systems can set up scheduled tasks to run rclone automatically.

* WINDOWS: Windows host users, if you do not have WSL2 installed use taskschd.msc (Task Scheduler) as Admin and run Action=>Create Basic Task

0 comments on commit aa0ca5c

Please sign in to comment.