There are several data storage types available to users. Each of which has different characteristics and policies, and it is suitable for different types of use. Please note that the cluster is intended to store data for as long as it is being processed and so none of the storage options are backed up.
The following table shows the different storage options that are currently available to users:
| Storage | Use | Exported to nodes | Total Capacity | Notes |
|---|---|---|---|---|
/users/ |
Home dir | Yes | 12TB | Quota: 150GB, No backup |
/mnt/data/ |
Local data | Yes | 30TB | No backup |
/mnt/scratch/ |
Compute | Yes | 93TB | No backup |
/tmp/users/ |
Compute | No | up to 1TB | No backup |
The easiest way to transfer data from/to the cluster is to use one of the standard
programs based on the SSH protocol such us scp or rsync.
The scp command creates a copy of a file, or a directory (if the -r flag is called) on a remote machine.
To copy data to the cluster:
scp [options] /source/path/to/object <user>@dmog.hw.ac.uk:/path/to/destinationTo copy data from the cluster:
scp [options] <user>@dmog.hw.ac.uk:/source/path/to/object /path/to/destinationFor a complete list of options available: man scp
rsync uses the same underlying protocol as scp, but it employs a special
delta transfer algorithm. It compares if there is any differences in the files
and only transfer those differences. Also, while copying if the connection drops,
it can pick up the transfer where it was left off.
The syntax to copy files to/from the cluster:
rsync [options] /source/path/to/object <user>@dmog.hw.ac.uk:/path/to/destination rsync [options] <user>@dmog.hw.ac.uk:/source/path/to/object /path/to/destination
For a list of options: man rsync