Best practices

Best practices

Storage management

Avoid storing redundant files

A lot of storage capacity is wasted by maintaining multiple copies of the same file in different locations. This inefficiency can be easily avoided using symbolic links. From Wikipedia:

A symbolic link is a special type of file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution.

Meaning, a symbolic link allows you to create a file that looks and acts like the original data file without creating a redundant copy. For example, if we wanted to include the large imputed-genotypes.csv file in both the gwas-project and geno-qc-project directories:

/home/user/
  |-- data/
      |-- imputed-genotypes.csv
  |-- gwas-project/
      |-- gwas-script.R
      |-- pheno-data.csv
  |-- geno-qc-project/
      |-- qc-script.R

the following commands would create symbolic links in each project directory without duplicating the original imputed-genotypes.csv file:

ln -s $HOME/data/imputed-genotypes.csv $HOME/gwas-project/
ln -s $HOME/data/imputed-genotypes.csv $HOME/geno-qc-project/

The home directory would then look like this:

/home/user/
  |-- data/
      |-- imputed-genotypes.csv
  |-- gwas-project/
      |-- gwas-script.R
      |-- pheno-data.csv
      |-- imputed-genotypes.csv
  |-- geno-qc-project/
      |-- qc-script.R
      |-- imputed-genotypes.csv

without requiring any additional storage space.

Another benefit of this approach is the symbolic links will always point to the same source file, even if it is changed. If we receive an updated version of imputed-genotypes.csv we only need to replace the reference copy in the data directory, the symbolic links in each project directory will automatically point to the newest version. This saves you the trouble of manually replacing every instance of the outdated file.

Data compression

Data compression can greatly reduce the size of large files (or directories) without losing any information. While many compression algorithms are available, we recommend using gzip.

Compressing files

Data compression is extremely effective for text files. For example, a 300 Mb tab-delimited spreadsheet file can be reduced to ~60 Mb using gzip. Not only does this take up must less space, it can also greatly increase transfer speeds when moving files to and from the cluster.

To compress a single file:

gzip imputed-genotypes.csv

will replace the original imputed-genotypes.csv file with a compressed version called imputed-genotypes.csv.gz. The original file can then be extracted with the following command:

gunzip gwas-data.csv.gz

Compressed files and R

Starting in version 2.10, R gained the ability to directly read compressed files. This means we can use the same functions to load data into R regardless of whether its efficiently stored as a gzip compressed file or as uncompressed text file:

data <- read.table("imputed-genotypes.csv")
data <- read.table("imputed-genotypes.csv.gz")

In some cases, the data will actually load more quickly when working with compressed files because of the reduced disk activity required.