Best practices

Best practices

Storage management

Avoid storing redundant files

A lot of storage capacity is wasted by maintaining multiple copies of the same file in different locations. This inefficiency can be easily avoided using symbolic links. From Wikipedia:

A symbolic link is a special type of file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution.

Meaning, a symbolic link allows you to create a file that looks and acts like the original data file but without creating a redundant copy of it. For example, if we wanted to include the large imputed-genotypes.csv file in both the gwas-project and geno-qc-project directories:

/home/user/
  |-- data/
      |-- imputed-genotypes.csv
  |-- gwas-project/
      |-- gwas-script.R
      |-- pheno-data.csv
  |-- geno-qc-project/
      |-- qc-script.R

the following commands would create symbolic links in the project directories without duplicating the original imputed-genotypes.csv file:

ln -s $HOME/data/imputed-genotypes.csv $HOME/gwas-project/
ln -s $HOME/data/imputed-genotypes.csv $HOME/geno-qc-project/

The home directory would then look like this:

/home/user/
  |-- data/
      |-- imputed-genotypes.csv
  |-- gwas-project/
      |-- gwas-script.R
      |-- pheno-data.csv
      |-- imputed-genotypes.csv
  |-- geno-qc-project/
      |-- qc-script.R
      |-- imputed-genotypes.csv

without requiring any additional storage space.

Another benefit of this approach is the symbolic links will always point to the same file. If we receive an updated version of imputed-genotypes.csv, we don't have to remember to make a copy of the new file in every project directory that includes it. Instead, we just replace the outdated copy and the symbolic links will automatically point to the updated version.

Data compression

Data compression can greatly reduce the size of large files (or directories) without losing any information. While many compression algorithms are available, we recommend using gzip.

Compressing files

Data compression is extremely effective for text files. For example, a 300 Mb tab-delimited spreadsheet file can be reduced to a ~60 Mb file using gzip. Not only does this take up less space but it can greatly increase transfer speeds when moving files to and from the cluster.

To compress a single file:

gzip gwas-data.csv

which will replace the original imputed-genotypes.csv file with a compressed version called imputed-genotypes.csv.gz. The original file can then be extracted with the following command:

gunzip gwas-data.csv.gz

Compressed files and R

In version 2.10, R gained the ability to directly read compressed files. This means we can use the same functions to load data into R regardless of whether its efficiently stored as a gzip compressed file or as uncompressed text file:

data <- read.table("imputed-genotypes.csv")
data <- read.table("imputed-genotypes.csv.gz")

In some cases, the data will actually be loaded more quickly when working with compressed files because of the reduced disk activity required.