General Conventions

Naming convention

File and directory names should be in lower camel case. Abbreviations such as DNA retain their capitalisation, e.g.:

ATACSeq/
sampleSheet.csv

Directory structure

The scripts and the data they run on follow a standardised directory structure so that scripts are generalisable.

Data

The general idea of the directory structure is that data is grouped first by the experimental technique used to generate it, and then by project. Each project has a 1_raw/ directory, which contains the raw data, and then a directory for each subsequent step of the processing pipeline. Directories are numbered by the step in the pipeline so that users can clearly see which stage the data is at and in what order it was processed. Quality control (QC) results are found within the directory of the data or process they are QC-ing.

The data directory structure is as follows:

ATACSeq
├── MRC
│   ├── 0_metadata
│   ├── 1_raw
│   │   └── fastqc
│   ├── 2_trimmed
│   │   └── fastqc
│   ├── 3_aligned
│   ├── 4_calledPeaks
│   └── 5_peakCounts
└── DNAm, WGBS etc.

Scripts

In parallel to the experimental folders, there is a scripts directory. Within this, the scripts are organised into folders by experiment, where the name should match the name where the corresponding data is stored. These are first split into sequencing/ and array/ directories, as these differ in initial steps of the processing pipeline. As processing becomes more specific, the scripts referring to these steps are organised by the experimental technique used to generate the data.

Scripts are organised within each data type directory into preprocessing/, config/, analysis/ and logFiles/. preprocessing/ contains the scripts called by the job submission script, found in preprocessing/jobSubmission/. Config files containing base variables for the job submission scripts are found in config/ while output and error files should be output to logFiles/. README files in the data type directory specify details of job submission scripts, the order in which these should be submitted and which scripts these call. Users running standard processing should be able to submit the job submission script from the details in the README, with a curated config.txt file, without editing the underlying scripts (in preprocessing/ or analysis/).

The script directory structure is in the format:

Research_Project-MRCXXXXXX
├── scripts/
│   ├── array/
│   ├── general/
│   ├── integrative/
│   └── sequencing/
│       ├── ATACSeq
│       │   ├── analysis/
│       │   |   ├── jobSubmission/
│       │   ├── config/
│       │   ├── logFiles/
│       │   ├── preprocessing/
│       │   |   ├── jobSubmission/
│       │   │   ├──alignment.sh
│       │   │   ├──shiftAlignedReads.sh
│       │   │   └──peakCalling.sh
│       │   └── README    
│       ├── preScripts
│       │   └── fastqc.sh
│       └── RNASeq
│       │   ├── analysis
│       │   |   ├── jobSubmission
│       │   ├── config
│       │   ├── logFiles
│       │   ├── preprocessing
│       │   |   ├── jobSubmission
│       │   └── README.txt 
│       └── WGBS etc.
└── DNAm, WGBS etc.

Config file

As far as possible, parameters and variables should be defined in config files, that processing scripts call. We want to limit how much these need to be edited. Using the standard file structure for data, bash scripts take advantage of a standard config file named config.txt, located in the config folder. The config file needs a "local"/command line variable with the project folder name to be declared prior to using the config file.

The format of the config file is

## Data filepaths
## Reference data filepaths
## Software/script filepaths
## conda filepaths
## parameters

The project name defined on the command line (as the first supplied flag) is passed to the config file as $PROJECT. This is required by the script and is used to define all following data directories, e.g.:

## Data filepaths
export DATADIR=<path/to/data/directory/<data-modality>/${PROJECT}
export METADIR=${DATADIR}/0_metadata
export RAWDATADIR=${DATADIR}/1_raw
export FASTQCDIR=${RAWDATADIR}/fastqc
export TRIMDIR=${DATADIR}/2_trimmed
export ALIGNEDDIR=${DATADIR}/3_aligned

Some modalities also have an r config file, config.r, this contains default parameters that will be used if no alternatives are provided.

Where R parameters are likely to be changed by users, those scripts will have specific configs, with distinct names. These need to be specified in the README and a description of how to populate. If a user wants to run their own parameters, they can provide an additional config file which will override the default parameters.

Sample sheets

All* modalities expect a sample sheet of a specified format to be found here ${DATADIR}/0_metadata/sampleSheet.csv This can be used to automate the scripts and make them more generalisable.

The following column names (case-sensitive) are required by the scripts:

cohort	sampleID	sampleName	target	fraction	individualID	tissue	readName

If control data is present, these columns must also be specified:

controlID	controlName	controlReadName

Messages

All updates/progress messages derived from processing scripts rather than job submission. Job submission scripts might want time stamps at start and end to set processing time.