Basic Project Structure_50759300

---

title: "TheNessLab : Basic Project Structure"

1. TheNessLab
2. The Ness Lab Wiki
3. HPCnode 1 Documentation

TheNessLab : Basic Project Structure

Created by Robert Ness, last modified on Jan 18, 2022

The idea here is how a default project should be organized. The basic idea is:

1) The project should "stand alone" and work only with things inside the project folder

2) Analysis contains code, but does not contain data or figures

3) raw_data contains the input data to the project but not data created by analysis. This should be sym linked from /research/data where raw data should be storded.

4) data contains the figures or  data created by analysis. These data should be symlinked into the scratch where they will be stored but not backed up because they can be recreated by analysis code.

5) Conda contains the conda environment used in the project

In the example below I made a new project called "wild_chlamy" using the reference genome and VCF for all chlamy as input data. 

species='chlamydomonas'
project='wild_chlamy'

#mkdirs
mkdir -p /research/projects/$species/$project/
mkdir -p /scratch/projects/$species/$project/data

cd /research/projects/$species/$project/

# Make a symlink to a scratch folder that holds your data until it gets deleted.

ln -s /scratch/projects/$species/$project/data data

# This folder is for things like  reference genomes or centralized data like VCFs/BAMs/FASTQs etc that are shared by multiple projects /research/data/{species}/

mkdir raw_data/     

#You can now link those raw data files into this folder

# examples of how to add raw data to you raw data folder using sym links
cd raw_data
# link the folder with the 5.3 reference genome in it
ln -s /research/references/chlamydomonas/5.3_chlamy_w_organelles_mt_minus/ 5.3_chlamy_w_organelles_mt_minus
cd ..

# All scripts that generate new data should be stored in analysis
mkdir analysis

#within this divide analysis in some logical way
cd analysis
mkdir snp_calling
cd snp_calling
# A good way to organize the data you create is any data from this analysis goes into a data folder of the same name
mkdir ../../data/snp_calling

#when your run your code in snp_calling you output it to data/snp_calling so that it is put into the scratch disk where we have more space
echo "I do science times" >../../data/snp_calling

cd ../../


# You should also make a conda environment just for this project so that all the versions etc are maintained for maximum reproducibility
mkdir conda

conda create --prefix /research/projects/$species/$project/conda/$project
conda config --append envs_dirs /research/projects/$species/$project/conda/


# You can also track the whole project via git
git init
# I would ignore data because it will take forever to upload and it can always be recreated from code.
touch .gitignore
echo "./raw_data" >>.gitignore
echo "./data" >>.gitignore

Document generated by Confluence on May 22, 2024 11:44

Atlassian