Remote Running

The analysis pipeline can be run remotely on computing clusters and allow us to efficiently process and analyze our large dataset without the need for powerful local hardware. The pipeline is tested on two different clusters, Genotoul and GAFL, as our computing resources.

If you want to perform the analysis on the Genotoul cluster, please make sure to include the following lines in the run_ezqtlseq_pipeline.slurm script:

module load bioinfo/snakemake-5.8.1
module load system/singularity-3.5.3

Otherwise, if you are working on the GAFL cluster, there is no need to include these lines as the required modules are already included on this cluster.

IMPORTANT: Please make sure that the singularity bind parameters and images location (see Configuration Settings) are relevant to the cluster used. Namely, to work on the Genotoul server, you would use:

BIND : "-B /work/project/gafl"
fastqc_bin    : "/work/project/gafl/tools/containers/fastqc_V0.11.8.sif"

Otherwise, to work on the GAFL server, you would use:

BIND : "-B /work2/project/gafl"
fastqc_bin    : "/work2/project/gafl/tools/containers/fastqc_V0.11.8.sif"

Generating DAG file

The DAG (Directed Acyclic Graph) shows the dependencies between the different steps in your pipeline and can help you understand the workflow of your pipeline. To generate a DAG for your pipeline, you can use one of these following commands:

# For samples
snakemake --configfile $CONFIG -s $RULES --dag | dot -Tpdf > dag.pdf
# For rules/steps 
snakemake --configfile $CONFIG -s $RULES --rulegraph | dot -Tpdf > dag.pdf

Ready to run

Before submitting a job to cluster, a dry run should be performed to simulate the job without actually executing it, and help identify any errors or issues in your pipeline configuration. For example, in the provided code below of the script run_ezqtlseq_pipeline.slurm, the first command is the dry run command, and the second command is the full run command. To perform a dry run, you would uncomment the first command and comment out the second one.

# Dry run (simulation)
snakemake --configfile $CONFIG -s $RULES -np -j $MAX_JOBS --cluster-config $CLUSTER_CONFIG --cluster "$CLUSTER" > snakemake_dryrun.out
# Full run
#snakemake --configfile $CONFIG -s $RULES -p -j $MAX_JOBS --cluster-config $CLUSTER_CONFIG --cluster "$CLUSTER" 

Then you can now run the run_ezqtlseq_pipeline.slurm script from your terminal:

./run_ezqtlseq_pipeline.slurm

To display the output of the dry run in the terminal, which you can use to check for any errors or issues:

cat snakemake_dryrun.out

If the head of the output file shows a list of jobs with their respective counts, this indicates that the dry run has been successful and there are no errors in the pipeline configuration.

Building DAG of jobs...
Job counts:
	count	jobs
	4	bam_stats
	1	bwa_index
	4	bwa_mapping_wow_merge
	1	combinevcf
	1	do_qtlseqR
	4	fastp
	4	fastqc
	1	filter_by_dp
	1	final_outs
	1	gatk4_filterSnps
	13	gatk4_gc
	13	gatk4_gdb
	13	gatk4_gvcf_map_file
	52	gatk4_hc
	1	gatk4_no_missing
	1	gatk4_ref_dict
	1	gatk4_select_snps_variants
	1	keep_biallelic
	1	keep_extract_bulks
	1	keep_variant_Parents
	1	multiqc_bam
	1	multiqc_fastqc
	4	remove_or_keep_duplicate
	1	vcf2table
	126
...

Now, you can proceed to submit the actual job to cluster with confidence. Remember to comment out the first command and uncomment the second one:

# Dry run (simulation)
#snakemake --configfile $CONFIG -s $RULES -np -j $MAX_JOBS --cluster-config $CLUSTER_CONFIG --cluster "$CLUSTER" > snakemake_dryrun.out
# Full run
snakemake --configfile $CONFIG -s $RULES -p -j $MAX_JOBS --cluster-config $CLUSTER_CONFIG --cluster "$CLUSTER" 

Then submit the full run job to the server:

sbatch run_ezqtlseq_pipeline.slurm

Once it submitted, you can see a message like this

Submitted batch job [JOB_ID]

Additionally, you can use the command:

 squeue -u [USER_NAME]

to check the status of your job and see if it's running, waiting, or completed. Once your job is completed, you can check the output files to ensure that the pipeline ran successfully.