Exercises series 2

In this second series of exercises, we will improve our workflow to make it more customizable, portable and maintainable.

---

Exercise 1

Context

When working with real data, the alignment process is computationally expensive; fortunately, it can be parallelized very efficiently. We would like to make use of this property by parallelizing the alignment step in our workflow.

Task

Add a threads directive to the alignment rule, and test its effect.

Details

• The number of threads is given to bwa mem with the parameter -t
• Specify a default value of 1 for the number of threads
• If you are running Snakemake locally, test the effect of the number of threads on the job's runtime. What happens if you do not specify --cores when running Snakemake ?

Relevant Wiki section

Threads, log, and benchmark

---

Exercise 2

Context

Several rules from our workflow have several parameters. We should make our workflow easily customizable by defining these parameters in a config file.

Task

Add a config file to specify parameter values for the workflow, and use it in the workflow's definition.

Details

• Try to identify the parameters that can be defined in the config file; there are at least three that are listed in the hint below
• Modify the rules to make use of the config file
• Remember that the content of the config file is stored in a dictionary named config

Relevant Wiki section

Config files

Hints

Parameters to implement

- Assembly file path
- Number of threads for BWA mem
- Prior mutation rate for bcftools call

---

Exercise 3

Context

Let's make our workflow easier to debug and optimize with some logs and benchmarks!

Task

Add a log and benchmark directives to the alignment rule and the variant calling rule.

Relevant Wiki section

Threads, log, and benchmark

---

Exercise 4

Context

At the moment, samples are defined in a list at the top of the Snakefile (implemented in the first question of the previous exercises series. To further improve our workflow's usability, we would like to define samples in the config file, so they can easily be added, removed, or modified by the user.

Task

Implement a parameter in the config file to specify samples and associated read files.

Details

• Add samples as named parameters in the config file; think about how to specify the path to read files
• Modify the variant calling rule to use sample names from the config file
• Modify the alignment rule to get the path to the reads file for a sample from the config file
• Add the sample C to the list of samples (reads file path: data/samples/C.fastq)

Relevant Wiki section

Config files

Hints

Implementing samples in the config file

There are several ways we could implement this parameter. Here, we recommend adding samples as named parameters with the path to the reads file as value: ": "

Get reads file for a sample from the config file

For this, you will have to use an input function which returns the path to the reads file from the sample name, which is contained in the value of the wildcard "sample"

---

Exercise 5

Context

To make our workflow reproducible, we will make use of the Conda integration feature of Snakemake, so that users do not need to manually manage software dependencies. Note: the first execution of the workflow after adding Conda environments will take some time, because Conda will have to install all the software.

Task

Update the workflow to make use of Conda environments. You can decide how to organize these environments (single environment for the entire workflow, one environment per rule, ...).

Details

• Create one or several conda environment files for your workflow
• The tools to include in the environment are bwa (version 0.7.17), samtools (version 1.9), and bcftools (version 1.9)
• Add a conda directive to all rules that use these tools

Relevant Wiki section

Automatic software deployment with Conda

---

Bonus exercise 1

Context

Although our workflow does not implement many rules, we can break it down in modules so that maintaining and extending the workflow is easier in the future.

Task

Break down the workflow in modules in a way that makes sense to you.

Details

• Think of the best way to group rules in modules
• Update the Snakefile to use your new modules

Hints

Suggested organization

You could group together rules related to alignment and handling of BAM files in one snakefile, and rules related to variant calling and post-processing in another snakefile.

Relevant Wiki section

Modularization: snakefiles and subworkflows

---

Bonus exercise 2

Context

After adding conda environments, we now have all the files that will be in the final version of our workflow. It's a good time to organize the directory according to the official guidelines.

Task

Organize the workflow directory to follow the official guidelines.

Details

• Implement the proper directory structure
• Name and organize files according to the guidelines; add comments if you would like

Relevant Wiki section

Workflow organization: guidelines and good practices

---

Bonus exercise 3

Context

This is it, the workflow is now complete ! All that's left to do is to create an archive for redistribution.

Task

Create an archive of your workflow using Snakemake's built-in functionality

Relevant Wiki section

Workflow organization: guidelines and good practices

---

If you followed the workshop to this point, you’ll have a good example of Snakemake workflow that you can use as reference to build your own workflows. Hopefully, you are now familiar with the proper practices to organize, implement, and run a solid Snakemake workflow. There are several additional advanced concepts, and we cannot cover them all. In the next section, we will present the ones we thought are the most important most users.

If you are seriously interested in using Snakemake in your own work, it is advised to read the entire documentation once to learn about all the features of Snakemake.

Finally, feel free to contact us with any questions about Snakemake. Although we are not official experts and we may not be able to answer your question, we have been using it for some time and we may save you a day or two of head scratching, just because we encountered the exact same problem as you and spent a day scratching our own head ;)