Exercises Series 2

In case you want a refresher, here are the General Guidelines

Exercise 8

Context

We have a basic implementation of our entire workflow. We have seen how to run Snakemake to generate a given target, and in this case we would run snakemake <output_of_substitution_count_rule>. We would like to be able to automatically generate the final output when calling snakemake without arguments.

Task

Implement a special rule so that the final output is generated by default when running snakemake without specifying a target.

Hints

Special rules

Remember how the default target is selected if not specified in command line (see the Executing Workflows section). Also, keep in mind that a rule is not required to have an output!

Structure of the special rule

You can define a rule at the top of the Snakefile that will take as input the final output that you want to generate (from the last rule you wrote). This rule should not have any output or any shell command, only an input directive.

---

Exercise 9

Context

We have now implemented all the steps of our pipeline. We would like to visualize the process with Snakemake's built-in DAG visualization feature.

Task

Visualize the entire workflow’s Directed Acyclic Graph.

Details

• Save the results as a PNG or SVG picture.
• Specify the right targets to visualize the entire workflow!

Relevant Wiki section

Executing workflows

Hint

Command

Command: snakemake --dag | dot -Tpng > dag.png

Targets

To visualize the entire workflow, you need to specify the results of the substitution count rule.

---

Exercise 10

Context

Several rules from our workflow have several parameters. We should make our workflow easily customizable by defining these parameters in a config file.

Task

Add a config file to specify parameter values for the workflow, and use it in the workflow's definition.

Details

• Try to identify the parameters that can be defined in the config file; there are at least three that are listed in the hint below
• Modify the rules to make use of the config file
• Remember that the content of the config file is stored in a dictionary named config

Relevant Wiki section

Config files

Hints

Parameters to implement

- Assembly file path
- Number of threads for BWA mem
- Prior mutation rate for bcftools call

---

Exercise 11

Context

Let's make our workflow easier to debug with some logs!

Task

Add a log directive to the alignment rule and the variant calling rule.

Relevant Wiki section

Threads, log, and benchmark

---

Exercise 12

Context

At the moment, samples are defined in a list at the top of the Snakefile (implemented in the first question of the previous exercises series. To further improve our workflow's usability, we would like to define samples in the config file, so they can easily be added, removed, or modified by the user.

Task

Implement a parameter in the config file to specify samples and associated read files.

Details

• Add samples as named parameters in the config file; think about how to specify the path to read files
• Modify the variant calling rule to use sample names from the config file
• Modify the alignment rule to get the path to the reads file for a sample from the config file
• Add the sample C to the list of samples (reads file path: data/samples/C.fastq)

Relevant Wiki section

Config files

Hints

Implementing samples in the config file

There are several ways we could implement this parameter. Here, we recommend adding samples as named parameters with the path to the reads file as value: ": "

Get reads file for a sample from the config file

For this, you will have to use an input function which returns the path to the reads file from the sample name, which is contained in the value of the wildcard "sample"

---

Exercise 13

Context

To make our workflow reproducible, we will make use of the Conda integration feature of Snakemake, so that users do not need to manually manage software dependencies. Note: the first execution of the workflow after adding Conda environments will take some time, because Conda will have to install all the software.

Task

Update the workflow to make use of Conda environments. You can decide how to organize these environments (single environment for the entire workflow, one environment per rule, ...).

Details

• Create one or several conda environment files for your workflow
• The tools to include in the environment are bwa (version 0.7.17), samtools (version 1.9), and bcftools (version 1.9)
• Add a conda directive to all rules that use these tools

Relevant Wiki section

Automatic software deployment with Conda

---

Bonus exercise 1

Context

Although our workflow does not implement many rules, we can break it down in modules so that maintaining and extending the workflow is easier in the future.

Task

Break down the workflow in modules in a way that makes sense to you.

Details

• Think of the best way to group rules in modules
• Update the Snakefile to use your new modules

Hints

Suggested organization

You could group together rules related to alignment and handling of BAM files in one snakefile, and rules related to variant calling and post-processing in another snakefile.

Relevant Wiki section

Modularization: snakefiles and subworkflows

---

Bonus exercise 2

Context

After adding conda environments, we now have all the files that will be in the final version of our workflow. It's a good time to organize the directory according to the official guidelines.

Task

Organize the workflow directory to follow the official guidelines.

Details

• Implement the proper directory structure
• Name and organize files according to the guidelines; add comments if you would like

Relevant Wiki section

Workflow organization: guidelines and good practices

---

Bonus exercise 3

Context

This is it, the workflow is now complete ! All that's left to do is to create an archive for redistribution.

Task

Create an archive of your workflow using Snakemake's built-in functionality

Relevant Wiki section

Workflow organization: guidelines and good practices

---

If you followed the workshop to this point, you’ll have a good example of Snakemake workflow that you can use as reference to build your own workflows. Hopefully, you are now familiar with the proper practices to organize, implement, and run a solid Snakemake workflow. There are several additional advanced concepts, and we cannot cover them all. In the next section, we will present the ones we thought are the most important most users.

If you are seriously interested in using Snakemake in your own work, it is advised to read the entire documentation once to learn about all the features of Snakemake.

Finally, feel free to contact us with any questions about Snakemake. Although we are not official experts and we may not be able to answer your question, we have been using it for some time and we may save you a day or two of head scratching, just because we encountered the exact same problem as you and spent a day scratching our own head ;)