-
Notifications
You must be signed in to change notification settings - Fork 516
/
cli.rst
248 lines (156 loc) · 10 KB
/
cli.rst
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
.. _executable:
======================
Command line interface
======================
This part of the documentation describes the ``snakemake`` executable. Snakemake
is primarily a command-line tool, so the ``snakemake`` executable is the primary way
to execute, debug, and visualize workflows.
.. user_manual-snakemake_envvars:
-------------------------------
Important environment variables
-------------------------------
Snakemake caches source files for performance and reproducibility.
The location of this cache is determined by the `appdirs <https://github.com/ActiveState/appdirs>`_ package.
If you want to change the location on a unix/linux system, you can define an override path via the environment variable ``XDG_CACHE_HOME``.
.. user_manual-snakemake_options:
-----------------------------
Useful Command Line Arguments
-----------------------------
If called with the number of cores to use, i.e.
.. code-block:: console
$ snakemake --cores 1
Snakemake tries to execute the workflow specified in a file called ``Snakefile`` in the same directory (the Snakefile can be given via the parameter ``-s``).
By issuing
.. code-block:: console
$ snakemake -n
a dry-run can be performed.
This is useful to test if the workflow is defined properly and to estimate the amount of needed computation.
Further, the reason for each rule execution can be printed via
.. code-block:: console
$ snakemake -n -r
Importantly, Snakemake can automatically determine which parts of the workflow can be run in parallel.
By specifying more than one available core, i.e.
.. code-block:: console
$ snakemake --cores 4
one can tell Snakemake to use up to 4 cores and solve a binary knapsack problem to optimize the scheduling of jobs.
If the number is omitted (i.e., only ``--cores`` is given), the number of used cores is determined as the number of available CPU cores in the machine.
Snakemake workflows usually define the number of used threads of certain rules. Sometimes, it makes sense to overwrite the defaults given in the workflow definition.
This can be done by using the ``--set-threads`` argument, e.g.,
.. code-block:: console
$ snakemake --cores 4 --set-threads myrule=2
would overwrite whatever number of threads has been defined for the rule ``myrule`` and use ``2`` instead.
Similarly, it is possible to overwrite other resource definitions in rules, via
.. code-block:: console
$ snakemake --cores 4 --set-resources myrule:partition="foo"
Both mechanisms can be particularly handy when used in combination with :ref:`cluster execution <cluster>`.
Dealing with very large workflows
---------------------------------
If your workflow has a lot of jobs, Snakemake might need some time to infer the dependencies (the job DAG) and which jobs are actually required to run.
The major bottleneck involved is the filesystem, which has to be queried for existence and modification dates of files.
To overcome this issue, Snakemake allows to run large workflows in batches.
This way, fewer files have to be evaluated at once, and therefore the job DAG can be inferred faster.
By running
.. code-block:: console
$ snakemake --cores 4 --batch myrule=1/3
you instruct to only compute the first of three batches of the inputs of the rule ``myrule``.
To generate the second batch, run
.. code-block:: console
$ snakemake --cores 4 --batch myrule=2/3
Finally, when running
.. code-block:: console
$ snakemake --cores 4 --batch myrule=3/3
Snakemake will process beyond the rule ``myrule``, because all of its input files have been generated, and complete the workflow.
Obviously, a good choice of the rule to perform the batching is a rule that has a lot of input files and upstream jobs, for example a central aggregation step within your workflow.
We advice all workflow developers to inform potential users of the best suited batching rule.
.. _profiles:
--------
Profiles
--------
Adapting Snakemake to a particular environment can entail many flags and options.
Therefore, since Snakemake 4.1, it is possible to specify configuration profiles
to be used to obtain default options.
Two kinds of profiles are supported:
* A global profile that is defined in a system-wide or user-specific configuration directory (on Linux, this will be ``$HOME/.config/snakemake`` and ``/etc/xdg/snakemake``, you can find the answer for your system via ``snakemake --help``).
* A workflow specific profile that is defined via a flag (``--workflow-profile``) or searched in a default location (``profile/default``) in the working directory or next to the Snakefile.
The workflow specific profile is meant to be used to define default options for a particular workflow, like providing constraints for certain custom resources the workflow uses (e.g. ``api_calls``) or overwriting the threads and resource definitions of individual rules without modifying the workflow code itself.
In contrast, the global profile is meant to be used to define default options for a particular environment, like the default cluster submission command or the default number of jobs to run in parallel.
For example, the command
.. code-block:: console
$ snakemake --profile myprofile
would expect a folder ``myprofile`` in per-user and global configuration directories (on Linux, this will be ``$HOME/.config/snakemake`` and ``/etc/xdg/snakemake``, you can find the answer for your system via ``snakemake --help``).
Alternatively, an absolute or relative path to the profile folder can be given.
The default profile to use when no ``--profile`` argument is specified can also be set via the environment variable ``SNAKEMAKE_PROFILE``,
e.g. by specifying ``export SNAKEMAKE_PROFILE=myprofile`` in your ``~/.bashrc`` or the system wide shell defaults means that the ``--profile`` flag can be omitted.
In order unset the profile defined by this environment variable for individual runs without specifying and alternative profile you can provide the special value ``none``, i.e. ``--profile none``.
The profile folder is expected to contain a file ``config.yaml`` that defines default values for the Snakemake command line arguments.
For example, the file
.. code-block:: yaml
cluster: qsub
jobs: 100
would setup Snakemake to always submit to the cluster via the ``qsub`` command, and never use more than 100 parallel jobs in total.
The profile can be used to set a default for each option of the Snakemake command line interface.
For this, option ``--someoption`` becomes ``someoption:`` in the profile.
The profile folder can additionally contain auxilliary files, e.g., jobscripts, or any kind of wrappers. See https://github.com/snakemake-profiles/doc for examples.
If options accept multiple arguments these must be given as YAML list in the profile.
If options expect structured arguments (like ``--set-threads RULE=VALUE`` or ``--set-resources RULE:RESOURCE=VALUE``), those can be given as strings in the expected forms, i.e.
.. code-block:: yaml
set-threads: myrule=5
set-resources: myrule:mem=500MB
or alternatively (which is preferable) as YAML maps, e.g.:
.. code-block:: yaml
set-threads:
myrule: 5
set-resources:
myrule:
mem: 500MB
Setting resources or threads via the profile is of course rather a job for the workflow profile instead of the global profile (as such settings are likely workflow specific).
Under https://github.com/snakemake-profiles/doc, you can find publicly available global profiles (e.g. for cluster systems).
Feel free to contribute your own.
Workflow specific profiles are either not shared at all, or can be distributed along with the workflow itself where it makes sense.
For example, when the workflow has its Snakefile at ``workflow/Snakefile``, the profile config should be placed at ``workflow/profiles/default/config.yaml``.
Use templating in profiles
~~~~~~~~~~~~~~~~~~~~~~~~~~
When the profile starts with
.. code-block:: yaml
__use_yte__: true
It will be treated as a `YTE template <https://yte-template-engine.github.io>`_ and parsed accordingly.
This can be handy to e.g. define values inside of the profile that are based on environment variables.
For example, admins could use this to define user-specific settings.
Another application would be the uniform redefinition of resource requirements for a larger set of rules in a workflow profile (see above).
However, it should be noted that templated profiles are harder to keep free of errors and the profile author has to make sure that they always work correctly for the user.
.. _getting_started-visualization:
-------------
Visualization
-------------
To visualize the workflow, one can use the option ``--dag``.
This creates a representation of the DAG in the graphviz dot language which has to be postprocessed by the graphviz tool ``dot``.
E.g. to visualize the DAG that would be executed, you can issue:
.. code-block:: console
$ snakemake --dag | dot | display
For saving this to a file, you can specify the desired format:
.. code-block:: console
$ snakemake --dag | dot -Tpdf > dag.pdf
To visualize the whole DAG regardless of the eventual presence of files, the ``forceall`` option can be used:
.. code-block:: console
$ snakemake --forceall --dag | dot -Tpdf > dag.pdf
Of course the visual appearance can be modified by providing further command line arguments to ``dot``.
**Note:** The DAG is printed in DOT format straight to the standard output, along with other ``print`` statements you may have in your Snakefile. Make sure to comment these other ``print`` statements so that ``dot`` can build a visual representation of your DAG.
.. _all_options:
-----------
All Options
-----------
.. argparse::
:module: snakemake
:func: get_argument_parser
:prog: snakemake
All command line options can be printed by calling ``snakemake -h``.
.. _getting_started-bash_completion:
---------------
Bash Completion
---------------
Snakemake supports bash completion for filenames, rulenames and arguments.
To enable it globally, just append
.. code-block:: bash
`snakemake --bash-completion`
including the backticks to your ``.bashrc``.
This only works if the ``snakemake`` command is in your path.