/
examples.txt
164 lines (125 loc) · 6.44 KB
/
examples.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
Example Usage
=============
.. currentmodule:: mush
To show how Mush works from a more practical point of view, let's
start by looking at a simple script that covers several common
patterns:
.. literalinclude:: ../mush/tests/example_without_mush.py
As you can see, the script above takes some command line arguments,
loads some configuration from a file, sets up log handling and then
loads a file into a database. While simple and effective, this script
is hard to test. Even using the :class:`~testfixtures.TempDirectory`
and :class:`~testfixtures.Replacer` helpers from the `TestFixtures`__
package, the only way to do so is to write one or more high level
tests such as the following:
__ http://pythonhosted.org/testfixtures
.. literalinclude:: ../mush/tests/test_example_without_mush.py
:lines: 3-31
The problem is that, in order to test the different paths through the
small piece of logic at the end of the script, we have to work around
all the set up and handling done by the rest of the script.
This also makes it hard to re-use parts of the script. It's common for
a project to have several scripts, all of which get some config from
the same file, have the same logging options and often use the same
database connection.
Encapsulating the re-usable parts of scripts
--------------------------------------------
So, let's start by looking at how these common sections of code can be
extracted into re-usable functions that can be assembled by Mush into
scripts:
.. literalinclude:: ../mush/tests/example_with_mush_clone.py
:lines: 1-45
We start with a function that adds the options needed by all our
scripts to an :mod:`argparse` parser. This uses the :class:`requires`
decorator to tell Mush that it must be called with an
:class:`~argparse.ArgumentParser` instance. See
:ref:`configuring-resources` for more details.
Next, we have a function that calls
:meth:`~argparse.ArgumentParser.parse_args` on the parser and returns
the resulting :class:`~argparse.Namespace`.
The :func:`parse_config` function reads configuration from a file specified
as a command line argument and so requires the :class:`~argparse.Namespace`.
Since the config is a :class:`dict`, we configure it as a named rather than
typed resource. See :ref:`named-resources` for more details.
Finally, there is a function that configures log handling and a
context manager that provides a database connection and handles
exceptions that occur by logging them and aborting the
transaction. Context managers like this are handled by Mush in a
specific way, see :ref:`context-managers` for more details.
Each of these components can be separately and thoroughly tested. The
Mush decorations are inert to all but the Mush :class:`Runner`,
meaning that they can be used independently of Mush in whatever way is
convenient. As an example, the following tests use a
:class:`~testfixtures.TempDirectory` and a
:class:`~testfixtures.LogCapture` to fully test the database-handling
context manager:
.. literalinclude:: ../mush/tests/test_example_with_mush_clone.py
:lines: 60-96
Writing the specific parts of your script
-----------------------------------------
Now that all the re-usable parts of the script have been abstracted,
writing a specific script becomes a case of writing just two
functions:
.. literalinclude:: ../mush/tests/example_with_mush_clone.py
:lines: 54-63
As you can imagine, this much smaller set of code is simpler to test
and easier to maintain.
Assembling the components into a script
---------------------------------------
So, we now have a library of re-usable components and the specific
callables we require for the current script. All we need to do now is
assemble these parts into the final script. The full details of this
are covered in :ref:`constructing-runners` but two common patterns are
covered below.
Cloning
~~~~~~~
With this pattern, a "base runner" is created, usually in the same
place that other re-usable parts of the original script are located:
.. literalinclude:: ../mush/tests/example_with_mush_clone.py
:lines: 3,46-53
The code above shows how to label a point, `args` in this case, enabling
callables to be inserted at that point at a later time. See :ref:`labels`
for full details.
It also shows some different ways of getting Mush to pass parts
of an object returned from a previous callable to the parameters of
another callable. See :ref:`resource-parts` for full details.
Now, for each specific script, the base runner is cloned and the
script-specific parts added to the clone leaving a callable that can
be put in the usual block at the bottom of the script:
.. literalinclude:: ../mush/tests/example_with_mush_clone.py
:lines: 67-
Using a factory
~~~~~~~~~~~~~~~
This pattern is most useful when you have lots of scripts that
all follow a similar pattern when it comes to assembling the runner
from the common parts and the specific parts. For example, if all
your scripts take a path to a config file and a path to a file that
needs processing, you can write a factory function that returns a
runner based on the callable that does the work as follows:
.. literalinclude:: ../mush/tests/example_with_mush_factory.py
:lines: 10-33
With this in place, the specific script becomes the :func:`do`
function we abstracted above and a very short call to the factory:
.. literalinclude:: ../mush/tests/example_with_mush_factory.py
:lines: 35
A combination of the clone and factory patterns can also be used to
get the best of both worlds. Setting up several base runners that are
clones of a parent runner and having factories that take common
callable patterns and return complete runners can be very powerful.
Testing
-------
The examples above have shown how using Mush can make it easier to
have smaller components that are easier to re-use and test, however
care should still be taken with testing. In particular, it's a good
idea to have some intergration tests that excercise the whole runner
checking that it behaves as expected when all command line options are
specified and when just the defaults are used.
When using the factory pattern, the factories themselves should be
unit tested. It can also make tests easier to write by having a
"testing runner" that sets up the required resources, such as database
connections, while maybe doing some things differently such as not
reading a configuration file from disk or using a
:class:`~testfixtures.LogCapture` instead of file or stream log
handlers.
Some specific tools that Mush provides to aid automated testing are covered in
:ref:`testing`.