use.txt

How Mush works
==============

.. currentmodule:: mush

.. note:: 

  This documentation explains how Mush works using fairly abstract
  examples. If you'd prefer more "real world" examples please see the
  :doc:`examples` documentation.

.. _constructing-runners:

Constructing runners
--------------------

Mush works by assembling a number of callables into a :class:`Runner`:

.. code-block:: python

  from mush import Runner

  def func1():
      print('func1')

  def func2():
      print('func2')

  runner = Runner(func1, func2)

Once assembled, a runner can be called any number of times. Each time
it is called, it will call each of its callables in turn:

>>> runner()
func1
func2

More callables can be added to a runner:

.. code-block:: python

 def func3():
     print('func3')

 runner.add(func3)

If you want to add several callables in one go, you can use the
runner's :meth:`~Runner.extend` method:

.. code-block:: python

 def func4():
     print('func4')

 def func5():
     print('func5')

 runner.extend(func4, func5)

Now, when called, the runner will call all five functions:

>>> runner()
func1
func2
func3
func4
func5

Runners can also be added together to create a new runner:

.. code-block:: python

  runner1 = Runner(func1)
  runner2 = Runner(func2)
  runner3 = runner1 + runner2

This addition does not modify the existing runners, but does give the
result you'd expect:

>>> runner1()
func1
>>> runner2()
func2
>>> runner3()
func1
func2

This can also be done by passing runners in when creating a new runner
or calling the extend method on a runner, for example:

.. code-block:: python

  runner1 = Runner(func1)
  runner2 = Runner(func2)
  runner4_1 = Runner(runner1, runner2)
  runner4_2 = Runner()
  runner4_2.extend(runner1, runner2)

In both cases, the results are as you would expect:

>>> runner4_1()
func1
func2
>>> runner4_2()
func1
func2

Finally, runners can be cloned, providing a way to encapsulate commonly
used base runners that can then be extended for each specific use case:

.. code-block:: python

  runner5 = runner3.clone()
  runner5.add(func4)

The existing runner is not modified, while the new runner behaves as
expected:

>>> runner3()
func1
func2
>>> runner5()
func1
func2
func4

.. _configuring-resources:

Configuring Resources
---------------------
Where Mush becomes useful is when the callables in a runner either
produce or require objects of a certain type. Given the right
configuration, Mush will wire these together enabling you to write
easily testable and reusable callables that encapsulate specific
pieces of functionality. This configuration is done either
imperatively, declaratively or using a combination of the two styles
as described in the sections below.

For the examples, we'll assume we have three types of resources:

.. code-block:: python

  class Apple: 
      def __str__(self):
          return 'an apple'
      __repr__ = __str__

  class Orange: 
      def __str__(self):
          return 'an orange'
      __repr__ = __str__

  class Juice:
      def __str__(self):
          return 'a refreshing fruit beverage' 
      __repr__ = __str__

Specifying requirements
~~~~~~~~~~~~~~~~~~~~~~~

When callables take parameters, Mush can be configured to pass objects of the
correct type that have been returned from previous callables in the runner.
For example, consider the following functions:

.. code-block:: python

 def apple_tree():
      print('I made an apple')
      return Apple()

 def magician(fruit):
      print('I turned {0} into an orange'.format(fruit))
      return Orange()

 def juicer(fruit1, fruit2):
      print('I made juice out of {0} and {1}'.format(fruit1, fruit2))

The requirements are specified by passing the required type in the `requires`
parameter when adding the callable to the runner using :meth:`~Runner.add`.
If more complex requirements need to be specified, a :class:`requires` instance
can be passed which accepts both positional and keyword parameters that
specify the types required by the callable being added:

.. code-block:: python

  from mush import Runner, requires

  runner = Runner()
  runner.add(apple_tree)
  runner.add(magician, requires=Apple)
  runner.add(juicer, requires(Apple, fruit2=Orange))

Calling this runner will now manage the resources, collecting them and
passing them in as configured:

>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange

.. _optional-resources:

Optional requirements
~~~~~~~~~~~~~~~~~~~~~

It may be that, while a callable needs an object of a particular type, a default
can be used if no such object is present. Runners can be configured to
take this into account. Take the following function:

.. code-block:: python

  def greet(name='stranger'):
      print('Hello ' + name + '!')

If a name is not always be available, it can be added to a runner as follows:

.. code-block:: python

  from mush import Runner, optional

  runner = Runner()
  runner.add(greet, requires=optional(str))

Now, when this runner is called, the default will be used:

>>> runner()
Hello stranger!

The same callable can be added to a runner where the required strings is
available:

.. code-block:: python

  from mush import Runner, optional

  def my_name_is():
     return 'Slim Shady'

  runner = Runner(my_name_is)
  runner.add(greet, requires=optional(str))

In this case, the string returned will be used:

>>> runner()
Hello Slim Shady!

.. _resource-parts:

Using parts of a resource
~~~~~~~~~~~~~~~~~~~~~~~~~

Resources can have attributes or items that are directly required by callables.
For example, consider these two functions that return such resources:

.. code-block:: python

    class Stuff(object):
        fruit = 'apple'
        tree = dict(fruit='pear')

    def some_attributes():
        return Stuff()

    def some_items():
        return dict(fruit='orange')

Also consider this function:

.. code-block:: python

    def pick(fruit1, fruit2, fruit3):
        print('I picked {0}, {1} and {2}'.format(fruit1, fruit2, fruit3))

All three can be added to a runner such that mush will pass the correct parts
of the returns resources through to the :func:`pick` function:

.. code-block:: python

    from mush import Runner, attr, item, requires

    runner = Runner(some_attributes, some_items)
    runner.add(pick, requires(fruit1=attr(Stuff, 'fruit'),
                              fruit2=item(dict, 'fruit'),
                              fruit3=item(attr(Stuff, 'tree'), 'fruit')))

So now we can pick fruit from some interesting places:

>>> runner()
I picked apple, orange and pear

The :func:`pick` function, however, remains usable and testable on its own:

>>> pick('apple', 'orange', 'pear')
I picked apple, orange and pear

Specifying returned resources
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As seen above, Mush will track resources returned by callables based on the type
of any object returned. This is usually what you want, but in some cases you
may want to specify something different.
For example, if you have a callable that returns a sequence of resources, this
would be added to a runner as follows:

.. code-block:: python

  from mush import Runner, returns_sequence

  def all_fruit():
      print('I made fruit')
      return Apple(), Orange()


  runner = Runner()
  runner.add(all_fruit, returns=returns_sequence())
  runner.add(juicer, requires(Apple, Orange))

Now, the juicer will use all the fruit returned:

>>> runner()
I made fruit
I made juice out of an apple and an orange

In rarer circumstances, you may need to override the types returned by a
callable. This can be done as follows:

.. code-block:: python

  from mush import Runner, returns

  class Tomato:
      def __str__(self):
          return 'a tomato'

  class Cucumber:
      def __str__(self):
          return 'a cucumber'

  def vegetables():
      print('I made vegetables')
      return Tomato(), Cucumber()

  runner = Runner()
  runner.add(vegetables, returns=returns(Apple, Orange))
  runner.add(juicer, requires(Apple, Orange))

Now, even when a callable requires fruit, we can force it to be happy with
vegetables:

>>> runner()
I made vegetables
I made juice out of a tomato and a cucumber

The :class:`returns` indicator can be used even if a single object is returned.

Another way that the type used to track the resource can be different from the
type of the resource itself is if a callable returns a mapping and Mush is
configured to use the types from that mapping:

.. code-block:: python

  from mush import Runner, returns_mapping

  def desperation():
      print('I sold vegetables as fruit')
      return {Apple: Tomato(), Orange: Cucumber()}

  runner = Runner()
  runner.add(desperation, returns=returns_mapping())
  runner.add(juicer, requires(Apple, Orange))

Once again, we can happily make juice out of vegetables:

>>> runner()
I sold vegetables as fruit
I made juice out of a tomato and a cucumber

Finally, if you have a callable that returns results that you wish to ignore,
you can do so using :attr:`~mush.declarations.nothing`:

.. code-block:: python

  from mush import Runner, nothing

  def spam():
      return 'spam'

  runner = Runner()
  runner.add(spam, returns=nothing)

.. _named-resources:

Named resources
~~~~~~~~~~~~~~~

Sometimes the types of resources are too common for them to uniquely identify
a resource:

.. code-block:: python

  def age():
      return 37

  def meaning():
      return 42

A callable such as the following cannot be configured to require the correct
resource from these two functions by type alone:

.. code-block:: python

  def profound(age, it):
      print('by the age of %s I realised the meaning of life was %s' % (
          age, it
      ))

For these situations, Mush supports the ability to name a resource using a
string:

.. code-block:: python

  runner = Runner()
  runner.add(age, returns='age')
  runner.add(meaning, returns='meaning')
  runner.add(profound, requires('age', it='meaning'))

Anywhere that a type can be used, a string name can be used instead:

>>> runner()
by the age of 37 I realised the meaning of life was 42

.. _type-annotations:

Using Type Annotations
~~~~~~~~~~~~~~~~~~~~~~

While the imperative configuration used so far means that callables do not
need to be modified, Python's type annotations can also be used to specify
requirements and returned resources:

.. code-block:: python

  from mush import requires

  def apple_tree():
      print('I made an apple')
      return Apple()

  def magician(fruit: Apple) -> 'citrus':
      print('I turned {0} into an orange'.format(fruit))
      return Orange()

  def juicer(fruit1: Apple, fruit2: 'citrus'):
      print('I made juice out of {0} and {1}'.format(fruit1, fruit2))
      return Juice()

These can now be combined into a runner and executed. The runner will
extract the requirements from the type annotations and will use them to
map the parameters as appropriate:

>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
a refreshing fruit beverage

Declarative configuration
~~~~~~~~~~~~~~~~~~~~~~~~~

When type annotations are not available, either because they're being used
for something else or because of the version of Python being used, the helpers
for specifying requirements and return types can also be used as decorators:

.. code-block:: python

  from mush import requires

  def apple_tree():
      print('I made an apple')
      return Apple()

  @requires(Apple)
  @returns('citrus')
  def magician(fruit):
      print('I turned {0} into an orange'.format(fruit))
      return Orange()

  @requires(fruit1=Apple, fruit2='citrus')
  def juicer(fruit1, fruit2):
      print('I made juice out of {0} and {1}'.format(fruit1, fruit2))
      return Juice()

These can now be combined into a runner and executed. The runner will
extract the requirements stored by the decorator and will use them to
map the parameters as appropriate:

>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
a refreshing fruit beverage

.. _default-configuration:

Default configuration
~~~~~~~~~~~~~~~~~~~~~

If no declarations are made using either decorators or type annotations,
then arguments that are needed by a callable will be looked up based on the
name of the argument:

.. code-block:: python

  from mush import requires

  def apple_tree() -> 'apple':
      print('I made an apple')
      return Apple()

  def magician(apple) -> 'citrus':
      print('I turned {0} into an orange'.format(apple))
      return Orange()

  def juicer(apple, citrus):
      print('I made juice out of {0} and {1}'.format(apple, citrus))
      return Juice()

These can now be combined into a runner and executed. The runner will
guess the requirements base on the names of the arguments for each function
and will use them to map the parameters as appropriate:

>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
a refreshing fruit beverage

If an argument has a default, then the requirement will be made
:ref:`optional <optional-resources>`.

Configuration Precedence
~~~~~~~~~~~~~~~~~~~~~~~~

The four styles of configuration are entirely interchangeable and you can
use any combination that suites your requirements.

In terms of precedence, requirements and returned resource specifications
will be used in the following order, with the first one found being the one
that is used:

- Imperative configuration.
- Declarative configuration.
- Type annotations.
- Default configuration.

The default configuration for requirements is described
:ref:`above <default-configuration>`.

The default configuration for return values is that a callable’s
return value is used as a resource and will be registered
against the type of the object returned. The
:class:`~mush.returns_result_type` declaration encapsulates this behaviour.

.. _labels:

Labels
------

One of the motivating reasons for Mush to be created was the ability to insert
callables at a point in a runner other than the end. This allows abstraction
of common sequences of calls without the risks of extracting them into a base
class.

The points at which more callables can be inserted are created by specifying a
label when adding a callable to the runner. This marks the point at which that
callable is included so that it can be retrieved and appended to later. As an
example, consider a ring and some things that can be done to it:

.. code-block:: python

  class Ring:
      def __str__(self):
          return 'a ring'

  def forge():
      return Ring()

  def engrave(ring):
      print('engraving {0}'.format(ring))

These might be added to a runner as follows:

.. code-block:: python

  from mush import Runner

  runner = Runner()
  runner.add(forge)
  runner.add_label('forged')
  runner.add(engrave, requires=Ring)
  runner.add_label('engraved')

Now, suppose we want to polish the ring before it's engraved and then
package it up when we're done:

.. code-block:: python

  def polish(ring):
      print('polishing {0}'.format(ring))

  def package(ring):
      print('packaging {0}'.format(ring))

We can insert these callables into the runner at the right points as follows:

.. code-block:: python

  runner['forged'].add(polish, requires=Ring)
  runner.add(package, requires=Ring)

This results in the desired call order:

>>> runner()
polishing a ring
engraving a ring
packaging a ring

Now, suppose we want to polish the ring again after it's been engraved. We can
insert another callable at the appropriate point:

.. code-block:: python

  def more_polish(ring):
      print('polishing {0} again'.format(ring))

  runner['engraved'].add(more_polish, requires=Ring)

Mush will do the right thing when this runner is called:

>>> runner()
polishing a ring
engraving a ring
polishing a ring again
packaging a ring

When using labels, it's often good to be able to see exactly what is in a
runner, what order it is in and where any labels point. For this reason,
the representation of a runner gives all this information:

>>> runner
<Runner>
    <function forge ...> requires() returns_result_type()
    <function polish ...> requires(Ring) returns_result_type() <-- forged
    <function engrave ...> requires(Ring) returns_result_type()
    <function more_polish ...> requires(Ring) returns_result_type() <-- engraved
    <function package at ...> requires(Ring) returns_result_type()
</Runner>

As you can see above, when a callable is inserted at a label, the label
moves to that callable. You may wish to keep track of the initial point that
was labelled, so Mush supports multiple labels at each point:

>>> runner = Runner()
>>> point = runner.add(forge)
>>> point.add_label('before_polish')
>>> point.add_label('after_polish')
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- after_polish, before_polish
</Runner>

Now, when you add to a specific label, only that label is moved:

>>> point = runner['after_polish']
>>> point.add(polish)
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- before_polish
    <function polish ...> requires('ring') returns_result_type() <-- after_polish
</Runner>

Of course, you can still add to the end of the runner:

>>> runner.add(package)
<mush.modifier.Modifier...>
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- before_polish
    <function polish ...> requires('ring') returns_result_type() <-- after_polish
    <function package ...> requires('ring') returns_result_type()
</Runner>

However, the point modifier returned by getting a label from a runner will
keep on moving the label as more callables are added using it:

>>> point.add(more_polish)
>>> runner
<Runner>
    <function forge ...> requires() returns_result_type() <-- before_polish
    <function polish ...> requires('ring') returns_result_type()
    <function more_polish ...> requires('ring') returns_result_type() <-- after_polish
    <function package ...> requires('ring') returns_result_type()
</Runner>

.. _plugs:

Plugs
-----

You may run into situations where you wish to group callables together and
add them in one go. Indeed, it might only make sense to add all callables in
a group and would cause problems to add them individually.

For example, suppose we have this runner:

.. code-block:: python

  def prepare():
      print('cleaning kitchen table')

  def wash(produce):
      print('washing '+str(produce))

  def finished():
      print('service please!')

  kitchen_runner = Runner()
  kitchen_runner.add(prepare)
  kitchen_runner.add_label('what')
  kitchen_runner.add(wash, requires='produce')
  kitchen_runner.add_label('how')
  kitchen_runner.add(finished)

For any use of the kitchen, we need to specify both what to use and how we want
to use it once it's washed. This can neatly be done as follows:

.. code-block:: python

  from mush import Plug

  class JuicePlug(Plug):

      def what(self) -> 'produce':
          return Apple()

      def how(self, produce):
          print('juicing '+str(produce))

  runner = kitchen_runner.clone()
  JuicePlug().add_to(runner)

The runner behaves as we require:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!

It may be that we want our plug to have helper methods, in which case they can
either be named with a leading underscore, or the plug can be set up to only
add explicitly marked methods, for example:

.. code-block:: python

  from mush.plug import Plug, insert

  class JuicePlug(Plug):

      explicit = True

      def juice(self, produce):
          print('juicing '+str(produce))

      @insert()
      def what(self) -> 'produce':
          return Apple()

      @insert()
      def how(self, produce: 'produce'):
          self.juice(produce)

  runner = kitchen_runner.clone()
  JuicePlug().add_to(runner)

The runner behaves as before:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!

As you can see, this might make for a lot of decorating if you only have one
helper method. If that's the case, you can just tell the plug to ignore the
helper:

.. code-block:: python

  from mush.plug import Plug, ignore

  class JuicePlug(Plug):

      @ignore()
      def juice(self, produce):
          print('juicing '+str(produce))

      def what(self) -> 'produce':
          return Apple()

      def how(self, produce: 'produce'):
          self.juice(produce)

  runner = kitchen_runner.clone()
  JuicePlug().add_to(runner)

The runner still behaves as before:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!

It may be that it makes sense to give your method a name different to
the label you wish to add it at. You may also wish to have a plug add a method
to the end of the runner where there is no label. Both of these are supported:

.. code-block:: python

  from mush.plug import Plug, insert, append

  class JuicePlug(Plug):

      @insert(label='what')
      def pick_fruit(self) -> 'produce':
          return Apple()

      def how(self, produce: 'produce'):
          print('juicing '+str(produce))

      @append()
      def relax(self):
          print('...and relax')

  runner = kitchen_runner.clone()
  JuicePlug().add_to(runner)

The runner now behaves as required:

>>> runner()
cleaning kitchen table
washing an apple
juicing an apple
service please!
...and relax

.. _context-managers:

Context manager resources
-------------------------

A frequent requirement when writing scripts is to make sure that
when unexpected things happen they are logged, transactions are
aborted, and other necessary cleanup is done. Mush supports this
pattern by allowing context managers to be added as callables:

.. code-block:: python

    from mush import Runner, requires

    class Transactions(object):

        def __enter__(self):
            print('starting transaction')

        def __exit__(self, type, obj, tb):
            if type:
                print(obj)
                print('aborting transaction')
            else:
                print('committing transaction')
            return True

    def a_func():
        print('doing my thing')

    def good_func():
        print('I have done my thing')

    def bad_func():
        raise Exception("I don't want to do my thing")

The context manager is wrapped around all callables that are called
after it:

>>> runner = Runner(Transactions, a_func, good_func)
>>> runner()
starting transaction
doing my thing
I have done my thing
committing transaction

This gives it a chance to clear up when things go wrong:

>>> runner = Runner(Transactions, a_func, bad_func)
>>> runner()
starting transaction
doing my thing
I don't want to do my thing
aborting transaction

.. _testing:

Testing
-------

Mush has a couple of features to help with automated testing of runners.
For example, if you wanted to test a runner that got configuration by calling
a remote web service:

.. code-block:: python

  def load_config() -> 'config':
      return json.loads(urllib2.urlopen('...').read())

  def do_stuff(username: item('config', 'username'),
               password: item('config', 'password')):
      print('doing stuff as ' + username + ' with '+ password)

  runner = Runner(load_config, do_stuff)

When testing this runner, we may want to inject a hard-coded config. This can
be done by cloning the original runner and replacing the :func:`load_config`
callable:

>>> def test_config():
...     return dict(username='test', password='pw')
>>> test_runner = runner.clone()
>>> test_runner.replace(load_config, test_config)
>>> test_runner()
doing stuff as test with pw

If you have a base runner such as this:

.. code-block:: python

  from argparse import ArgumentParser, Namespace

  def base_args(parser):
      parser.add_argument('config_url')

  def parse_args(parser):
      return parser.parse_args()

  def load_config():
      return json.loads(urllib2.urlopen('...').read())

  def finalise_things():
      print('all done')

  base_runner = Runner(ArgumentParser)
  base_runner.add(base_args, requires=ArgumentParser, label='args')
  base_runner.add(parse_args, requires=ArgumentParser)
  point = base_runner.add(load_config, requires=attr(Namespace, 'config_url'),
                          returns='config')
  point.add_label('body')
  base_runner.add(finalise_things, label='ending')

That runner might be used for a specific script as follows:

.. code-block:: python

  def job_args(parser: ArgumentParser):
      parser.add_argument('--colour')

  def do_stuff(username: item('config', 'username'),
               colour: attr(Namespace, 'colour')):
      print(username + ' is '+ colour)

  runner = base_runner.clone()
  runner['args'].add(job_args)
  runner['body'].add(do_stuff)

To test this runner, we want to use a dummy configuration and command line
and not have any finalisation take place. This can be achieved with a helper
function such as the following:

.. code-block:: python

  def run_with(source_runner, config, argv):
      runner = Runner(ArgumentParser)
      runner.extend(source_runner.clone(added_using='args'))
      runner.add(lambda parser: parser.parse_args(argv),
                 requires=ArgumentParser)
      runner.add(lambda: config, returns='config')
      runner.extend(source_runner.clone(added_using='body'))
      runner()

The helper can then be used as follows:

>>> run_with(runner,
...          config=dict(username='test', password='pw'),
...          argv=['--colour', 'red'])
test is red

Debugging
---------

Mush has a couple of features to aid debugging of runners. The first of which
is that the representation of a runner will show everything in it, in the order
it will be called and what each callable has been declared as requiring and
returning along with where any labels currently point.

For example, consider this runner:

.. code-block:: python

  from mush import Runner

  def make_config() -> 'config':
      return {'foo': 'bar'}

  def connect(foo: item('config', 'foo')):
      return 'connection'

  def process(connection):
      print('using ' + repr(connection))

  runner = Runner()
  point = runner.add(make_config, label='config')
  point.add(connect)
  runner.add(process)

To see how the configuration panned out, we would look at the :func:`repr`:

>>> runner
<Runner>
    <function make_config ...> requires() returns('config')
    <function connect ...> requires(foo='config'['foo']) returns_result_type() <-- config
    <function process ...> requires('connection') returns_result_type()
</Runner>

As you can see, there is a problem with this configuration that will be exposed
when it is run. To help make sense of these kinds of problems, Mush will add
more context when a :class:`TypeError` or :class:`~.context.ContextError`
is raised.