Skip to content

Commit

Permalink
Switch to validating without voluptuous
Browse files Browse the repository at this point in the history
  • Loading branch information
@abadger
abadger committed Nov 13, 2017
1 parent 5640a12 commit 067f96d
Show file tree
Hide file tree
Showing 7 changed files with 942 additions and 312 deletions.
302 changes: 270 additions & 32 deletions doc/configuration.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,54 +14,92 @@ To quickly configure output, use the `quick_setup` function. Quick setup is lim
.. autofunction:: quick_setup
:noindex:

.. _twiggy-setup:
.. _dict_config:

*******************
twiggy_setup.py
dict_config()
*******************
.. testsetup:: twiggy-setup
.. testsetup:: dict_config

from twiggy import emitters
emitters.clear()

Twiggy's output side features modern, loosely coupled design.

By convention, your configuration lives in a file in your application called ``twiggy_setup.py``, in a function called ``twiggy_setup()``. You can of course put your configuration elsewhere, but using a separate module makes integration with configuration management systems easy. You should import and run ``twiggy_setup`` near the top of your application. It's particularly important to set up Twiggy *before spawning new processes*.

A ``twiggy_setup`` function should create ouputs and use the :func:`add_emitters` convenience function to link those outputs to the :data:`log`.

.. testcode:: twiggy-setup

from twiggy import add_emitters, outputs, levels, filters, formats, emitters # import * is also ok
def twiggy_setup():
alice_output = outputs.FileOutput("alice.log", format=formats.line_format)
bob_output = outputs.FileOutput("bob.log", format=formats.line_format)
Twiggy's output side features modern, loosely coupled design. It allows specifying :ref:`outputs`
(where the log messages should go), :ref:`formats` (how the messages should look), and
:term:`emitter` (which logs should go to a particular output) as separate pieces.

Twiggy lets you specify configuration as a dictionary with the configuration information and then
pass the dictionary to :func:`twiggy.dict_config` to set things up. The dictionary can be
constructed programmatically, loaded from a configuration file, or hardcoded into an application.
This allows the programmer to easily set defaults and allow the user to override those from
a configuration file. Here's an example:

.. testcode:: dict_config

twiggy_config = {'version': '1.0',
'outputs': {
'alice_output': {
'output': 'twiggy.outputs.FileOutput',
'args': ['alice.log']
},
'bob_output': {
'output': 'twiggy.outputs.FileOutput',
'args': ['bob.log']
format: 'twiggy.formats.line_format'
}
},
'emitters': {
'alice': {
'level': 'DEBUG',
'output_name': 'alice_output'
},
'betty': {
'level': 'INFO',
'filters': [ {
'filter': 'twiggy.filters.names',
'args': ['betty']
}
],
'output_name': 'bob_output'
},
'brian.*': {
'level': 'DEBUG',
'filters': [ {
'filter': 'twiggy.filters.glob_names',
'args': ['brian.*']
}
],
'output_name': 'bob_output'
}
}
}

twiggy.dict_config(twiggy_config)

In this example, the programmer creates a twiggy configuration in the application's code and uses it
to configure twiggy. The configuration should be done near the start of your application. It's
particularly important to set up Twiggy *before spawning new processes*.

With this configuration, :func:`twiggy.dict_config` will create two log destinations (:term:`outputs`):
:file:`alice.log` and :file:`bob.log`. These :term:`outputs` are then associated with the set of
messages that they will receive in the ``emitters`` section. :file:`alice.log` will receive all
messages and :file:`bob.log` will receive two sets of messages:

add_emitters(
# (name, min_level, filter, output),
("alice", levels.DEBUG, None, alice_output),
("betty", levels.INFO, filters.names("betty"), bob_output),
("brian.*", levels.DEBUG, filters.glob_names("brian.*"), bob_output),
)
* messages with the name field equal to ``betty`` and level >= ``INFO``
* messages with the name field glob-matching ``brian.*``

# near the top of your __main__
twiggy_setup()
The :term:`emitters` are stored inside of twiggy in the :data:`twiggy.emitters` dictionary:

:func:`add_emitters` populates the :data:`emitters` dictionary:

.. doctest:: twiggy-setup
.. doctest:: dict_config

>>> from twiggy import emitters
>>> sorted(emitters.keys())
['alice', 'betty', 'brian.*']

In this example, we create two log destinations: ``alice.log`` and ``bob.log``. alice will receive all messages, and bob will receive two sets of messages:

* messages with the name field equal to ``betty`` and level >= ``INFO``
* messages with the name field glob-matching ``brian.*``

:class:`Emitters <.Emitter>` can be removed by deleting them from this dict. :attr:`~.Emitter.filter` and :attr:`~.Emitter.min_level` may be modified during the running of the application, but outputs *cannot* be changed. Instead, remove the emitter and re-add it.

.. doctest:: twiggy-setup
.. doctest:: dict_config

>>> # bump level
... emitters['alice'].min_level = levels.WARNING
Expand All @@ -70,7 +108,146 @@ In this example, we create two log destinations: ``alice.log`` and ``bob.log``.
>>> # remove entirely
... del emitters['alice']

We'll examine the various parts in more detail.

See the :ref:`Twiggy Config Schema` documentation for details of what each of the fields in the
configuration dictionary mean.


User Overrides
==============

Each site that runs an application is likely to have different views on where they want the
application to log to. With Twiggy's `dict_config` it is easy to let the user override the
configuration specified by the program. For instance, the application could have a yaml
configuration file with a ``logging_config`` section which contains a Twiggy Config. Allowing the
site to override the default with the configuration from the config file is as simple as::

import yaml
config = yaml.safe_load('config_file.yml')
if 'logging_config' in config:
try:
twiggy.dict_config(config['logging_config'])
except Exception as e:
print('User provided logging configuration was flawed: {0}'.format(e))


Twiggy Config Schema
====================

The dict taken by :func:`twiggy.dict_config` may contain the following keys:

version
Set to the value representing the schema version as a string. Currently, the only valid value
is "1.0".

incremental
(*Optional*) If True, the dictionary will update any existing configuration. If False, this
will override any existing configuration. This allows user defined logging configuration to
decide whether to override the logging configuration set be the application or merely supplement
it. The default is False.

outputs
(*Optional*) Mapping of output names to outputs. Outputs consist of

output
A :class:`twiggy.outputs.Output` or the string representation with which to import
a :class:`~twiggy.outputs.Output`. For instance, to use the builtin,
:class:`twiggy.outputs.FileOutput` either set output directly to the class or the string
``twiggy.outputs.FileOutput``.

args
(*Optional*) A list of arguments to pass to the :class:`Twiggy.outputs.Output` class
constructor. For instance, :class:`~twiggy.outputs.FileOutput` takes the filename of a file
to log to. So ``args`` could be set to: ``["logfile.log"]``.

kwargs
(*Optional*) A dict of keyword arguments to pass to the :class:`Twiggy.outputs.Output` class
constructor. For instance, :class:`~twiggy.outputs.StreamOutput` takes a stream as
a keyword argument so ``kwargs`` could be set to: ``{"stream": "ext://sys.stdout"}``.

format
(*Optional*) A formatter function which transforms the log message for the output. The
default is :func:`twiggy.formats.line_format`

If both ``outputs`` and ``emitters`` are None and `incremental` is False then
:data:`twiggy.emitters` will be cleared.

emitters
(*Optional*) Mapping of emitter names to emitters. Emitters consist of:

level
String name of the log level at which log messages will be passed to this emitter.
May be one of (In order of severity) ``CRITICAL``, ``ERROR``, ``WARNING``, ``NOTICE``,
``INFO``, ``DEBUG``, ``DISABLED``.

output_name
The name of an output in this configuration dict.

filters
(*Optional*) A list of filters which filter out messages which will go to this emitter.
Each filter is a mapping which consists of:

filter
String name for a twiggy filter function.

args
(*Optional*) A list of arguments to pass to the :class:`Twiggy.outputs.Output` class
constructor. For instance, :class:`~twiggy.outputs.FileOutput` takes the filename of a file
to log to. So ``args`` could be set to: ``["logfile.log"]``.

kwargs
(*Optional*) A dict of keyword arguments to pass to the :class:`Twiggy.outputs.Output` class
constructor. For instance, :class:`~twiggy.outputs.StreamOutput` takes a stream as
a keyword argument so ``kwargs`` could be set to: ``{"stream": "ext://sys.stdout"}``.

If both ``emitters`` and ``output`` are None and `incremental` is False then
:data:`twiggy.emitters` will be cleared.

Sometimes you want to have an entry in ``args`` or ``kwargs`` that is a python object. For
instance, :class:`~twiggy.outputs.StreamOutput` takes a stream keyword argument so you may want to
give ``sys.stdout`` to it. If you are writing the configuration in Python code you can simply
include the actual object in the field. However, if you are writing in a text configuration file,
you need another way to specify this. Twiggy allows you to prefix the string with ``ext://`` in
this case. When Twiggy sees that the string starts with ``ext://`` it will strip off the prefix and
then try to import an object with the rest of the name.

Here's an example config that you might find in a YAML config file:

.. code-block:: yaml
version: '1.0'
outputs:
alice_output:
output: 'twiggy.outputs.FileOutput'
args:
- 'alice.log'
bob_output:
output: 'twiggy.outputs.StreamOutput'
kwargs:
stream: 'ext://sys.stdout'
format: 'twiggy.formats.line_format'
emitters:
alice:
level: 'DEBUG'
output_name: 'alice_output'
betty:
level: 'INFO'
filters:
filter: 'twiggy.filters.names'
args:
- 'betty'
output_name: 'bob_output'
brian.*:
levels: 'DEBUG'
filters:
filter: 'twiggy.filters.glob_names'
args:
-'brian.*'
output_name: 'bob_output'
.. _outputs:

**************************
Outputs
Expand All @@ -85,6 +262,8 @@ Many outputs can be configured to use a separate, dedicated process to log messa

.. warning: There is a slight, but non-zero, chance that messages may be lost if something goes awry with the child process.
.. _formats:

*********************
Formats
*********************
Expand Down Expand Up @@ -133,3 +312,62 @@ The messages output by an emitter are determined by its :attr:`~.Emitter.min_lev
e.filter = ["^mem.y$", lambda msg: msg.fields['address'] > 0xDECAF]

.. seealso:: Available :mod:`.filters`


.. _twiggy-setup:

*******************
twiggy_setup.py
*******************
.. testsetup:: twiggy-setup

from twiggy import emitters
emitters.clear()

Twiggy's output side features modern, loosely coupled design.

By convention, your configuration lives in a file in your application called ``twiggy_setup.py``, in a function called ``twiggy_setup()``. You can of course put your configuration elsewhere, but using a separate module makes integration with configuration management systems easy. You should import and run ``twiggy_setup`` near the top of your application. It's particularly important to set up Twiggy *before spawning new processes*.

A ``twiggy_setup`` function should create outputs and use the :func:`add_emitters` convenience function to link those outputs to the :data:`log`.

.. testcode:: twiggy-setup

from twiggy import add_emitters, outputs, levels, filters, formats, emitters # import * is also ok
def twiggy_setup():
alice_output = outputs.FileOutput("alice.log", format=formats.line_format)
bob_output = outputs.FileOutput("bob.log", format=formats.line_format)

add_emitters(
# (name, min_level, filter, output),
("alice", levels.DEBUG, None, alice_output),
("betty", levels.INFO, filters.names("betty"), bob_output),
("brian.*", levels.DEBUG, filters.glob_names("brian.*"), bob_output),
)

# near the top of your __main__
twiggy_setup()

:func:`add_emitters` populates the :data:`emitters` dictionary:

.. doctest:: twiggy-setup

>>> sorted(emitters.keys())
['alice', 'betty', 'brian.*']

In this example, we create two log destinations: ``alice.log`` and ``bob.log``. alice will receive all messages, and bob will receive two sets of messages:

* messages with the name field equal to ``betty`` and level >= ``INFO``
* messages with the name field glob-matching ``brian.*``

:class:`Emitters <.Emitter>` can be removed by deleting them from this dict. :attr:`~.Emitter.filter` and :attr:`~.Emitter.min_level` may be modified during the running of the application, but outputs *cannot* be changed. Instead, remove the emitter and re-add it.

.. doctest:: twiggy-setup

>>> # bump level
... emitters['alice'].min_level = levels.WARNING
>>> # change filter
... emitters['alice'].filter = filters.names('alice', 'andy')
>>> # remove entirely
... del emitters['alice']

We'll examine the various parts in more detail.
2 changes: 1 addition & 1 deletion setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
author_email='pete@wearpants.org',
url='http://twiggy.wearpants.org',
packages=['twiggy', 'twiggy.lib', 'twiggy.features'],
install_requires=['six', 'voluptuous'],
install_requires=['six'],
license = "BSD",
classifiers = [
"Topic :: System :: Logging",
Expand Down

0 comments on commit 067f96d

Please sign in to comment.