Generic access to configuration files in any formats (to be in the future)
Python Shell HTML

README.rst

python-anyconfig

[Latest Version] [Python versions] [Test status] [Coverage Status] [Code Health] [Code Quality] [Doc Status]

Introduction

python-anyconfig [1] is a MIT licensed python library provides common APIs to access to configuration files in various formats with some useful features such as contents merge, templates and support for schema validation/generation.

Satoru SATOH <ssato@redhat.com>, developed and maintain this library with the help of others (see AUTHORS.txt for more details).

[1]The name follows the pattern of 'anydbm' from the python standard library.

Features

python-anyconfig provides very simple and unified APIs to process configuration files in various formats:

  • Loading configuration files:
    • anyconfig.load() loads configuration files or file/file-like objects and return a dict-like object represents loaded configuration
    • anyconfig.loads() loads configuration from a string just like json.loads does
  • Dumping configuration files:
    • anyconfig.dump() dumps a configuration file from a dict or dict-like object represents configuration
    • anyconfig.dumps() dumps a configuration string from a dict or dict-like object represents configuration
  • Schema validation and generation of configuration files:
    • anyconfig.validate() validates configuration loaded with anyconfig.load() with JSON schema [2] (object) also loaded with anyconfig.load(). anyconfig.load() may help loading JSON schema file[s] in any formats anyconfig supports.
    • anyconfig.gen_schema() generates a minimum JSON schema object to validate given configuration file[s] later.

It enables loading configuration file[s] in various formats in the same manner, and in some cases, it is transparent to the actual format of configuration file[s] like the followings:

import anyconfig

# Config type (format) is automatically detected by filename (file
# extension) in some cases.
conf1 = anyconfig.load("/path/to/foo/conf.d/a.yml")

# Similar to the above but load from file object opened:
with open("/path/to/foo/conf.d/a.yml") as fileobj:
    conf1_1 = anyconfig.load(fileobj)

# Loaded config data is a dict-like object, for example:
#
#   conf1["a"] => 1
#   conf1["b"]["b1"] => "xyz"
#   conf1["c"]["c1"]["c13"] => [1, 2, 3]

# Or you can specify the format (config type) explicitly if automatic
# detection may not work.
conf2 = anyconfig.load("/path/to/foo/conf.d/b.conf", ac_parser="yaml")

# Likewise.
with open("/path/to/foo/conf.d/b.conf") as fileobj:
    conf2_2 = anyconfig.load(fileobj, ac_parser="yaml")

# Specify multiple config files by the list of paths. Configurations of each
# file are merged.
conf3 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"])

# Similar to the above but all or one of config file[s] is/are missing:
conf4 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"],
                       ignore_missing=True)

# Specify config files by glob path pattern:
conf5 = anyconfig.load("/etc/foo.d/*.json")

# Similar to the above, but parameters will simply be
# overwritten by declaration in later files (instead of merged):
conf6 = anyconfig.load("/etc/foo.d/*.json", ac_merge=anyconfig.MS_REPLACE)

Also, it can process configuration files which are actually jinja2-based template files:

  • Enables loading a substantial configuration rendered from half-baked configuration template files when a context is supplied
  • Enables loading a series of configuration files indirectly 'include'-d from a/some configuration file[s] with using jinja2's 'include' directive.
In [1]: import anyconfig

In [2]: open("/tmp/a.yml", 'w').write("a: {{ a|default('aaa') }}\n")

In [3]: anyconfig.load("/tmp/a.yml", ac_template=True)
Out[3]: {'a': 'aaa'}

In [4]: anyconfig.load("/tmp/a.yml", ac_template=True, ac_context=dict(a='bbb'))
Out[4]: {'a': 'bbb'}

In [5]: open("/tmp/b.yml", 'w').write("{% include 'a.yml' %}\n")  # 'include'

In [6]: anyconfig.load("/tmp/b.yml", ac_template=True, ac_context=dict(a='ccc'))
Out[6]: {'a': 'ccc'}

And python-anyconfig enables validation of configuration files in various formats by using JSON schema as follows:

# Validate a JSON config file (conf.json) with JSON schema (schema.yaml).
# If validatation suceeds, `rc` -> True, `err` -> ''.
conf1 = anyconfig.load("/path/to/conf.json")
schema1 = anyconfig.load("/path/to/schema.yaml")
(rc, err) = anyconfig.validate(conf1, schema1)  # err is empty if success, rc == 0

# Validate a config file (conf.yml) with JSON schema (schema.yml) while
# loading the config file.
conf2 = anyconfig.load("/a/b/c/conf.yml", ac_schema="/c/d/e/schema.yml")

# Validate config loaded from multiple config files with JSON schema
# (schema.json) while loading them.
conf3 = anyconfig.load("conf.d/*.yml", ac_schema="/c/d/e/schema.json")

# Generate jsonschema object from config files loaded and get string
# representation.
conf4 = anyconfig.load("conf.d/*.yml")
scm4 = anyconfig.gen_schema(conf4)
scm4_s = anyconfig.dumps(scm4, "json")

Finally, python-anyconfig provides a CLI tool called anyconfig_cli to process configuration files and:

  • Convert a/multiple configuration file[s] to a different format
  • Get configuration value in a/multiple configuration file[s]
  • Validate configuration file[s] with JSON schema
  • Generate minimum JSON schema file to validate given configuration file[s]
[2]http://json-schema.org

Supported configuration formats

python-anyconfig supports various (configuration) file formats if the required module is available and the corresponding backend is ready to use:

Supported formats
Format Type Requirement Notes
JSON json json (standard lib) or simplejson [3] Enabled by default.
Ini-like ini configparser (standard lib) Ditto.
Java properties [4] properties None (native implementation with standard lib) Ditto.
XML xml lxml [5] or ElementTree Ditto.
YAML yaml PyYAML [6] Enabled automatically if the left requirement is satisfied.
ConifgObj configobj configobj [7] Ditto.
MessagePack msgpack msgpack-python [8] Ditto.
TOML toml toml [9] Ditto.
BSON bson bson in pymongo [10] Ditto.
B-sh shellvars None (native implementation with standard lib) Ditto.

The supported formats of python-anyconfig on your system can be listed by 'anyconfig_cli -L' like this:

$ anyconfig_cli -L
Supported config types: bson, configobj, ini, json, msgpack, toml, xml, yaml
$

or with the API 'anyconfig.list_types()' will show them:

In [8]: anyconfig.list_types()
Out[8]: ['bson', 'configobj', 'ini', 'json', 'msgpack', 'toml', 'xml', 'yaml']

In [9]:

It utilizes the plugin mechanism provided by setuptools [11] and other formats may be supported by corresponding pluggable backends like the following:

[3]https://pypi.python.org/pypi/simplejson
[4]ex. https://docs.oracle.com/javase/7/docs/api/java/util/Properties.html
[5]https://pypi.python.org/pypi/PyYAML
[6]https://pypi.python.org/pypi/lxml
[7]https://pypi.python.org/pypi/configobj
[8]https://pypi.python.org/pypi/msgpack-python
[9]https://pypi.python.org/pypi/toml
[10]https://pypi.python.org/pypi/pymongo
[11]http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins
[12]https://pypi.python.org/pypi/pyjavaproperties

Installation

Requirements

Many runtime dependencies are resolved dynamically and python-anyconfig just disables specific features if required dependencies are not satisfied. Therefore, only the python standard library is required to install and use python-anyconfig at its most minimal.

The following packages need to be installed along with python-anyconfig to enable the features.

Feature Requirements Notes
YAML load/dump PyYAML none
ConifgObj load/dump configobj none
MessagePack load/dump msgpack-python none
TOML load/dump toml none
BSON load/dump bson bson from pymongo package may work and bson [13] does not
Template config Jinja2 [14] none
Validation with JSON schema jsonschema [15] Not required to generate JSON schema.
[13]https://pypi.python.org/pypi/bson/
[14]https://pypi.python.org/pypi/Jinja2/
[15]https://pypi.python.org/pypi/jsonschema/

How to install

There is a couple of ways to install python-anyconfig:

  • Binary RPMs:

    If you're a Fedora or Red Hat Enterprise Linux user, you can install RPMs from the copr repository, http://copr.fedoraproject.org/coprs/ssato/python-anyconfig/.

    If you're running Fedora, you might enable this repo with DNF's copr plugin's help [16] like this:

    # dnf copr enable ssato/python-anyconfig
  • PyPI: You can install python-anyconfig from PyPI with using pip:

    $ pip install anyconfig
  • pip from git repo:

    $ pip install git+https://github.com/ssato/python-anyconfig/
  • Build RPMs from source: It's easy to build python-anyconfig with using rpm-build and mock:

    $ python setup.py srpm && mock dist/python-anyconfig-<ver_dist>.src.rpm

    or:

    $ python setup.py rpm

    and install built RPMs.

  • Build from source: Of course you can build and/or install python modules in the usual way such as 'python setup.py bdist', 'pip install git+https://github.com/ssato/python-anyconfig/' and so on.

[16]http://dnf-plugins-core.readthedocs.org/en/latest/copr.html

Help and feedback

If you have any issues / feature request / bug reports with python-anyconfig, please open an issue ticket on github.com (https://github.com/ssato/python-anyconfig/issues).

The following areas are still insufficient, I think.

  • Make python-anyconfig robust for invalid inputs
  • Make python-anyconfig scaled: some functions are limited by max recursion depth.
  • Documentation:
    • Especially API docs need more fixes and enhancements! CLI doc is non-fulfilling also.
    • English is not my native lang and there are many wrong and hard-to-understand expressions.

Any feedbacks, helps, suggestions are welcome! Please open github issues for these kind of problems also!