Skip to content
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
465 lines (352 sloc) 18.1 KB

Extending PyScaffold

PyScaffold is carefully designed to cover the essentials of authoring and distributing Python packages. Most of time, tweaking putup options is enough to ensure proper configuration of a project. However, for advanced use cases a deeper level of programmability may be required and PyScaffold's extension systems provides this.

PyScaffold can be extended at runtime by other Python packages. Therefore it is possible to change the behaviour of the putup command line tool without changing the PyScaffold code itself. In order to explain how this mechanism works, the following sections define a few important concepts and present a comprehensive guide about how to create custom extensions.

Additionally, :ref:`Cookiecutter templates <cookiecutter-integration>` can also be used but writing a native PyScaffold extension is the preferred way.


A perfect start for your own custom extension is the extension custom_extension for PyScaffold. Just install it with pip install pyscaffoldext-custom-extension and then create your own extension template with putup --custom-extension pyscaffoldext-my-own-extension.

Project Structure Representation

Each Python package project is internally represented by PyScaffold as a tree data structure, that directly relates to a directory entry in the file system. This tree is implemented as a simple (and possibly nested) :obj:`dict` in which keys indicate the path where files will be generated, while values indicate their content. For instance, the following dict:

    'project': {
        'folder': {
            'file.txt': 'Hello World!',
            'another-folder': {
              'empty-file.txt': ''

represents a project/folder directory in the file system containing two entries. The first entry is a file named file.txt with content Hello World! while the second entry is a sub-directory named another-folder. In turn, another-folder contains an empty file named empty-file.txt.

Additionally, tuple values are also allowed in order to specify some useful metadata. In this case, the first element of the tuple is the file content. For example, the dict:

    'project': {
        'namespace': {
            '': ('print("Hello World!")', helpers.NO_OVERWRITE)

represents a project/namespace/ file with content print("Hello World!"), that will not be overwritten if already exists.


The NO_OVERWRITE flag is defined in the module :mod:`pyscaffold.api.helpers`

This tree representation is often referred in this document as project structure or simply structure.

Scaffold Actions

PyScaffold organizes the generation of a project into a series of steps with well defined purposes. Each step is called action and is implemented as a simple function that receives two arguments: a project structure and a dict with options (some of them parsed from command line arguments, other from default values).

An action MUST return a tuple also composed by a project structure and a dict with options. The return values, thus, are usually modified versions of the input arguments. Additionally an action can also have side effects, like creating directories or adding files to version control. The following pseudo-code illustrates a basic action:

def action(project_structure, options):
    new_struct, new_opts = modify(project_structure, options)
    return new_struct, new_opts

The output of each action is used as the input of the subsequent action, and initially the structure argument is just an empty dict. Each action is uniquely identified by a string in the format <module name>:<function name>, similarly to the convention used for a setuptools entry point. For example, if an action is defined in the action function of the file that is part of the pyscaffoldext.contrib project, the action identifier is pyscaffoldext.contrib.extras:action.

By default, the sequence of actions taken by PyScaffold is:

  1. :obj:`pyscaffold.api:get_default_options <pyscaffold.api.get_default_options>`
  2. :obj:`pyscaffold.api:verify_options_consistency <pyscaffold.api.verify_options_consistency>`
  3. :obj:`pyscaffold.structure:define_structure <pyscaffold.structure.define_structure>`
  4. :obj:`pyscaffold.structure:apply_update_rules <pyscaffold.structure.apply_update_rules>`
  5. :obj:`pyscaffold.structure:create_structure <pyscaffold.structure.create_structure>`
  6. :obj:`pyscaffold.api:init_git <pyscaffold.api.init_git>`

The project structure is usually empty until define_structure. This action just loads the in-memory dict representation, that is only written to disk by the create_structure action.

Note that, this sequence varies according to the command line options. To retrieve an updated list, please use putup --list-actions or putup --dry-run.

What are Extensions?

From the standpoint of PyScaffold, an extension is just an class inheriting from :obj:`Extension <pyscaffold.api.Extension>` overriding and implementing certain methods. This methods allow inject actions at arbitrary positions in the aforementioned list. Furthermore, extensions can also remove actions.

Creating an Extension

In order to create an extension it is necessary to write a class that inherits from :obj:`Extension <pyscaffold.api.Extension>` and implements the method :obj:`activate <pyscaffold.api.Extension.activate>` that receives a list of actions, registers a custom action that will be called later and returns a modified version of the list of actions:

from ..api import Extension
from ..api import helpers

class MyExtension(Extension):
    """Help text on commandline when running putup -h"""
    def activate(self, actions):
        """Activate extension

            actions (list): list of actions to perform

            list: updated list of actions
        actions = helpers.register(actions, self.action, after='create_structure')
        actions = helpers.unregister(actions, 'init_git')
        return actions

    def action(self, struct, opts):
        """Perform some actions that modifies the structure and options

            struct (dict): project representation as (possibly) nested
            opts (dict): given options, see :obj:`create_project` for
                an extensive list.

            struct, opts: updated project representation and options
        return new_actions, new_opts


The register and unregister methods implemented in the module :mod:`pyscaffold.api.helpers` basically create modified copies of the action list by inserting/removing the specified functions, with some awareness about their execution order.

Action List Helper Methods

As implied by the previous example, the :mod:`~pyscaffold.api.helpers` module provides a series of useful functions and makes it easier to manipulate the action list, by using :obj:`~pyscaffold.api.helpers.register` and :obj:`~pyscaffold.api.helpers.unregister`.

Since the action order is relevant, the first function accepts special keyword arguments (before and after) that should be used to place the extension actions precisely among the default actions. The value of these arguments can be presented in 2 different forms:

helpers.register(actions, hook1, before='define_structure')
helpers.register(actions, hook2, after='pyscaffold.structure:create_structure')

The first form uses as a position reference the first action with a matching name, regardless of the module. Accordingly, the second form tries to find an action that matches both the given name and module. When no reference is given, :obj:`~pyscaffold.api.helpers.register` assumes as default position after='pyscaffold.structure:define_structure'. This position is special since most extensions are expected to create additional files inside the project. Therefore, it is possible to easily amend the project structure before it is materialized by create_structure.

The :obj:`~pyscaffold.api.helpers.unregister` function accepts as second argument a position reference which can similarly present the module name:

helpers.unregister(actions, 'init_git')
helpers.unregister(actions, 'pyscaffold.api:init_git')


These functions DO NOT modify the actions list, instead they return a new list with the changes applied.


For convenience, the functions :obj:`~pyscaffold.api.helpers.register` and :obj:`~pyscaffold.api.helpers.unregister` are aliased as instance methods of the :obj:`~pyscaffold.api.Extension` class.

Therefore, inside the :obj:`~pyscaffold.api.Extension.activate` method, one could simply call actions = self.register(actions, self.my_action).

Structure Helper Methods

PyScaffold also provides extra facilities to manipulate the project structure. The following functions are accessible through the :mod:`~pyscaffold.api.helpers` module:

The first function can be used to deep merge a dictionary argument with the current representation of the to-be-generated directory tree, automatically considering any metadata present in tuple values. On the other hand, the second and third functions can be used to ensure a single file is present or absent in the current representation of the project structure, automatically handling parent directories. Finally, :obj:`~pyscaffold.api.helpers.modify` can be used to change the contents of an existing file in the project structure and/or its metadata (for example adding :obj:`~pyscaffold.api.helpers.NO_OVERWRITE` or :obj:`~pyscaffold.api.helpers.NO_CREATE` flags).


Similarly to the actions list helpers, these functions also DO NOT modify the project structure. Instead they return a new structure with the changes applied.

The following example illustrates the implementation of a AwesomeFiles extension which defines the define_awesome_files action:

from pathlib import PurePath

from ..api import Extension
from ..api import helpers

# -*- coding: utf-8 -*-

__author__ = "{author}"
__copyright__ = "{author}"
__license__ = "{license}"

def awesome():
    return "Awesome!"

import pytest
from {qual_pkg}.awesome import awesome

def test_awesome():
    assert awesome() == "Awesome!"

class AwesomeFiles(Extension):
    """Adding some additional awesome files"""
    def activate(self, actions):
        return helpers.register(actions, self.define_awesome_files)

    def define_awesome_files(self, struct, opts):
        struct = helpers.merge(struct, {
            opts['project']: {
                'src': {
                    opts['package']: {
                        '': MY_AWESOME_FILE.format(**opts)
                'tests': {
                    '': (

        struct['.python-version'] = ('3.6.1', helpers.NO_OVERWRITE)

        for filename in ['awesome_file1', 'awesome_file2']:
            struct = helpers.ensure(
                PurePath(opts['project'], 'src', 'awesome', filename),
                content='AWESOME!', update_rule=helpers.NO_CREATE)
                # The second argument is the file path, represented by a
                # list of file parts or a string.
                # Alternatively in this example:
                # path = '{project}/src/awesome/{filename}'.format(
                #           filename=filename, **opts)

        # The `reject` can be used to avoid default files being generated.
        struct = helpers.reject(
            struct, '{project}/src/{package}/'.format(**opts))
            # Alternatively in this example:
            # path = [opts['project'], 'src', opts['package'], ''])

        # `modify` can be used to change contents in an existing file
        struct = helpers.modify(
            PurePath(opts['project'], 'tests', ''),
            lambda content: 'import pdb\n' + content)

        # And/or change the update behavior
        struct = helpers.modify(struct, [opts['project'], '.travis.yml'],

        # It is import to remember the return values
        return struct, opts


The project and package options should be used to provide the correct location of the files relative to the current working directory.

As shown by the previous example, the :mod:`~pyscaffold.api.helpers` module also presents constants that can be used as metadata. The NO_OVERWRITE flag avoids an existing file to be overwritten when putup is used in update mode. Similarly, NO_CREATE avoids creating a file from template in update mode, even if it does not exist.

For more sophisticated extensions which need to read and parse their own command line arguments it is necessary to override :obj:`activate <pyscaffold.api.Extension.augment_cli>` that receives an :class:`argparse.ArgumentParser` argument. This object can then be modified in order to add custom command line arguments that will later be stored in the opts dictionary. Just remember the convention that after the command line arguments parsing, the extension function should be stored under the extensions attribute (a list) of the :mod:`argparse` generated object. For reference check out the implementation of the :ref:`namespace extension <examples/namespace-extension>` as well as the pyproject extension which serves as a blueprint for new extensions.

Activating Extensions

PyScaffold extensions are not activated by default. Instead, it is necessary to add a CLI option to do it. This is possible by setting up a setuptools entry point under the pyscaffold.cli group. This entry point should point to our extension class, e.g. AwesomeFiles like defined above. If you for instance use a scaffold generated by PyScaffold to write a PyScaffold extension (we hope you do ;-), you would add the following to the options.entry_points section in setup.cfg:

pyscaffold.cli =
    awesome_files = your_package.your_module:AwesomeFiles


Some options for the putup command are already implemented as extensions and can be used as reference implementation:

.. toctree::
   :maxdepth: 2

   namespace <examples/namespace-extension>
   no-skeleton <examples/no-skeleton-extension>
   cookiecutter <examples/cookiecutter-extension>
   django <examples/django-extension>
   pre-commit <examples/pre-commit-extension>
   tox <examples/tox-extension>
   travis <examples/travis-extension>
   gitlab <examples/gitlab-ci-extension>

Conventions for Community Extensions

In order to make it easy to find PyScaffold extensions, community packages should be namespaced as in pyscaffoldext.${EXT_NAME} (where ${EXT_NAME} is the name of the extension being developed). Although this naming convention slightly differs from PEP423, it is close enough and shorter.

Similarly to sphinxcontrib-* packages, names registered in PyPI should contain a dash -, instead of a dot .. This way, third-party extension development can be easily bootstrapped with the command:

putup pyscaffoldext-${EXT_NAME} -p ${EXT_NAME} --namespace pyscaffoldext --no-skeleton

If you put your extension code in the module then the options.entry_points section in setup.cfg looks like:

pyscaffold.cli =
    awesome_files = pyscaffoldext.${EXT_NAME}.extension:AwesomeFiles

In this example, AwesomeFiles represents the name of the class that implementes the extension and awesome_files is the string used to create the flag for the putup command (--awesome-files).

Final Considerations

When writing extensions, it is important to be consistent with the default PyScaffold behavior. In particular, PyScaffold uses a pretend option to indicate when the actions should not run but instead just indicate the expected results to the user, that MUST be respected.

The pretend option is automatically observed for files registered in the project structure representation, but complex actions may require specialized coding. The :mod:`~pyscaffold.api.helpers` module provides a special :class:`logger <pyscaffold.log.ReportLogger>` object useful in these situations. Please refer to :ref:`cookiecutter-extension` for a practical example.

Other options that should be considered are the update and force flags. See :obj:`pyscaffold.api.create_project` for a list of available options.

You can’t perform that action at this time.