Enhance hook implementation and create plugin docs

- Simplify calling of hooks via 'plugin_manager'. Replaces use of
  get_plugin_manager()
- Allow hooks to be disabled with `KSCONF_PLUGIN_DISABLE` (space separated
  list)
- Support type hinting for `plugin_manager.hook.<name>` using Prototype.
- Add plugin monitoring with simple callbacks, enabled with KSCONF_DEBUG=1
- Improve plugin spec docs.  Add new plugins page in the docs with some basic
  overviews and examples.
- Improve freeze definition in AppPackager.

docs/source/api/ksconf.rst

@@ -66,7 +66,7 @@ ksconf.filter module
    :show-inheritance:
 
 ksconf.hook module
--------------------
+------------------
 
 .. automodule:: ksconf.hook
    :members:

docs/source/changelog.rst

@@ -11,7 +11,7 @@ Ksconf 0.11
 **Highlights:**
 
 *  Ksconf is beginning to treat Splunk apps more holistically and not just as a collection of ``.conf`` files.
-*  Significant portions of this new code base is directly leveraged by the Ansible modules located in the `cdillc.splunk <https://github.com/Kintyre/ansible-collection-splunk>`__ collection, a sibling project to Ksconf.
+*  Significant portions of this new code base is directly leveraged by the Ansible modules located in the `cdillc.splunk`_ collection, a sibling project to Ksconf.
    some of the code code there has made it's way into the core ksconf project in this release.
 
 
@@ -20,14 +20,21 @@ Ksconf 0.11
 *  Added :py:class:`~ksconf.app.facts.AppFacts` to easily collect Splunk application name, version, label, and other nuggets from ``app.conf``.
 *  Added :py:class:`~ksconf.app.manifest.AppManifest` to inventory the contents of a Splunk application and create a unique content fingerprint that can be used to quickly identify application changes.
 *  Added :py:mod:`ksconf.app.deploy` to assist with Splunk application deployment planning and execution.
-*  Added :py:mod:`ksconf.hookspec` to define all available pluggy integration points.
+*  Added :py:class:`ksconf.hookspec.KsconfHookSpecs` to define all available `pluggy`_  integration points.
    Anyone wanting to implement a new plugin should use the public-facing :py:mod:`ksconf.hook` module.
 
 
+Ksconf v0.11.7 (DRAFT)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*  Support disabling of plugins by name via ``KSCONF_PLUGIN_DISABLE`` environment variable.  This expects a space separated lists of plugin names.
+*  Add new plugins documentation.
+
+
 Ksconf v0.11.6 (2023-09-20)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-*  Introducing plugin functionality using `pluggy <https://pluggy.readthedocs.io/en/latest/>`__ plugin management system.
+*  Introducing plugin functionality using `pluggy`_ plugin management system.
    This adds a small, single-package dependency that can greatly increase customization potential of ksconf.
    The first demo of this can be seen in the ``ksconf-jinja-markdown`` package that enables ``.j2`` payloads to be rendered by registering a custom Jinja filter named ``markdown2html``.
 
@@ -99,7 +106,7 @@ Ksconf 0.10
 
 **API Changes**
 
-*  Core layer combining logic now lives in :py:class:`~ksconf.combine.LayerCombine`.
+*  Core layer combining logic now lives in :py:class:`~ksconf.combine.LayerCombiner`.
    The new :py:class:`~ksconf.command.combine.RepeatableCombiner` class has logic for marker safety checks and settings for removing or preserving existing files.
    The :py:class:`~ksconf.command.combine.CombineCmd` now contains only the command line functionality.
 
@@ -745,3 +752,6 @@ Release legacy-v1.0.0 (2018-04-16)
 -  Unit test coverage over 85%
 -  Includes pre-commit hook configuration (so that other repos can use this to run ``ksconf sort``
    and ``ksconf check`` against their conf files.
+
+
+..  include:: common

docs/source/common

@@ -5,15 +5,16 @@
 ..  _GitHub Releases: https://github.com/Kintyre/ksconf/releases
 ..  _argcomplete: https://argcomplete.readthedocs.io/en/latest/
 ..  _gitlint: https://jorisroovers.github.io/gitlint/
-..  _ksconf-wheel:  https://pypi.org/project/ksconf/#files
-..  _ksconf:  https://pypi.org/project/ksconf
 ..  _ksconf app for splunk: https://splunkbase.splunk.com/app/4383/
 ..  _ksconf repo: https://github.com/Kintyre/ksconf
+..  _ksconf-wheel:  https://pypi.org/project/ksconf/#files
+..  _ksconf:  https://pypi.org/project/ksconf
+..  _pluggy: https://pluggy.readthedocs.io/en/latest/
 ..  _pre-commit: https://pre-commit.com/
 ..  _pyenv: https://github.com/pyenv/pyenv
 ..  _python: https://www.python.org/downloads/
 ..  _tox: https://tox.readthedocs.io/en/latest/
 ..  _virtualenv: https://virtualenv.pypa.io/en/stable/
-
+..  _cdillc.splunk: https://github.com/Kintyre/ansible-collection-splunk
 
 .. vim:ft=rst:

docs/source/index.rst

@@ -65,6 +65,7 @@ User Guide
     install
     cmd
     cheatsheet
+    plugins
     contrib
     devel
     git

docs/source/plugins.rst

@@ -0,0 +1,171 @@
+Plugins
+-------
+
+Ksconf supports a growing number of plugins to enable custom workflow and and elegantly handle custom use cases that don't make sense to implement in the core tool.
+Plugins functionality is implemented using `pluggy`_.
+
+Note that, much like the pluggy docs themselves, we use the term "hook" and "plugin" are used interchangeably at times.
+Generally, the term "hook" is a specific handoff point where control can be passed from the ksconf codebase to some hook function that you've implemented to perform a specific operation.
+The term "plugin" refers to a package (or collection) of implemented hooks.
+
+There are multiple ways of enabling these hooks or collections, but the easiest way is by means of registration process built into Python's packaging system.
+This means that by simply installing a package, brand new functionality can be enabled within your ``ksconf`` command line.
+Over time, we hope that more of these plugins can be published and made available to a wider audience on pypi.
+
+
+Using plugins
+=============
+
+Existing plugins can be found on pypi by search for the `ksconf-* <https://pypi.org/search/?q=ksconf&o=&c=Environment+%3A%3A+Plugins>`__ package prefix.
+With a little bit of Python experience, it's relatively simple to write your own.
+
+Installation should be as simple as using your favorite package manager to install the plugin.  For example:
+
+.. code-block:: sh
+
+    pip install ksconf-<plugin-name>
+
+Once installed, you can confirm which plugins are loaded and activated using ``--version``.
+
+
+.. code-block:: sh
+
+    ksconf --version
+
+Output:
+
+::
+
+    _                   ___
+    | |_ ___ ___ ___ ___|  _|
+    | '_|_ -|  _| . |   |  _|
+    |_,_|___|___|___|_|_|_|
+
+    ksconf 0.11.6.dev3+e508597.dirty
+    Python: 3.9.16  (/Users/username/venv/bin/python)
+    Git SHA1 e508597d committed on 2023-09-20
+    Installed at: /Users/username/sandbox/ksconf
+    Platform:  Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000
+    Git support:  (/usr/bin/git) git version 2.39.2 (Apple Git-143)
+    Plugins:
+      package ksconf-jinja-markdown (1.0.0) from /Users/lalleman/ksconf/plugins/jinja-markdown/ksconf_jinja_markdown.py
+        hook    modify_jinja_env via add_jinja_filters
+    ...
+
+Note that your installation will likely look different.
+
+
+Troubleshooting
+===============
+
+Review hook execution
+~~~~~~~~~~~~~~~~~~~~~
+
+Currently enabling hook monitoring is handled by ``KSCONF_DEBUG`` which also controls several other troubleshooting operations, such as enabling stack traces when exceptions occur.
+
+
+Disable individual plugins
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+Plugins can be temporarily banned by using the ``KSCONF_PLUGIN_DISABLE`` environment variable.
+
+..  code-block:: sh
+
+    # Block for your entire session (or add to ~/.bashrc?)
+    export KSCONF_PLUGIN_DISABLE="jinja-markdown test-plugin2"
+
+    # Quick interactive ban (for a quick test)
+    KSCONF_PLUGIN_DISABLE=jinja-markdown ksconf package ...
+
+To permanently ban the plugin, simply remove the corresponding python package.
+
+..  code-block:: sh
+
+    pip uninstall ksconf-jinja-markdown
+
+
+
+List of plugins
+===============
+
+All plugins are defined within :py:class:`~ksconf.hookspec.KsconfHookSpecs`.
+
+.. it would be nice to have a table here...  Not sure how to do that easily... (more importantly, how to keep that in-sync with the code.
+
+
+Plugin examples
+===============
+
+
+Modify Jinja Environment
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :py:func:`~ksconf.hookspec.KsconfHookSpecs.modify_jinja_env` hook allows for modification of the Jinja2 environment so that custom filters can be added.
+This very specific hook allows a rendered Jinja2 layer file to use custom Jinja filter, so that in this case, markdown content can be rendered as HTML.
+
+.. code-block:: python
+
+    from ksconf.hook import ksconf_hook
+    from jinja2 import Environment
+
+    def markdown_to_html(md):
+        """ Jinja filter for markdown to html """
+        import commonmark
+        return commonmark.commonmark(md)
+
+    @ksconf_hook(specname="modify_jinja_env")
+    def add_jinja_filters(env: Environment):
+        """ Register new filter(s) to the Jinja environment, for use within templates. """
+        env.filters["markdown2html"] = markdown_to_html
+
+
+This specific example is bundled up as python package and is installable via:
+
+..  code-block:: sh
+
+    pip install ksconf-jinja-markdown
+
+
+
+Packaging a Plugin
+==================
+
+Packing is fairy easy, and there are examples in the ``plugins`` folder in the ksconf GitHub repository.
+Briefly, here's an example of a ``setup.py`` file.
+This example assumes your packing a plugin that lives in a ``ksconf_fancy_plugin.py``:
+
+.. code-block:: python
+
+    from setuptools import setup
+    setup(name="ksconf-fancy-plugin",
+          version="0.5.0",
+          install_requires=[
+              "ksconf>=0.11.6",
+              "some-fancy-library",   # Add 3rd party libraries here, if needed
+          ],
+          entry_points={"ksconf_plugin": ["fancy-plugin = ksconf_fancy_plugin"]},
+          py_modules=["ksconf_fancy_plugin"],
+          description="Adds general fanciness within Ksconf",
+          classifiers=["Environment :: Plugins"],
+          author="Your name",
+          author_email="your@name.example",
+          url="Repo")
+
+Then simply build and install your package.
+
+..  code-block:: sh
+
+    python setup.py install
+
+
+If you need to remove it, you can always run:
+
+..  code-block:: sh
+
+    pip uninstall ksconf-fancy-plugin
+
+All python package building and general development best practices apply, but this should be enough to get you started.
+
+
+..  include:: common

ksconf/cli.py

@@ -24,8 +24,8 @@
 import ksconf
 import ksconf.util
 from ksconf.commands import DescriptionHelpFormatterPreserveLayout, get_all_ksconf_cmds
-from ksconf.consts import EXIT_CODE_ENV_BUSTED, EXIT_CODE_INTERNAL_ERROR, is_debug, KSCONF_DEBUG
-from ksconf.hook import get_plugin_manager
+from ksconf.consts import EXIT_CODE_ENV_BUSTED, EXIT_CODE_INTERNAL_ERROR, KSCONF_DEBUG, is_debug
+from ksconf.hook import plugin_manager
 from ksconf.util.completers import autocomplete
 
 # Workaround PY2:  WindowsError: [Error -2146893795] Provider DLL failed to initialize correctly
@@ -103,8 +103,6 @@ def build_cli_parser(do_formatter=False):
     parser = argparse.ArgumentParser(**parser_kwargs)
     subparsers = parser.add_subparsers()
 
-    plugin_manager = get_plugin_manager()
-
     # XXX: Lazyload version information; this launches 'git' which is expensive.
     version_info = []
 
@@ -258,8 +256,6 @@ def cli(argv=None, _unittest=False):
     # TODO:  Rename '_unitest' to something more appropriate, maybe _exit=True?
     check_py()
 
-    plugin_manager = get_plugin_manager()
-
     parser = build_cli_parser(True)
     if not _unittest:
         autocomplete(parser)

ksconf/combine.py

@@ -17,7 +17,7 @@
 from ksconf.conf.merge import merge_conf_files
 from ksconf.conf.parser import PARSECONF_MID, PARSECONF_STRICT
 from ksconf.consts import SMART_CREATE, SMART_NOCHANGE, SMART_UPDATE
-from ksconf.hook import get_plugin_manager
+from ksconf.hook import plugin_manager
 from ksconf.layer import (DirectLayerRoot, DotDLayerRoot, LayerContext,
                           LayerFile, LayerFilter, LayerRootBase)
 from ksconf.util.compare import file_compare
@@ -126,6 +126,8 @@ def add_layer_filter(self, action, pattern):
     def combine(self, target: Path, *, hook_label=""):
         """
         Combine layers into ``target`` directory.
+        Any ``hook_label`` given will be passed to the plugin system via the
+        ``usage`` field.
         """
         layer_root = self.layer_root
         if layer_root is None:
@@ -137,8 +139,7 @@ def combine(self, target: Path, *, hook_label=""):
         self.combine_files(target, src_file_listing)
         self.post_combine(target)
 
-        pm = get_plugin_manager()
-        pm.hook.post_combine(target=target, usage=hook_label)
+        plugin_manager.hook.post_combine(target=target, usage=hook_label)
 
     def prepare(self, target: Path):
         """ Start the combine process.  This includes directory checking,

ksconf/commands/__init__.py

@@ -19,7 +19,7 @@
                                 write_conf)
 from ksconf.consts import (EXIT_CODE_BAD_ARGS, EXIT_CODE_BAD_CONF_FILE,
                            EXIT_CODE_NO_SUCH_FILE, SMART_CREATE, SmartEnum)
-from ksconf.hook import get_plugin_manager
+from ksconf.hook import plugin_manager
 from ksconf.util import debug_traceback
 
 __all__ = [
@@ -362,7 +362,6 @@ def add_parser(self, subparser):
         self.parser = subparser.add_parser(self.name, **kwargs)
         self.parser.set_defaults(funct=self.launch)
         self.register_args(self.parser)
-        plugin_manager = get_plugin_manager()
         plugin_manager.hook.ksconf_cli_modify_argparse(parser=self.parser, name=self.name)
 
     def register_args(self, parser: ArgumentParser):        # pragma: no cover

ksconf/consts.py

@@ -25,6 +25,7 @@ class _UNSET:
 MANIFEST_HASH = "sha256"
 
 PLUGGY_HOOK = "ksconf"
+PLUGGY_DISABLE_HOOK = "KSCONF_PLUGIN_DISABLE"
 
 # Legacy names
 SMART_CREATE = SmartEnum.CREATE