Backport/2.8/docs rst omnibus

docs/docsite/rst/dev_guide/developing_locally.rst

@@ -5,7 +5,7 @@
 Adding modules and plugins locally
 **********************************
 
-.. contents:: Topics
+.. contents::
    :local:
 
 The easiest, quickest, and most popular way to extend Ansible is to copy or write a module or a plugin for local use. You can store local modules and plugins on your Ansible control node for use within your team or organization. You can also share a local plugin or module by embedding it in a role and publishing it on Ansible Galaxy. If you've been using roles off Galaxy, you may have been using local modules and plugins without even realizing it. If you're using a local module or plugin that already exists, this page is all you need.
@@ -26,8 +26,8 @@ Modules and plugins: what's the difference?
 ===========================================
 If you're looking to add local functionality to Ansible, you may be wondering whether you need a module or a plugin. Here's a quick overview of the differences:
 
-* Modules are reusable, standalone scripts that can be used by the Ansible API, the :command:`ansible` command, or the :command:`ansible-playbook` command. Modules provide a defined interface, accepting arguments and returning information to Ansible by printing a JSON string to stdout before exiting.
-* Plugins are shared code that can be used by any module. They provide abilities like caching information or copying files that are useful for many modules.
+* Modules are reusable, standalone scripts that can be used by the Ansible API, the :command:`ansible` command, or the :command:`ansible-playbook` command. Modules provide a defined interface, accepting arguments and returning information to Ansible by printing a JSON string to stdout before exiting. Modules execute on the target system (usually that means on a remote system) in separate processes.
+* :ref:`Plugins <plugins_lookup>` augment Ansible's core functionality and execute on the control node within the ``/usr/bin/ansible`` process. Plugins offer options and extensions for the core features of Ansible - transforming data, logging output, connecting to inventory, and more.
 
 .. _local_modules:
 

docs/docsite/rst/dev_guide/developing_module_utilities.rst

@@ -1,87 +1,77 @@
-.. _appendix_module_utilities:
+.. _developing_module_utilities:
 
-**************************
-Appendix: Module Utilities
-**************************
+*************************************
+Using and Developing Module Utilities
+*************************************
 
-Ansible provides a number of module utilities that provide helper functions that you can use when developing your own modules. The ``basic.py`` module utility provides the main entry point for accessing the Ansible library, and all Python Ansible modules must, at minimum, import ``AnsibleModule``::
+Ansible provides a number of module utilities, or snippets of shared code, that
+provide helper functions you can use when developing your own modules. The
+``basic.py`` module utility provides the main entry point for accessing the
+Ansible library, and all Python Ansible modules must import something from
+``ansible.module_utils``. A common option is to import ``AnsibleModule``::
 
   from ansible.module_utils.basic import AnsibleModule
 
-If you need to share Python code between some of your own local modules, you can use Ansible's ``module_utils`` directories for this. When you run ``ansible-playbook``, Ansible will merge any files in the local ``module_utils`` directory into the ``ansible.module_utils`` namespace. For example, if you have your own custom modules that import a ``my_shared_code`` library, you can place that into a ``./module_utils/my_shared_code.py`` file in the root location where your playbook lives, and then import it in your modules like so::
+The ``ansible.module_utils`` namespace is not a plain Python package: it is
+constructed dynamically for each task invocation, by extracting imports and
+resolving those matching the namespace against a :ref:`search path <ansible_search_path>` derived from the
+active configuration.
+
+To reduce the maintenance burden on your own local modules, you can extract
+duplicated code into one or more module utilities and import them into your modules. For example, if you have your own custom modules that import a ``my_shared_code`` library, you can place that into a ``./module_utils/my_shared_code.py`` file like this::
 
   from ansible.module_utils.my_shared_code import MySharedCodeClient
 
-Your custom ``module_utils`` directories can live in the root directory of your playbook, or in the individual role directories, or in the directories specified by the ``ANSIBLE_MODULE_UTILS`` configuration setting.
+When you run ``ansible-playbook``, Ansible will merge any files in your local ``module_utils`` directories into the ``ansible.module_utils`` namespace in the order defined by the :ref:`Ansible search path <ansible_search_path>`.
+
+Naming and finding module utilities
+===================================
+
+You can generally tell what a module utility does from its name and/or its location. For example, ``openstack.py`` contains utilities for modules that work with OpenStack instances.
+Generic utilities (shared code used by many different kinds of modules) live in the ``common`` subdirectory or in the root directory. Utilities
+used by a particular set of modules generally live in a sub-directory that mirrors
+the directory for those modules. For example:
+
+* ``lib/ansible/module_utils/urls.py`` contains shared code for parsing URLs
+* ``lib/ansible/module_utils/storage/emc/`` contains shared code related to EMC
+*  ``lib/ansible/modules/storage/emc/`` contains modules related to EMC
+
+Following this pattern with your own module utilities makes everything easy to find and use.
+
+.. _standard_mod_utils:
+
+Standard module utilities
+=========================
 
-Ansible ships with the following list of ``module_utils`` files. The module utility source code lives in the ``./lib/ansible/module_utils`` directory under your main Ansible path - for more details on any specific module utility, please see the source code.
+Ansible ships with an extensive library of ``module_utils`` files.
+You can find the module
+utility source code in the ``lib/ansible/module_utils`` directory under
+your main Ansible path. We've described the most widely used utilities below. For more details on any specific module utility,
+please see the `source code for module_utils <https://github.com/ansible/ansible/tree/devel/lib/ansible/module_utils>`_.
 
 .. include:: shared_snippets/licensing.txt
 
-- alicloud_ecs.py - Definitions and utilities for modules working with Alibaba Cloud ECS.
-- api.py - Adds shared support for generic API modules.
-- azure_rm_common.py - Definitions and utilities for Microsoft Azure Resource Manager template deployments.
-- basic.py - General definitions and helper utilities for Ansible modules.
-- cloudstack.py  - Utilities for CloudStack modules.
-- database.py - Miscellaneous helper functions for PostGRES and MySQL
-- docker_common.py - Definitions and helper utilities for modules working with Docker.
-- ec2.py - Definitions and utilities for modules working with Amazon EC2
-- facts/- Folder containing helper functions for modules that return facts. See https://github.com/ansible/ansible/pull/23012 for more information.
-- gce.py - Definitions and helper functions for modules that work with Google Compute Engine resources.
-- ismount.py - Contains single helper function that fixes os.path.ismount
-- keycloak.py - Definitions and helper functions for modules working with the Keycloak API
-- known_hosts.py - utilities for working with known_hosts file
-- manageiq.py - Functions and utilities for modules that work with ManageIQ platform and its resources.
-- memset.py - Helper functions and utilities for interacting with Memset's API.
-- mysql.py - Allows modules to connect to a MySQL instance
-- netapp.py - Functions and utilities for modules that work with the NetApp storage platforms.
-- network/a10/a10.py - Utilities used by the a10_server module to manage A10 Networks devices.
-- network/aci/aci.py - Definitions and helper functions for modules that manage Cisco ACI Fabrics.
-- network/aireos/aireos.py - Definitions and helper functions for modules that manage Cisco WLC devices.
-- network/aos/aos.py - Module support utilities for managing Apstra AOS Server.
-- network/aruba/aruba.py - Helper functions for modules working with Aruba networking devices.
-- network/asa/asa.py - Module support utilities for managing Cisco ASA network devices.
-- network/avi/avi.py - Helper functions for modules working with AVI networking devices.
-- network/bigswitch/bigswitch_utils.py - Utilities used by the bigswitch module to manage Big Switch Networks devices.
-- network/cloudengine/ce.py - Module support utilities for managing Huawei Cloudengine switch.
-- network/cnos/cnos.py - Helper functions for modules working on devices running Lenovo CNOS.
-- network/common/config.py - Configuration utility functions for use by networking modules
-- network/common/netconf.py - Definitions and helper functions for modules that use Netconf transport.
-- network/common/parsing.py - Definitions and helper functions for Network modules.
-- network/common/network.py - Functions for running commands on networking devices
-- network/common/utils.py - Defines commands and comparison operators and other utilises for use in networking modules
-- network/dellos6/dellos6.py - Module support utilities for managing device running Dell OS6.
-- network/dellos9/dellos9.py - Module support utilities for managing device running Dell OS9.
-- network/dellos10/dellos10.py - Module support utilities for managing device running Dell OS10.
-- network/enos/enos.py - Helper functions for modules working with Lenovo ENOS devices.
-- network/eos/eos.py - Helper functions for modules working with EOS networking devices.
-- network/fortios/fortios.py - Module support utilities for managing FortiOS devices.
-- network/ios/ios.py - Definitions and helper functions for modules that manage Cisco IOS networking devices
-- network/iosxr/iosxr.py - Definitions and helper functions for modules that manage Cisco IOS-XR networking devices.
-- network/ironware/ironware.py - Module support utilities for managing Brocade IronWare devices.
-- network/junos/junos.py -  Definitions and helper functions for modules that manage Junos networking devices.
-- network/meraki/meraki.py - Utilities specifically for the Meraki network modules.
-- network/netscaler/netscaler.py - Utilities specifically for the netscaler network modules.
-- network/nso/nso.py - Utilities for modules that work with Cisco NSO.
-- network/nxos/nxos.py - Contains definitions and helper functions specific to Cisco NXOS networking devices.
-- network/onyx/onyx.py - Definitions and helper functions for modules that manage Mellanox ONYX networking devices.
-- network/ordance/ordance.py - Module support utilities for managing Ordnance devices.
-- network/sros/sros.py - Helper functions for modules working with Open vSwitch bridges.
-- network/vyos/vyos.py - Definitions and functions for working with VyOS networking
-- openstack.py - Utilities for modules that work with Openstack instances.
-- openswitch.py - Definitions and helper functions for modules that manage OpenSwitch devices
-- powershell.ps1 - Utilities for working with Microsoft Windows clients
-- pure.py - Functions and utilities for modules that work with the Pure Storage storage platforms.
-- pycompat24.py - Exception workaround for Python 2.4.
-- rax.py -  Definitions and helper functions for modules that work with Rackspace resources.
-- redhat.py - Functions for modules that manage Red Hat Network registration and subscriptions
-- service.py - Contains utilities to enable modules to work with Linux services (placeholder, not in use).
-- shell.py - Functions to allow modules to create shells and work with shell commands
-- six/__init__.py - Bundled copy of the `Six Python library <https://pythonhosted.org/six/>`_ to aid in writing code compatible with both Python 2 and Python 3.
-- splitter.py - String splitting and manipulation utilities for working with Jinja2 templates
-- urls.py - Utilities for working with http and https requests
-- utm_utils.py - Contains base class for creating new Sophos UTM Modules and helper functions for handling the rest interface of Sophos UTM
-- vca.py - Contains utilities for modules that work with VMware vCloud Air
-- vexata.py - Utilities for modules that work with Vexata storage platforms.
-- vmware.py - Contains utilities for modules that work with VMware vSphere VMs
-- xenserver.py - Contains utilities for modules that work with XenServer.
+- ``api.py`` - Supports generic API modules
+- ``basic.py`` - General definitions and helper utilities for Ansible modules
+- ``common/dict_transformations.py`` - Helper functions for dictionary transformations
+- ``common/file.py`` - Helper functions for working with files
+- ``common/text/`` - Helper functions for converting and formatting text.
+- ``common/parameters.py`` - Helper functions for dealing with module parameters
+- ``common/sys_info.py`` - Functions for getting distribution and platform information
+- ``common/validation.py`` - Helper functions for validating module parameters against a module argument spec
+- ``facts/`` - Directory of utilities for modules that return facts. See `PR 23012 <https://github.com/ansible/ansible/pull/23012>`_ for more information
+- ``ismount.py`` - Single helper function that fixes os.path.ismount
+- ``json_utils.py`` - Utilities for filtering unrelated output around module JSON output, like leading and trailing lines
+- ``known_hosts.py`` - utilities for working with known_hosts file
+- ``network/common/config.py`` - Configuration utility functions for use by networking modules
+- ``network/common/netconf.py`` - Definitions and helper functions for modules that use Netconf transport
+- ``network/common/parsing.py`` - Definitions and helper functions for Network modules
+- ``network/common/network.py`` - Functions for running commands on networking devices
+- ``network/common/utils.py`` - Defines commands and comparison operators and other utilises for use in networking modules
+- ``powershell/`` - Directory of definitions and helper functions for Windows PowerShell modules
+- ``pycompat24.py`` - Exception workaround for Python 2.4
+- ``service.py`` - Utilities to enable modules to work with Linux services (placeholder, not in use)
+- ``shell.py`` - Functions to allow modules to create shells and work with shell commands
+- ``six/__init__.py`` - Bundled copy of the `Six Python library <https://pythonhosted.org/six/>`_ to aid in writing code compatible with both Python 2 and Python 3
+- ``splitter.py`` - String splitting and manipulation utilities for working with Jinja2 templates
+- ``urls.py`` - Utilities for working with http and https requests

docs/docsite/rst/dev_guide/developing_modules_best_practices.rst

@@ -67,7 +67,7 @@ Python tips
 Importing and using shared code
 ===============================
 
-* Use shared code whenever possible - don't reinvent the wheel. Ansible offers the ``AnsibleModule`` common Python code, plus :ref:`utilities <appendix_module_utilities>` for many common use cases and patterns.
+* Use shared code whenever possible - don't reinvent the wheel. Ansible offers the ``AnsibleModule`` common Python code, plus :ref:`utilities <developing_module_utilities>` for many common use cases and patterns. You can also create documentation fragments for docs that apply to multiple modules.
 * Import ``ansible.module_utils`` code in the same place as you import other libraries.
 * Do NOT use wildcards (*) for importing other python modules; instead, list the function(s) you are importing (for example, ``from some.other_python_module.basic import otherFunction``).
 * Import custom packages in ``try``/``except``, capture any import errors, and handle them with ``fail_json()`` in ``main()``. For example: