diff --git a/docs/docsite/rst/dev_guide/developing_locally.rst b/docs/docsite/rst/dev_guide/developing_locally.rst index 0bf69a2998e1bd..68f91eb9afc47b 100644 --- a/docs/docsite/rst/dev_guide/developing_locally.rst +++ b/docs/docsite/rst/dev_guide/developing_locally.rst @@ -5,7 +5,7 @@ Adding modules and plugins locally ********************************** -.. contents:: Topics +.. contents:: :local: The easiest, quickest, and most popular way to extend Ansible is to copy or write a module or a plugin for local use. You can store local modules and plugins on your Ansible control node for use within your team or organization. You can also share a local plugin or module by embedding it in a role and publishing it on Ansible Galaxy. If you've been using roles off Galaxy, you may have been using local modules and plugins without even realizing it. If you're using a local module or plugin that already exists, this page is all you need. @@ -26,8 +26,8 @@ Modules and plugins: what's the difference? =========================================== If you're looking to add local functionality to Ansible, you may be wondering whether you need a module or a plugin. Here's a quick overview of the differences: -* Modules are reusable, standalone scripts that can be used by the Ansible API, the :command:`ansible` command, or the :command:`ansible-playbook` command. Modules provide a defined interface, accepting arguments and returning information to Ansible by printing a JSON string to stdout before exiting. -* Plugins are shared code that can be used by any module. They provide abilities like caching information or copying files that are useful for many modules. +* Modules are reusable, standalone scripts that can be used by the Ansible API, the :command:`ansible` command, or the :command:`ansible-playbook` command. Modules provide a defined interface, accepting arguments and returning information to Ansible by printing a JSON string to stdout before exiting. Modules execute on the target system (usually that means on a remote system) in separate processes. +* :ref:`Plugins ` augment Ansible's core functionality and execute on the control node within the ``/usr/bin/ansible`` process. Plugins offer options and extensions for the core features of Ansible - transforming data, logging output, connecting to inventory, and more. .. _local_modules: diff --git a/docs/docsite/rst/dev_guide/developing_module_utilities.rst b/docs/docsite/rst/dev_guide/developing_module_utilities.rst index 064e42169cd5ce..62ce781cc3aa2e 100644 --- a/docs/docsite/rst/dev_guide/developing_module_utilities.rst +++ b/docs/docsite/rst/dev_guide/developing_module_utilities.rst @@ -1,87 +1,77 @@ -.. _appendix_module_utilities: +.. _developing_module_utilities: -************************** -Appendix: Module Utilities -************************** +************************************* +Using and Developing Module Utilities +************************************* -Ansible provides a number of module utilities that provide helper functions that you can use when developing your own modules. The ``basic.py`` module utility provides the main entry point for accessing the Ansible library, and all Python Ansible modules must, at minimum, import ``AnsibleModule``:: +Ansible provides a number of module utilities, or snippets of shared code, that +provide helper functions you can use when developing your own modules. The +``basic.py`` module utility provides the main entry point for accessing the +Ansible library, and all Python Ansible modules must import something from +``ansible.module_utils``. A common option is to import ``AnsibleModule``:: from ansible.module_utils.basic import AnsibleModule -If you need to share Python code between some of your own local modules, you can use Ansible's ``module_utils`` directories for this. When you run ``ansible-playbook``, Ansible will merge any files in the local ``module_utils`` directory into the ``ansible.module_utils`` namespace. For example, if you have your own custom modules that import a ``my_shared_code`` library, you can place that into a ``./module_utils/my_shared_code.py`` file in the root location where your playbook lives, and then import it in your modules like so:: +The ``ansible.module_utils`` namespace is not a plain Python package: it is +constructed dynamically for each task invocation, by extracting imports and +resolving those matching the namespace against a :ref:`search path ` derived from the +active configuration. + +To reduce the maintenance burden on your own local modules, you can extract +duplicated code into one or more module utilities and import them into your modules. For example, if you have your own custom modules that import a ``my_shared_code`` library, you can place that into a ``./module_utils/my_shared_code.py`` file like this:: from ansible.module_utils.my_shared_code import MySharedCodeClient -Your custom ``module_utils`` directories can live in the root directory of your playbook, or in the individual role directories, or in the directories specified by the ``ANSIBLE_MODULE_UTILS`` configuration setting. +When you run ``ansible-playbook``, Ansible will merge any files in your local ``module_utils`` directories into the ``ansible.module_utils`` namespace in the order defined by the :ref:`Ansible search path `. + +Naming and finding module utilities +=================================== + +You can generally tell what a module utility does from its name and/or its location. For example, ``openstack.py`` contains utilities for modules that work with OpenStack instances. +Generic utilities (shared code used by many different kinds of modules) live in the ``common`` subdirectory or in the root directory. Utilities +used by a particular set of modules generally live in a sub-directory that mirrors +the directory for those modules. For example: + +* ``lib/ansible/module_utils/urls.py`` contains shared code for parsing URLs +* ``lib/ansible/module_utils/storage/emc/`` contains shared code related to EMC +* ``lib/ansible/modules/storage/emc/`` contains modules related to EMC + +Following this pattern with your own module utilities makes everything easy to find and use. + +.. _standard_mod_utils: + +Standard module utilities +========================= -Ansible ships with the following list of ``module_utils`` files. The module utility source code lives in the ``./lib/ansible/module_utils`` directory under your main Ansible path - for more details on any specific module utility, please see the source code. +Ansible ships with an extensive library of ``module_utils`` files. +You can find the module +utility source code in the ``lib/ansible/module_utils`` directory under +your main Ansible path. We've described the most widely used utilities below. For more details on any specific module utility, +please see the `source code for module_utils `_. .. include:: shared_snippets/licensing.txt -- alicloud_ecs.py - Definitions and utilities for modules working with Alibaba Cloud ECS. -- api.py - Adds shared support for generic API modules. -- azure_rm_common.py - Definitions and utilities for Microsoft Azure Resource Manager template deployments. -- basic.py - General definitions and helper utilities for Ansible modules. -- cloudstack.py - Utilities for CloudStack modules. -- database.py - Miscellaneous helper functions for PostGRES and MySQL -- docker_common.py - Definitions and helper utilities for modules working with Docker. -- ec2.py - Definitions and utilities for modules working with Amazon EC2 -- facts/- Folder containing helper functions for modules that return facts. See https://github.com/ansible/ansible/pull/23012 for more information. -- gce.py - Definitions and helper functions for modules that work with Google Compute Engine resources. -- ismount.py - Contains single helper function that fixes os.path.ismount -- keycloak.py - Definitions and helper functions for modules working with the Keycloak API -- known_hosts.py - utilities for working with known_hosts file -- manageiq.py - Functions and utilities for modules that work with ManageIQ platform and its resources. -- memset.py - Helper functions and utilities for interacting with Memset's API. -- mysql.py - Allows modules to connect to a MySQL instance -- netapp.py - Functions and utilities for modules that work with the NetApp storage platforms. -- network/a10/a10.py - Utilities used by the a10_server module to manage A10 Networks devices. -- network/aci/aci.py - Definitions and helper functions for modules that manage Cisco ACI Fabrics. -- network/aireos/aireos.py - Definitions and helper functions for modules that manage Cisco WLC devices. -- network/aos/aos.py - Module support utilities for managing Apstra AOS Server. -- network/aruba/aruba.py - Helper functions for modules working with Aruba networking devices. -- network/asa/asa.py - Module support utilities for managing Cisco ASA network devices. -- network/avi/avi.py - Helper functions for modules working with AVI networking devices. -- network/bigswitch/bigswitch_utils.py - Utilities used by the bigswitch module to manage Big Switch Networks devices. -- network/cloudengine/ce.py - Module support utilities for managing Huawei Cloudengine switch. -- network/cnos/cnos.py - Helper functions for modules working on devices running Lenovo CNOS. -- network/common/config.py - Configuration utility functions for use by networking modules -- network/common/netconf.py - Definitions and helper functions for modules that use Netconf transport. -- network/common/parsing.py - Definitions and helper functions for Network modules. -- network/common/network.py - Functions for running commands on networking devices -- network/common/utils.py - Defines commands and comparison operators and other utilises for use in networking modules -- network/dellos6/dellos6.py - Module support utilities for managing device running Dell OS6. -- network/dellos9/dellos9.py - Module support utilities for managing device running Dell OS9. -- network/dellos10/dellos10.py - Module support utilities for managing device running Dell OS10. -- network/enos/enos.py - Helper functions for modules working with Lenovo ENOS devices. -- network/eos/eos.py - Helper functions for modules working with EOS networking devices. -- network/fortios/fortios.py - Module support utilities for managing FortiOS devices. -- network/ios/ios.py - Definitions and helper functions for modules that manage Cisco IOS networking devices -- network/iosxr/iosxr.py - Definitions and helper functions for modules that manage Cisco IOS-XR networking devices. -- network/ironware/ironware.py - Module support utilities for managing Brocade IronWare devices. -- network/junos/junos.py - Definitions and helper functions for modules that manage Junos networking devices. -- network/meraki/meraki.py - Utilities specifically for the Meraki network modules. -- network/netscaler/netscaler.py - Utilities specifically for the netscaler network modules. -- network/nso/nso.py - Utilities for modules that work with Cisco NSO. -- network/nxos/nxos.py - Contains definitions and helper functions specific to Cisco NXOS networking devices. -- network/onyx/onyx.py - Definitions and helper functions for modules that manage Mellanox ONYX networking devices. -- network/ordance/ordance.py - Module support utilities for managing Ordnance devices. -- network/sros/sros.py - Helper functions for modules working with Open vSwitch bridges. -- network/vyos/vyos.py - Definitions and functions for working with VyOS networking -- openstack.py - Utilities for modules that work with Openstack instances. -- openswitch.py - Definitions and helper functions for modules that manage OpenSwitch devices -- powershell.ps1 - Utilities for working with Microsoft Windows clients -- pure.py - Functions and utilities for modules that work with the Pure Storage storage platforms. -- pycompat24.py - Exception workaround for Python 2.4. -- rax.py - Definitions and helper functions for modules that work with Rackspace resources. -- redhat.py - Functions for modules that manage Red Hat Network registration and subscriptions -- service.py - Contains utilities to enable modules to work with Linux services (placeholder, not in use). -- shell.py - Functions to allow modules to create shells and work with shell commands -- six/__init__.py - Bundled copy of the `Six Python library `_ to aid in writing code compatible with both Python 2 and Python 3. -- splitter.py - String splitting and manipulation utilities for working with Jinja2 templates -- urls.py - Utilities for working with http and https requests -- utm_utils.py - Contains base class for creating new Sophos UTM Modules and helper functions for handling the rest interface of Sophos UTM -- vca.py - Contains utilities for modules that work with VMware vCloud Air -- vexata.py - Utilities for modules that work with Vexata storage platforms. -- vmware.py - Contains utilities for modules that work with VMware vSphere VMs -- xenserver.py - Contains utilities for modules that work with XenServer. +- ``api.py`` - Supports generic API modules +- ``basic.py`` - General definitions and helper utilities for Ansible modules +- ``common/dict_transformations.py`` - Helper functions for dictionary transformations +- ``common/file.py`` - Helper functions for working with files +- ``common/text/`` - Helper functions for converting and formatting text. +- ``common/parameters.py`` - Helper functions for dealing with module parameters +- ``common/sys_info.py`` - Functions for getting distribution and platform information +- ``common/validation.py`` - Helper functions for validating module parameters against a module argument spec +- ``facts/`` - Directory of utilities for modules that return facts. See `PR 23012 `_ for more information +- ``ismount.py`` - Single helper function that fixes os.path.ismount +- ``json_utils.py`` - Utilities for filtering unrelated output around module JSON output, like leading and trailing lines +- ``known_hosts.py`` - utilities for working with known_hosts file +- ``network/common/config.py`` - Configuration utility functions for use by networking modules +- ``network/common/netconf.py`` - Definitions and helper functions for modules that use Netconf transport +- ``network/common/parsing.py`` - Definitions and helper functions for Network modules +- ``network/common/network.py`` - Functions for running commands on networking devices +- ``network/common/utils.py`` - Defines commands and comparison operators and other utilises for use in networking modules +- ``powershell/`` - Directory of definitions and helper functions for Windows PowerShell modules +- ``pycompat24.py`` - Exception workaround for Python 2.4 +- ``service.py`` - Utilities to enable modules to work with Linux services (placeholder, not in use) +- ``shell.py`` - Functions to allow modules to create shells and work with shell commands +- ``six/__init__.py`` - Bundled copy of the `Six Python library `_ to aid in writing code compatible with both Python 2 and Python 3 +- ``splitter.py`` - String splitting and manipulation utilities for working with Jinja2 templates +- ``urls.py`` - Utilities for working with http and https requests diff --git a/docs/docsite/rst/dev_guide/developing_modules_best_practices.rst b/docs/docsite/rst/dev_guide/developing_modules_best_practices.rst index 90a1bc4ada7dbd..35a8b296507807 100644 --- a/docs/docsite/rst/dev_guide/developing_modules_best_practices.rst +++ b/docs/docsite/rst/dev_guide/developing_modules_best_practices.rst @@ -67,7 +67,7 @@ Python tips Importing and using shared code =============================== -* Use shared code whenever possible - don't reinvent the wheel. Ansible offers the ``AnsibleModule`` common Python code, plus :ref:`utilities ` for many common use cases and patterns. +* Use shared code whenever possible - don't reinvent the wheel. Ansible offers the ``AnsibleModule`` common Python code, plus :ref:`utilities ` for many common use cases and patterns. You can also create documentation fragments for docs that apply to multiple modules. * Import ``ansible.module_utils`` code in the same place as you import other libraries. * Do NOT use wildcards (*) for importing other python modules; instead, list the function(s) you are importing (for example, ``from some.other_python_module.basic import otherFunction``). * Import custom packages in ``try``/``except``, capture any import errors, and handle them with ``fail_json()`` in ``main()``. For example: diff --git a/docs/docsite/rst/dev_guide/developing_modules_documenting.rst b/docs/docsite/rst/dev_guide/developing_modules_documenting.rst index 6d4802295c6534..10e3b7280d712f 100644 --- a/docs/docsite/rst/dev_guide/developing_modules_documenting.rst +++ b/docs/docsite/rst/dev_guide/developing_modules_documenting.rst @@ -23,21 +23,23 @@ Every Ansible module written in Python must begin with seven standard sections i .. _shebang: -Python shebang -============== +Python shebang & UTF-8 coding +=============================== Every Ansible module must begin with ``#!/usr/bin/python`` - this "shebang" allows ``ansible_python_interpreter`` to work. +This is immediately followed by ``# -*- coding: utf-8 -*-`` to clarify that the file is UTF-8 encoded. .. _copyright: Copyright and license ===================== -After the shebang, there should be a `copyright line `_ with the original copyright holder and a license declaration. The license declaration should be ONLY one line, not the full GPL prefix.: +After the shebang and UTF-8 coding, there should be a `copyright line `_ with the original copyright holder and a license declaration. The license declaration should be ONLY one line, not the full GPL prefix.: .. code-block:: python #!/usr/bin/python + # -*- coding: utf-8 -*- # Copyright: (c) 2018, Terry Jones # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt) @@ -47,6 +49,7 @@ Major additions to the module (for instance, rewrites) may add additional copyri .. code-block:: python #!/usr/bin/python + # -*- coding: utf-8 -*- # Copyright: (c) 2017, [New Contributor(s)] # Copyright: (c) 2015, [Original Contributor(s)] @@ -57,7 +60,7 @@ Major additions to the module (for instance, rewrites) may add additional copyri ANSIBLE_METADATA block ====================== -After the shebang, the copyright, and the license, your module file should contain an ``ANSIBLE_METADATA`` section. This section provides information about the module for use by other tools. For new modules, the following block can be simply added into your module: +After the shebang, the UTF-8 coding, the copyright, and the license, your module file should contain an ``ANSIBLE_METADATA`` section. This section provides information about the module for use by other tools. For new modules, the following block can be simply added into your module: .. code-block:: python @@ -96,10 +99,10 @@ Ansible metadata fields The default value is a single element list ["preview"]. The following strings are valid statuses and have the following meanings: - :stableinterface: The module's parameters are stable. Every effort will be made not to remove parameters or change + :stableinterface: The module's options (the parameters or arguments it accepts) are stable. Every effort will be made not to remove options or change their meaning. **Not** a rating of the module's code quality. :preview: The module is in tech preview. It may be - unstable, the parameters may change, or it may require libraries or + unstable, the options may change, or it may require libraries or web services that are themselves subject to incompatible changes. :deprecated: The module is deprecated and will be removed in a future release. :removed: The module is not present in the release. A stub is @@ -111,13 +114,13 @@ Ansible metadata fields DOCUMENTATION block =================== -After the shebang, the copyright line, the license, and the ``ANSIBLE_METADATA`` section comes the ``DOCUMENTATION`` block. Ansible's online module documentation is generated from the ``DOCUMENTATION`` blocks in each module's source code. The ``DOCUMENTATION`` block must be valid YAML. You may find it easier to start writing your ``DOCUMENTATION`` string in an :ref:`editor with YAML syntax highlighting ` before you include it in your Python file. You can start by copying our `example documentation string `_ into your module file and modifying it. If you run into syntax issues in your YAML, you can validate it on the `YAML Lint `_ website. +After the shebang, the UTF-8 coding, the copyright line, the license, and the ``ANSIBLE_METADATA`` section comes the ``DOCUMENTATION`` block. Ansible's online module documentation is generated from the ``DOCUMENTATION`` blocks in each module's source code. The ``DOCUMENTATION`` block must be valid YAML. You may find it easier to start writing your ``DOCUMENTATION`` string in an :ref:`editor with YAML syntax highlighting ` before you include it in your Python file. You can start by copying our `example documentation string `_ into your module file and modifying it. If you run into syntax issues in your YAML, you can validate it on the `YAML Lint `_ website. Module documentation should briefly and accurately define what each module and option does, and how it works with others in the underlying system. Documentation should be written for broad audience--readable both by experts and non-experts. * Descriptions should always start with a capital letter and end with a full stop. Consistency always helps. * Verify that arguments in doc and module spec dict are identical. * For password / secret arguments no_log=True should be set. - * If an optional parameter is sometimes required, reflect this fact in the documentation, e.g. "Required when I(state=present)." + * If an option is only sometimes required, describe the conditions. For example, "Required when I(state=present)." * If your module allows ``check_mode``, reflect this fact in the documentation. Each documentation field is described below. Before committing your module documentation, please test it at the command line and as HTML: @@ -147,16 +150,19 @@ All fields in the ``DOCUMENTATION`` block are lower-case. All fields are require * A detailed description (generally two or more sentences). * Must be written in full sentences, i.e. with capital letters and periods/full stops. * Shouldn't mention the module name. + * Make use of multiple entries rather than using one long paragraph. + * Don't quote complete values unless it is required by YAML. :version_added: * The version of Ansible when the module was added. - * This is a string, and not a float, i.e. ``version_added: "2.1"`` + * This is a string, and not a float, i.e. ``version_added: '2.1'`` :author: * Name of the module author in the form ``First Last (@GitHubID)``. * Use a multi-line list if there is more than one author. + * Don't use quotes as it should not be required by YAML. :deprecated: @@ -164,19 +170,22 @@ All fields in the ``DOCUMENTATION`` block are lower-case. All fields are require :options: + * Options are often called `parameters` or `arguments`. Because the documentation field is called `options`, we will use that term. * If the module has no options (for example, it's a ``_facts`` module), all you need is one line: ``options: {}``. - * If your module has options (in other words, accepts arguments), each option should be documented thoroughly. For each module argument/option, include: + * If your module has options (in other words, accepts arguments), each option should be documented thoroughly. For each module option, include: :option-name: * Declarative operation (not CRUD), to focus on the final state, for example `online:`, rather than `is_online:`. * The name of the option should be consistent with the rest of the module, as well as other modules in the same category. + * When in doubt, look for other modules to find option names that are used for the same purpose, we like to offer consistency to our users. :description: * Detailed explanation of what this option does. It should be written in full sentences. - * Should not list the possible values (that's what ``choices:`` is for, though it should explain `what` the values do if they aren't obvious). - * If an optional parameter is sometimes required this need to be reflected in the documentation, e.g. "Required when I(state=present)." + * The first entry is a description of the option itself; subsequent entries detail its use, dependencies, or format of possible values. + * Should not list the possible values (that's what ``choices:`` is for, though it should explain what the values do if they aren't obvious). + * If an option is only sometimes required, describe the conditions. For example, "Required when I(state=present)." * Mutually exclusive options must be documented as the final sentence on each of the options. :required: @@ -187,8 +196,8 @@ All fields in the ``DOCUMENTATION`` block are lower-case. All fields are require :default: * If ``required`` is false/missing, ``default`` may be specified (assumed 'null' if missing). - * Ensure that the default parameter in the docs matches the default parameter in the code. - * The default option must not be listed as part of the description. + * Ensure that the default value in the docs matches the default value in the code. + * The default field must not be listed as part of the description, unless it requires additional information or conditions. * If the option is a boolean value, you can use any of the boolean values recognized by Ansible: (such as true/false or yes/no). Choose the one that reads better in the context of the option. @@ -209,11 +218,11 @@ All fields in the ``DOCUMENTATION`` block are lower-case. All fields are require :version_added: * Only needed if this option was extended after initial Ansible release, i.e. this is greater than the top level `version_added` field. - * This is a string, and not a float, i.e. ``version_added: "2.3"``. + * This is a string, and not a float, i.e. ``version_added: '2.3'``. :suboptions: - * If this option takes a dict, you can define it here. + * If this option takes a dict, you can define its structure here. * See :ref:`azure_rm_securitygroup_module`, :ref:`os_ironic_node_module` for examples. :requirements: @@ -221,11 +230,6 @@ All fields in the ``DOCUMENTATION`` block are lower-case. All fields are require * List of requirements (if applicable). * Include minimum versions. -:notes: - - * Details of any important information that doesn't fit in one of the above sections. - * For example, whether ``check_mode`` is or is not supported. - :seealso: * A list of references to other modules, documentation or Internet resources @@ -252,6 +256,11 @@ All fields in the ``DOCUMENTATION`` block are lower-case. All fields are require description: Complete reference of the APIC object model. link: https://developer.cisco.com/docs/apic-mim-ref/ +:notes: + + * Details of any important information that doesn't fit in one of the above sections. + * For example, whether ``check_mode`` is or is not supported. + Linking within module documentation ----------------------------------- @@ -272,22 +281,22 @@ You can link from your module documentation to other module docs, other resource Documentation fragments ----------------------- -If you're writing multiple related modules, they may share common documentation, such as authentication details or file mode settings. Rather than duplicate that information in each module's ``DOCUMENTATION`` block, you can save it once as a doc_fragment plugin and use it in each module's documentation. In Ansible, shared documentation fragments are contained in a ``ModuleDocFragment`` class in `lib/ansible/plugins/doc_fragments/ `_. To include a documentation fragment, add ``extends_documentation_fragment: FRAGMENT_NAME`` in your module's documentation. +If you're writing multiple related modules, they may share common documentation, such as authentication details, file mode settings, ``notes:`` or ``seealso:`` entries. Rather than duplicate that information in each module's ``DOCUMENTATION`` block, you can save it once as a doc_fragment plugin and use it in each module's documentation. In Ansible, shared documentation fragments are contained in a ``ModuleDocFragment`` class in `lib/ansible/plugins/doc_fragments/ `_. To include a documentation fragment, add ``extends_documentation_fragment: FRAGMENT_NAME`` in your module's documentation. .. _note: - * in 2.8 the Ansible directories for doc fragments changed, see documentation of previous versions to find the old locations. + * Prior to Ansible 2.8, documentation fragments were kept in ``lib/ansible/utils/module_docs_fragments``. .. versionadded:: 2.8 -Since version 2.8, you can have user supplied doc_fragments by using a ``doc_fragments`` directory adjacent to play or role, just like any other plugin. +Since Ansible 2.8, you can have user-supplied doc_fragments by using a ``doc_fragments`` directory adjacent to play or role, just like any other plugin. For example, all AWS modules should include: .. code-block:: yaml+jinja extends_documentation_fragment: - - aws - - ec2 + - aws + - ec2 You can find more examples by searching for ``extends_documentation_fragment`` under the Ansible source tree. @@ -297,17 +306,21 @@ You can find more examples by searching for ``extends_documentation_fragment`` u EXAMPLES block ============== -After the shebang, the copyright line, the license, the ``ANSIBLE_METADATA`` section, and the ``DOCUMENTATION`` block comes the ``EXAMPLES`` block. Here you show users how your module works with real-world examples in multi-line plain-text YAML format. The best examples are ready for the user to copy and paste into a playbook. Review and update your examples with every change to your module. +After the shebang, the UTF-8 coding, the copyright line, the license, the ``ANSIBLE_METADATA`` section, and the ``DOCUMENTATION`` block comes the ``EXAMPLES`` block. Here you show users how your module works with real-world examples in multi-line plain-text YAML format. The best examples are ready for the user to copy and paste into a playbook. Review and update your examples with every change to your module. Per playbook best practices, each example should include a ``name:`` line:: - EXAMPLES = ''' + EXAMPLES = r''' - name: Ensure foo is installed modulename: name: foo state: present ''' +The ``name:`` line should be capitalized and not include a trailing dot. + +If your examples use boolean options, use yes/no values. Since the documentation generates boolean values as yes/no, having the examples use these values as well makes the module documentation more consistent. + If your module returns facts that are often needed, an example of how to use them can be helpful. .. _return_block: @@ -315,16 +328,16 @@ If your module returns facts that are often needed, an example of how to use the RETURN block ============ -After the shebang, the copyright line, the license, the ``ANSIBLE_METADATA`` section, ``DOCUMENTATION`` and ``EXAMPLES`` blocks comes the ``RETURN`` block. This section documents the information the module returns for use by other modules. +After the shebang, the UTF-8 coding, the copyright line, the license, the ``ANSIBLE_METADATA`` section, ``DOCUMENTATION`` and ``EXAMPLES`` blocks comes the ``RETURN`` block. This section documents the information the module returns for use by other modules. -If your module doesn't return anything (apart from the standard returns), this section of your module should read: ``RETURN = ''' # '''`` +If your module doesn't return anything (apart from the standard returns), this section of your module should read: ``RETURN = r''' # '''`` Otherwise, for each value returned, provide the following fields. All fields are required unless specified otherwise. :return name: Name of the returned field. :description: - Detailed description of what this value represents. + Detailed description of what this value represents. Capitalized and with trailing dot. :returned: When this value is returned, such as ``always``, or ``on success``. :type: @@ -333,31 +346,31 @@ Otherwise, for each value returned, provide the following fields. All fields are One or more examples. :version_added: Only needed if this return was extended after initial Ansible release, i.e. this is greater than the top level `version_added` field. - This is a string, and not a float, i.e. ``version_added: "2.3"``. + This is a string, and not a float, i.e. ``version_added: '2.3'``. :contains: Optional. To describe nested return values, set ``type: complex`` and repeat the elements above for each sub-field. Here are two example ``RETURN`` sections, one with three simple fields and one with a complex nested field:: - RETURN = ''' + RETURN = r''' dest: - description: destination file/path + description: Destination file/path. returned: success - type: string + type: str sample: /path/to/file.txt src: - description: source file used for the copy on the target machine + description: Source file used for the copy on the target machine. returned: changed - type: string + type: str sample: /home/httpd/.ansible/tmp/ansible-tmp-1423796390.97-147729857856000/source md5sum: - description: md5 checksum of the file after running copy + description: MD5 checksum of the file after running copy. returned: when supported - type: string + type: str sample: 2a5aeecc61dc98c4d780b14b330e3282 ''' - RETURN = ''' + RETURN = r''' packages: description: Information about package requirements returned: On success @@ -385,7 +398,7 @@ Here are two example ``RETURN`` sections, one with three simple fields and one w Python imports ============== -After the shebang, the copyright line, the license, and the sections for ``ANSIBLE_METADATA``, ``DOCUMENTATION``, ``EXAMPLES``, and ``RETURN``, you can finally add the python imports. All modules must use Python imports in the form: +After the shebang, the UTF-8 coding, the copyright line, the license, and the sections for ``ANSIBLE_METADATA``, ``DOCUMENTATION``, ``EXAMPLES``, and ``RETURN``, you can finally add the python imports. All modules must use Python imports in the form: .. code-block:: python diff --git a/docs/docsite/rst/dev_guide/developing_modules_in_groups.rst b/docs/docsite/rst/dev_guide/developing_modules_in_groups.rst index 89d32ccadcb556..1ded61370efd56 100644 --- a/docs/docsite/rst/dev_guide/developing_modules_in_groups.rst +++ b/docs/docsite/rst/dev_guide/developing_modules_in_groups.rst @@ -91,7 +91,6 @@ The first PR should include the following files: * ``lib/ansible/modules/$category/$topic/$yourfirstmodule.py`` - A single module. *Required new file* * ``lib/ansible/plugins/doc_fragments/$topic.py`` - Code documentation, such as details regarding common arguments. *Optional new file* * ``lib/ansible/module_utils/$topic.py`` - Code shared between more than one module, such as common arguments. *Optional new file* -* ``docs/docsite/rst/dev_guide/developing_module_utilities.rst`` - Document your new `module_utils` file. *Optional update to existing file* And that's it. diff --git a/docs/docsite/rst/dev_guide/developing_program_flow_modules.rst b/docs/docsite/rst/dev_guide/developing_program_flow_modules.rst index 718f46548be131..2b18e3d29ffde8 100644 --- a/docs/docsite/rst/dev_guide/developing_program_flow_modules.rst +++ b/docs/docsite/rst/dev_guide/developing_program_flow_modules.rst @@ -5,13 +5,9 @@ Ansible module architecture *************************** -This in-depth dive helps you understand Ansible's program flow to execute -modules. It is written for people working on the portions of the Core Ansible -Engine that execute a module. Those writing Ansible Modules may also find this -in-depth dive to be of interest, but individuals simply using Ansible Modules -will not likely find this to be helpful. +If you're working on Ansible's Core code, writing an Ansible module, or developing an action plugin, this deep dive helps you understand how Ansible's program flow executes. If you're just using Ansible Modules in playbooks, you can skip this section. -.. contents:: Topics +.. contents:: :local: .. _flow_types_of_modules: @@ -19,7 +15,7 @@ will not likely find this to be helpful. Types of modules ================ -Ansible supports several different types of modules in its code base. Some of +Ansible supports several different types of modules in its code base. Some of these are for backwards compatibility and others are to enable flexibility. .. _flow_action_plugins: @@ -27,20 +23,18 @@ these are for backwards compatibility and others are to enable flexibility. Action plugins -------------- -Action Plugins look like modules to end users who are writing :term:`playbooks` but -they're distinct entities for the purposes of this document. Action Plugins -always execute on the controller and are sometimes able to do all work there -(for instance, the ``debug`` Action Plugin which prints some text for the user to -see or the ``assert`` Action Plugin which can test whether several values in -a playbook satisfy certain criteria.) - -More often, Action Plugins set up some values on the controller, then invoke an -actual module on the managed node that does something with these values. An -easy to understand version of this is the :ref:`template Action Plugin -`. The :ref:`template Action Plugin ` takes values from +Action plugins look like modules to anyone writing a playbook. Usage documentation for most action plugins lives inside a module of the same name. Some action plugins do all the work, with the module providing only documentation. Some action plugins execute modules. The ``normal`` action plugin executes modules that don't have special action plugins. Action plugins always execute on the controller. + +Some action plugins do all their work on the controller. For +example, the :ref:`debug ` action plugin (which prints text for +the user to see) and the :ref:`assert ` action plugin (which +tests whether values in a playbook satisfy certain criteria) execute entirely on the controller. + +Most action plugins set up some values on the controller, then invoke an +actual module on the managed node that does something with these values. For example, the :ref:`template ` action plugin takes values from the user to construct a file in a temporary location on the controller using -variables from the playbook environment. It then transfers the temporary file -to a temporary file on the remote system. After that, it invokes the +variables from the playbook environment. It then transfers the temporary file +to a temporary file on the remote system. After that, it invokes the :ref:`copy module ` which operates on the remote system to move the file into its final location, sets file permissions, and so on. @@ -49,23 +43,20 @@ into its final location, sets file permissions, and so on. New-style modules ----------------- -All of the modules that ship with Ansible fall into this category. +All of the modules that ship with Ansible fall into this category. While you can write modules in any language, all official modules (shipped with Ansible) use either Python or PowerShell. New-style modules have the arguments to the module embedded inside of them in -some manner. Non-new-style modules must copy a separate file over to the +some manner. Old-style modules must copy a separate file over to the managed node, which is less efficient as it requires two over-the-wire connections instead of only one. .. _flow_python_modules: Python ------- +^^^^^^ New-style Python modules use the :ref:`Ansiballz` framework for constructing -modules. All official modules (shipped with Ansible) use either this or the -:ref:`powershell module framework `. - -These modules use imports from :code:`ansible.module_utils` in order to pull in +modules. These modules use imports from :code:`ansible.module_utils` to pull in boilerplate module code, such as argument parsing, formatting of return values as :term:`JSON`, and various file operations. @@ -76,21 +67,21 @@ values as :term:`JSON`, and various file operations. .. _flow_powershell_modules: -Powershell ----------- +PowerShell +^^^^^^^^^^ -New-style powershell modules use the :ref:`module_replacer` framework for -constructing modules. These modules get a library of powershell code embedded +New-style PowerShell modules use the :ref:`module_replacer` framework for +constructing modules. These modules get a library of PowerShell code embedded in them before being sent to the managed node. .. _flow_jsonargs_modules: -JSONARGS --------- +JSONARGS modules +---------------- -Scripts can arrange for an argument string to be placed within them by placing -the string ``<>`` somewhere inside of the -file. The module typically sets a variable to that value like this: +These modules are scripts that include the string +``<>`` in their body. +This string is replaced with the JSON-formatted argument string. These modules typically set a variable to that value like this: .. code-block:: python @@ -102,20 +93,20 @@ Which is expanded as: json_arguments = """{"param1": "test's quotes", "param2": "\"To be or not to be\" - Hamlet"}""" -.. note:: Ansible outputs a :term:`JSON` string with bare quotes. Double quotes are +.. note:: Ansible outputs a :term:`JSON` string with bare quotes. Double quotes are used to quote string values, double quotes inside of string values are backslash escaped, and single quotes may appear unescaped inside of - a string value. To use JSONARGS, your scripting language must have a way - to handle this type of string. The example uses Python's triple quoted - strings to do this. Other scripting languages may have a similar quote + a string value. To use JSONARGS, your scripting language must have a way + to handle this type of string. The example uses Python's triple quoted + strings to do this. Other scripting languages may have a similar quote character that won't be confused by any quotes in the JSON or it may allow you to define your own start-of-quote and end-of-quote characters. If the language doesn't give you any of these then you'll need to write a :ref:`non-native JSON module ` or :ref:`Old-style module ` instead. -The module typically parses the contents of ``json_arguments`` using a JSON -library and then use them as native variables throughout the rest of its code. +These modules typically parse the contents of ``json_arguments`` using a JSON +library and then use them as native variables throughout the code. .. _flow_want_json_modules: @@ -124,12 +115,12 @@ Non-native want JSON modules If a module has the string ``WANT_JSON`` in it anywhere, Ansible treats it as a non-native module that accepts a filename as its only command line -parameter. The filename is for a temporary file containing a :term:`JSON` -string containing the module's parameters. The module needs to open the file, +parameter. The filename is for a temporary file containing a :term:`JSON` +string containing the module's parameters. The module needs to open the file, read and parse the parameters, operate on the data, and print its return data as a JSON encoded dictionary to stdout before exiting. -These types of modules are self-contained entities. As of Ansible 2.1, Ansible +These types of modules are self-contained entities. As of Ansible 2.1, Ansible only modifies them to change a shebang line if present. .. seealso:: Examples of Non-native modules written in ruby are in the `Ansible @@ -140,14 +131,14 @@ only modifies them to change a shebang line if present. Binary modules -------------- -From Ansible 2.2 onwards, modules may also be small binary programs. Ansible +From Ansible 2.2 onwards, modules may also be small binary programs. Ansible doesn't perform any magic to make these portable to different systems so they may be specific to the system on which they were compiled or require other -binary runtime dependencies. Despite these drawbacks, a site may sometimes -have no choice but to compile a custom module against a specific binary -library if that's the only way they have to get access to certain resources. +binary runtime dependencies. Despite these drawbacks, you may have +to compile a custom module against a specific binary +library if that's the only way to get access to certain resources. -Binary modules take their arguments and will return data to Ansible in the same +Binary modules take their arguments and return data to Ansible in the same way as :ref:`want JSON modules `. .. seealso:: One example of a `binary module @@ -162,10 +153,8 @@ Old-style modules Old-style modules are similar to :ref:`want JSON modules `, except that the file that they take contains ``key=value`` pairs for their parameters instead of -:term:`JSON`. - -Ansible decides that a module is old-style when it doesn't have any of the -markers that would show that it is one of the other types. +:term:`JSON`. Ansible decides that a module is old-style when it doesn't have +any of the markers that would show that it is one of the other types. .. _flow_how_modules_are_executed: @@ -173,8 +162,8 @@ How modules are executed ======================== When a user uses :program:`ansible` or :program:`ansible-playbook`, they -specify a task to execute. The task is usually the name of a module along -with several parameters to be passed to the module. Ansible takes these +specify a task to execute. The task is usually the name of a module along +with several parameters to be passed to the module. Ansible takes these values and processes them in various ways before they are finally executed on the remote machine. @@ -185,44 +174,43 @@ Executor/task_executor The TaskExecutor receives the module name and parameters that were parsed from the :term:`playbook ` (or from the command line in the case of -:command:`/usr/bin/ansible`). It uses the name to decide whether it's looking -at a module or an :ref:`Action Plugin `. If it's +:command:`/usr/bin/ansible`). It uses the name to decide whether it's looking +at a module or an :ref:`Action Plugin `. If it's a module, it loads the :ref:`Normal Action Plugin ` and passes the name, variables, and other information about the task and play to that Action Plugin for further processing. .. _flow_normal_action_plugin: -Normal action plugin --------------------- +The ``normal`` action plugin +---------------------------- -The ``normal`` action plugin executes the module on the remote host. It is +The ``normal`` action plugin executes the module on the remote host. It is the primary coordinator of much of the work to actually execute the module on the managed machine. -* It takes care of creating a connection to the managed machine by - instantiating a ``Connection`` class according to the inventory - configuration for that host. -* It adds any internal Ansible variables to the module's parameters (for +* It loads the appropriate connection plugin for the task, which then transfers + or executes as needed to create a connection to that host. +* It adds any internal Ansible properties to the module's parameters (for instance, the ones that pass along ``no_log`` to the module). -* It takes care of creating any temporary files on the remote machine and +* It works with other plugins (connection, shell, become, other action plugins) + to create any temporary files on the remote machine and cleans up afterwards. -* It does the actual work of pushing the module and module parameters to the +* It pushes the module and module parameters to the remote host, although the :ref:`module_common ` - code described in the next section does the work of deciding which format + code described in the next section decides which format those will take. -* It handles any special cases regarding modules (for instance, various - complications around Windows modules that must have the same names as Python - modules, so that internal calling of modules from other Action Plugins work.) +* It handles any special cases regarding modules (for instance, async + execution, or complications around Windows modules that must have the same names as Python modules, so that internal calling of modules from other Action Plugins work.) Much of this functionality comes from the `BaseAction` class, -which lives in :file:`plugins/action/__init__.py`. It makes use of +which lives in :file:`plugins/action/__init__.py`. It uses the ``Connection`` and ``Shell`` objects to do its work. .. note:: When :term:`tasks ` are run with the ``async:`` parameter, Ansible uses the ``async`` Action Plugin instead of the ``normal`` Action Plugin - to invoke it. That program flow is currently not documented. Read the + to invoke it. That program flow is currently not documented. Read the source for information on how that works. .. _flow_executor_module_common: @@ -230,25 +218,27 @@ which lives in :file:`plugins/action/__init__.py`. It makes use of Executor/module_common.py ------------------------- -Code in :file:`executor/module_common.py` takes care of assembling the module -to be shipped to the managed node. The module is first read in, then examined -to determine its type. :ref:`PowerShell ` and -:ref:`JSON-args modules ` are passed through -:ref:`Module Replacer `. New-style -:ref:`Python modules ` are assembled by :ref:`Ansiballz`. -:ref:`Non-native-want-JSON `, -:ref:`Binary modules `, and -:ref:`Old-Style modules ` aren't touched by either of -these and pass through unchanged. After the assembling step, one final -modification is made to all modules that have a shebang line. Ansible checks +Code in :file:`executor/module_common.py` assembles the module +to be shipped to the managed node. The module is first read in, then examined +to determine its type: + +* :ref:`PowerShell ` and :ref:`JSON-args modules ` are passed through :ref:`Module Replacer `. +* New-style :ref:`Python modules ` are assembled by :ref:`Ansiballz`. +* :ref:`Non-native-want-JSON `, :ref:`Binary modules `, and :ref:`Old-Style modules ` aren't touched by either of these and pass through unchanged. + +After the assembling step, one final +modification is made to all modules that have a shebang line. Ansible checks whether the interpreter in the shebang line has a specific path configured via -an ``ansible_$X_interpreter`` inventory variable. If it does, Ansible -substitutes that path for the interpreter path given in the module. After +an ``ansible_$X_interpreter`` inventory variable. If it does, Ansible +substitutes that path for the interpreter path given in the module. After this, Ansible returns the complete module data and the module type to the :ref:`Normal Action ` which continues execution of the module. -Next we'll go into some details of the two assembler frameworks. +Assembler frameworks +-------------------- + +Ansible supports two assembler frameworks: Ansiballz and the older Module Replacer. .. _module_replacer: @@ -256,12 +246,12 @@ Module Replacer framework ^^^^^^^^^^^^^^^^^^^^^^^^^ The Module Replacer framework is the original framework implementing new-style -modules. It is essentially a preprocessor (like the C Preprocessor for those -familiar with that programming language). It does straight substitutions of -specific substring patterns in the module file. There are two types of +modules, and is still used for PowerShell modules. It is essentially a preprocessor (like the C Preprocessor for those +familiar with that programming language). It does straight substitutions of +specific substring patterns in the module file. There are two types of substitutions: -* Replacements that only happen in the module file. These are public +* Replacements that only happen in the module file. These are public replacement strings that modules can utilize to get helpful boilerplate or access to arguments. @@ -272,12 +262,10 @@ substitutions: :code:`from ansible.module_utils.basic import *` and should also only apply to new-style Python modules. - :code:`# POWERSHELL_COMMON` substitutes the contents of - :file:`ansible/module_utils/powershell.ps1`. It should only be used with + :file:`ansible/module_utils/powershell.ps1`. It should only be used with :ref:`new-style Powershell modules `. -* Replacements that are used by ``ansible.module_utils`` code. These are internal - replacement patterns. They may be used internally, in the above public - replacements, but shouldn't be used directly by modules. +* Replacements that are used by ``ansible.module_utils`` code. These are internal replacement patterns. They may be used internally, in the above public replacements, but shouldn't be used directly by modules. - :code:`"<>"` is substituted with the Ansible version. In :ref:`new-style Python modules ` under the @@ -286,29 +274,29 @@ substitutions: :attr:``AnsibleModule.ansible_version``. - :code:`"<>"` is substituted with a string which is the Python ``repr`` of the :term:`JSON` encoded module - parameters. Using ``repr`` on the JSON string makes it safe to embed in - a Python file. In new-style Python modules under the Ansiballz framework + parameters. Using ``repr`` on the JSON string makes it safe to embed in + a Python file. In new-style Python modules under the Ansiballz framework this is better accessed by instantiating an `AnsibleModule` and then using :attr:`AnsibleModule.params`. - :code:`<>` substitutes a string which is a comma separated list of file systems which have a file system dependent - security context in SELinux. In new-style Python modules, if you really + security context in SELinux. In new-style Python modules, if you really need this you should instantiate an `AnsibleModule` and then use - :attr:`AnsibleModule._selinux_special_fs`. The variable has also changed + :attr:`AnsibleModule._selinux_special_fs`. The variable has also changed from a comma separated string of file system names to an actual python list of filesystem names. - :code:`<>` substitutes the module - parameters as a JSON string. Care must be taken to properly quote the - string as JSON data may contain quotes. This pattern is not substituted + parameters as a JSON string. Care must be taken to properly quote the + string as JSON data may contain quotes. This pattern is not substituted in new-style Python modules as they can get the module parameters another way. - The string :code:`syslog.LOG_USER` is replaced wherever it occurs with the ``syslog_facility`` which was named in :file:`ansible.cfg` or any ``ansible_syslog_facility`` inventory variable that applies to this host. In - new-style Python modules this has changed slightly. If you really need to + new-style Python modules this has changed slightly. If you really need to access it, you should instantiate an `AnsibleModule` and then use - :attr:`AnsibleModule._syslog_facility` to access it. It is no longer the - actual syslog facility and is now the name of the syslog facility. See + :attr:`AnsibleModule._syslog_facility` to access it. It is no longer the + actual syslog facility and is now the name of the syslog facility. See the :ref:`documentation on internal arguments ` for details. @@ -317,32 +305,36 @@ substitutions: Ansiballz framework ^^^^^^^^^^^^^^^^^^^ -Ansible 2.1 switched from the :ref:`module_replacer` framework to the -Ansiballz framework for assembling modules. The Ansiballz framework differs -from module replacer in that it uses real Python imports of things in -:file:`ansible/module_utils` instead of merely preprocessing the module. It +The Ansiballz framework was adopted in Ansible 2.1 and is used for all new-style Python modules. Unlike the Module Replacer, Ansiballz uses real Python imports of things in +:file:`ansible/module_utils` instead of merely preprocessing the module. It does this by constructing a zipfile -- which includes the module file, files in :file:`ansible/module_utils` that are imported by the module, and some -boilerplate to pass in the module's parameters. The zipfile is then Base64 +boilerplate to pass in the module's parameters. The zipfile is then Base64 encoded and wrapped in a small Python script which decodes the Base64 encoding -and places the zipfile into a temp directory on the managed node. It then -extracts just the ansible module script from the zip file and places that in -the temporary directory as well. Then it sets the PYTHONPATH to find python -modules inside of the zip file and invokes :command:`python` on the extracted -ansible module. +and places the zipfile into a temp directory on the managed node. It then +extracts just the Ansible module script from the zip file and places that in +the temporary directory as well. Then it sets the PYTHONPATH to find Python +modules inside of the zip file and imports the Ansible module as the special name, ``__main__``. +Importing it as ``__main__`` causes Python to think that it is executing a script rather than simply +importing a module. This lets Ansible run both the wrapper script and the module code in a single copy of Python on the remote machine. .. note:: - Ansible wraps the zipfile in the Python script for two reasons: + * Ansible wraps the zipfile in the Python script for two reasons: - * for compatibility with Python 2.6 which has a less - functional version of Python's ``-m`` command line switch. - * so that pipelining will function properly. Pipelining needs to pipe the - Python module into the Python interpreter on the remote node. Python - understands scripts on stdin but does not understand zip files. + * for compatibility with Python 2.6 which has a less + functional version of Python's ``-m`` command line switch. + + * so that pipelining will function properly. Pipelining needs to pipe the + Python module into the Python interpreter on the remote node. Python + understands scripts on stdin but does not understand zip files. + + * Prior to Ansible 2.7, the module was executed via a second Python interpreter instead of being + executed inside of the same process. This change was made once Python-2.4 support was dropped + to speed up module execution. In Ansiballz, any imports of Python modules from the :py:mod:`ansible.module_utils` package trigger inclusion of that Python file -into the zipfile. Instances of :code:`#<>` in +into the zipfile. Instances of :code:`#<>` in the module are turned into :code:`from ansible.module_utils.basic import *` and :file:`ansible/module-utils/basic.py` is then included in the zipfile. Files that are included from :file:`module_utils` are themselves scanned for @@ -351,34 +343,34 @@ the zipfile as well. .. warning:: At present, the Ansiballz Framework cannot determine whether an import - should be included if it is a relative import. Always use an absolute + should be included if it is a relative import. Always use an absolute import that has :py:mod:`ansible.module_utils` in it to allow Ansiballz to determine that the file should be included. + .. _flow_passing_module_args: Passing args ------------ -In :ref:`module_replacer`, module arguments are turned into a JSON-ified -string and substituted into the combined module file. In :ref:`Ansiballz`, -the JSON-ified string is passed into the module via stdin. When -a :class:`ansible.module_utils.basic.AnsibleModule` is instantiated, -it parses this string and places the args into -:attr:`AnsibleModule.params` where it can be accessed by the module's -other code. +Arguments are passed differently by the two frameworks: + +* In :ref:`module_replacer`, module arguments are turned into a JSON-ified string and substituted into the combined module file. +* In :ref:`Ansiballz`, the JSON-ified string is part of the script which wraps the zipfile. Just before the wrapper script imports the Ansible module as ``__main__``, it monkey-patches the private, ``_ANSIBLE_ARGS`` variable in ``basic.py`` with the variable values. When a :class:`ansible.module_utils.basic.AnsibleModule` is instantiated, it parses this string and places the args into :attr:`AnsibleModule.params` where it can be accessed by the module's other code. + +.. warning:: + If you are writing modules, remember that the way we pass arguments is an internal implementation detail: it has changed in the past and will change again as soon as changes to the common module_utils + code allow Ansible modules to forgo using :class:`ansible.module_utils.basic.AnsibleModule`. Do not rely on the internal global ``_ANSIBLE_ARGS`` variable. + + Very dynamic custom modules which need to parse arguments before they + instantiate an ``AnsibleModule`` may use ``_load_params`` to retrieve those parameters. + Although ``_load_params`` may change in breaking ways if necessary to support + changes in the code, it is likely to be more stable than either the way we pass parameters or the internal global variable. .. note:: - Internally, the `AnsibleModule` uses the helper function, - :py:func:`ansible.module_utils.basic._load_params`, to load the parameters - from stdin and save them into an internal global variable. Very dynamic - custom modules which need to parse the parameters prior to instantiating - an ``AnsibleModule`` may use ``_load_params`` to retrieve the - parameters. Be aware that ``_load_params`` is an internal function and - may change in breaking ways if necessary to support changes in the code. - However, we'll do our best not to break it gratuitously, which is not - something that can be said for either the way parameters are passed or - the internal global variable. + Prior to Ansible 2.7, the Ansible module was invoked in a second Python interpreter and the + arguments were then passed to the script over the script's stdin. + .. _flow_internal_arguments: @@ -386,93 +378,85 @@ Internal arguments ------------------ Both :ref:`module_replacer` and :ref:`Ansiballz` send additional arguments to -the module beyond those which the user specified in the playbook. These +the module beyond those which the user specified in the playbook. These additional arguments are internal parameters that help implement global -Ansible features. Modules often do not need to know about these explicitly as +Ansible features. Modules often do not need to know about these explicitly as the features are implemented in :py:mod:`ansible.module_utils.basic` but certain features need support from the module so it's good to know about them. +The internal arguments listed here are global. If you need to add a local internal argument to a custom module, create an action plugin for that specific module - see ``_original_basename`` in the `copy action plugin `_ for an example. + _ansible_no_log ^^^^^^^^^^^^^^^ -This is a boolean. If it's True then the playbook specified ``no_log`` (in -a task's parameters or as a play parameter). This automatically affects calls -to :py:meth:`AnsibleModule.log`. If a module implements its own logging then -it needs to check this value. The best way to look at this is for the module -to instantiate an `AnsibleModule` and then check the value of -:attr:`AnsibleModule.no_log`. +Boolean. Set to True whenever a parameter in a task or play specifies ``no_log``. Any module that calls :py:meth:`AnsibleModule.log` handles this automatically. If a module implements its own logging then +it needs to check this value. To access in a module, instantiate an +``AnsibleModule`` and then check the value of :attr:`AnsibleModule.no_log`. .. note:: - ``no_log`` specified in a module's argument_spec are handled by a different mechanism. + ``no_log`` specified in a module's argument_spec is handled by a different mechanism. _ansible_debug ^^^^^^^^^^^^^^^ -This is a boolean that turns on more verbose logging. If a module uses +Boolean. Turns more verbose logging on or off and turns on logging of +external commands that the module executes. If a module uses :py:meth:`AnsibleModule.debug` rather than :py:meth:`AnsibleModule.log` then -the messages are only logged if this is True. This also turns on logging of -external commands that the module executes. This can be changed via -the ``debug`` setting in :file:`ansible.cfg` or the environment variable -:envvar:`ANSIBLE_DEBUG`. If, for some reason, a module must access this, it -should do so by instantiating an `AnsibleModule` and accessing -:attr:`AnsibleModule._debug`. +the messages are only logged if ``_ansible_debug`` is set to ``True``. +To set, add ``debug: True`` to :file:`ansible.cfg` or set the environment +variable :envvar:`ANSIBLE_DEBUG`. To access in a module, instantiate an +``AnsibleModule`` and access :attr:`AnsibleModule._debug`. _ansible_diff ^^^^^^^^^^^^^^^ -This boolean is turned on via the ``--diff`` command line option. If a module -supports it, it will tell the module to show a unified diff of changes to be -made to templated files. The proper way for a module to access this is by -instantiating an `AnsibleModule` and accessing +Boolean. If a module supports it, tells the module to show a unified diff of +changes to be made to templated files. To set, pass the ``--diff`` command line +option. To access in a module, instantiate an `AnsibleModule` and access :attr:`AnsibleModule._diff`. _ansible_verbosity ^^^^^^^^^^^^^^^^^^ -This value could be used for finer grained control over logging. However, it -is currently unused. +Unused. This value could be used for finer grained control over logging. _ansible_selinux_special_fs ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This is a list of names of filesystems which should have a special selinux -context. They are used by the `AnsibleModule` methods which operate on -files (changing attributes, moving, and copying). The list of names is set -via a comma separated string of filesystem names from :file:`ansible.cfg`:: +List. Names of filesystems which should have a special SELinux +context. They are used by the `AnsibleModule` methods which operate on +files (changing attributes, moving, and copying). To set, add a comma separated string of filesystem names in :file:`ansible.cfg`:: # ansible.cfg [selinux] special_context_filesystems=nfs,vboxsf,fuse,ramfs -If a module cannot use the builtin ``AnsibleModule`` methods to manipulate -files and needs to know about these special context filesystems, it should -instantiate an ``AnsibleModule`` and then examine the list in +Most modules can use the built-in ``AnsibleModule`` methods to manipulate +files. To access in a module that needs to know about these special context filesystems, instantiate an ``AnsibleModule`` and examine the list in :attr:`AnsibleModule._selinux_special_fs`. This replaces :attr:`ansible.module_utils.basic.SELINUX_SPECIAL_FS` from -:ref:`module_replacer`. In module replacer it was a comma separated string of -filesystem names. Under Ansiballz it's an actual list. +:ref:`module_replacer`. In module replacer it was a comma separated string of +filesystem names. Under Ansiballz it's an actual list. .. versionadded:: 2.1 _ansible_syslog_facility ^^^^^^^^^^^^^^^^^^^^^^^^ -This parameter controls which syslog facility ansible module logs to. It may -be set by changing the ``syslog_facility`` value in :file:`ansible.cfg`. Most +This parameter controls which syslog facility Ansible module logs to. To set, change the ``syslog_facility`` value in :file:`ansible.cfg`. Most modules should just use :meth:`AnsibleModule.log` which will then make use of -this. If a module has to use this on its own, it should instantiate an +this. If a module has to use this on its own, it should instantiate an `AnsibleModule` and then retrieve the name of the syslog facility from -:attr:`AnsibleModule._syslog_facility`. The code will look slightly different -than it did under :ref:`module_replacer` due to how hacky the old way was +:attr:`AnsibleModule._syslog_facility`. The Ansiballz code is less hacky than the old :ref:`module_replacer` code: .. code-block:: python - # Old way + # Old module_replacer way import syslog syslog.openlog(NAME, 0, syslog.LOG_USER) - # New way + # New Ansiballz way import syslog facility_name = module._syslog_facility facility = getattr(syslog, facility_name, syslog.LOG_USER) @@ -483,14 +467,29 @@ than it did under :ref:`module_replacer` due to how hacky the old way was _ansible_version ^^^^^^^^^^^^^^^^ -This parameter passes the version of ansible that runs the module. To access +This parameter passes the version of Ansible that runs the module. To access it, a module should instantiate an `AnsibleModule` and then retrieve it -from :attr:`AnsibleModule.ansible_version`. This replaces +from :attr:`AnsibleModule.ansible_version`. This replaces :attr:`ansible.module_utils.basic.ANSIBLE_VERSION` from :ref:`module_replacer`. .. versionadded:: 2.1 + +.. _flow_module_return_values: + +Module return values & Unsafe strings +------------------------------------- + +At the end of a module's execution, it formats the data that it wants to return as a JSON string and prints the string to its stdout. The normal action plugin receives the JSON string, parses it into a Python dictionary, and returns it to the executor. + +If Ansible templated every string return value, it would be vulnerable to an attack from users with access to managed nodes. If an unscrupulous user disguised malicious code as Ansible return value strings, and if those strings were then templated on the controller, Ansible could execute arbitrary code. To prevent this scenario, Ansible marks all strings inside returned data as ``Unsafe``, emitting any Jinja2 templates in the strings verbatim, not expanded by Jinja2. + +Strings returned by invoking a module through ``ActionPlugin._execute_module()`` are automatically marked as ``Unsafe`` by the normal action plugin. If another action plugin retrieves information from a module through some other means, it must mark its return data as ``Unsafe`` on its own. + +In case a poorly-coded action plugin fails to mark its results as "Unsafe," Ansible audits the results again when they are returned to the executor, +marking all strings as ``Unsafe``. The normal action plugin protects itself and any other code that it calls with the result data as a parameter. The check inside the executor protects the output of all other action plugins, ensuring that subsequent tasks run by Ansible will not template anything from those results either. + .. _flow_special_considerations: Special considerations @@ -510,7 +509,7 @@ Ansible can transfer a module to a remote machine in one of two ways: into the remote interpreter's stdin. Pipelining only works with modules written in Python at this time because -Ansible only knows that Python supports this mode of operation. Supporting +Ansible only knows that Python supports this mode of operation. Supporting pipelining means that whatever format the module payload takes before being sent over the wire must be executable by Python via stdin. @@ -522,13 +521,13 @@ Why pass args over stdin? Passing arguments via stdin was chosen for the following reasons: * When combined with :ref:`ANSIBLE_PIPELINING`, this keeps the module's arguments from - temporarily being saved onto disk on the remote machine. This makes it + temporarily being saved onto disk on the remote machine. This makes it harder (but not impossible) for a malicious user on the remote machine to steal any sensitive information that may be present in the arguments. * Command line arguments would be insecure as most systems allow unprivileged users to read the full commandline of a process. * Environment variables are usually more secure than the commandline but some - systems limit the total size of the environment. This could lead to + systems limit the total size of the environment. This could lead to truncation of the parameters if we hit that limit. diff --git a/docs/docsite/rst/dev_guide/overview_architecture.rst b/docs/docsite/rst/dev_guide/overview_architecture.rst index a9ed81da9fe19e..c62d33cd06c854 100644 --- a/docs/docsite/rst/dev_guide/overview_architecture.rst +++ b/docs/docsite/rst/dev_guide/overview_architecture.rst @@ -1,5 +1,5 @@ ******************** -Ansible Architecture +Ansible architecture ******************** Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs. @@ -10,26 +10,35 @@ It uses no agents and no additional custom security infrastructure, so it's easy In this section, we'll give you a really quick overview of how Ansible works so you can see how the pieces fit together. +.. contents:: + :local: + Modules ======= -Ansible works by connecting to your nodes and pushing out small programs, called "Ansible Modules" to them. These programs are written to be resource models of the desired state of the system. Ansible then executes these modules (over SSH by default), and removes them when finished. +Ansible works by connecting to your nodes and pushing out scripts called "Ansible modules" to them. Most modules accept parameters that describe the desired state of the system. +Ansible then executes these modules (over SSH by default), and removes them when finished. Your library of modules can reside on any machine, and there are no servers, daemons, or databases required. + +You can :ref:`write your own modules `, though you should first consider :ref:`whether you should `. Typically you'll work with your favorite terminal program, a text editor, and probably a version control system to keep track of changes to your content. You may write specialized modules in any language that can return JSON (Ruby, Python, bash, etc). + +Module utilities +================ -Your library of modules can reside on any machine, and there are no servers, daemons, or databases required. Typically you'll work with your favorite terminal program, a text editor, and probably a version control system to keep track of changes to your content. +When multiple modules use the same code, Ansible stores those functions as module utilities to minimize duplication and maintenance. For example, the code that parses URLs is ``lib/ansible/module_utils/url.py``. You can :ref:`write your own module utilities ` as well. Module utilities may only be written in Python or in PowerShell. Plugins ======= -Plugins are pieces of code that augment Ansible's core functionality. Ansible ships with a number of handy plugins, and you can easily write your own. +:ref:`Plugins ` augment Ansible's core functionality. While modules execute on the target system in separate processes (usually that means on a remote system), plugins execute on the control node within the ``/usr/bin/ansible`` process. Plugins offer options and extensions for the core features of Ansible - transforming data, logging output, connecting to inventory, and more. Ansible ships with a number of handy plugins, and you can easily :ref:`write your own `. For example, you can write an :ref:`inventory plugin ` to connect to any datasource that returns JSON. Plugins must be written in Python. Inventory ========= -By default, Ansible represents what machines it manages using a very simple INI file that puts all of your managed machines in groups of your own choosing. +By default, Ansible represents the machines it manages in a file (INI, YAML, etc.) that puts all of your managed machines in groups of your own choosing. To add new machines, there is no additional SSL signing server involved, so there's never any hassle deciding why a particular machine didn't get linked up due to obscure NTP or DNS issues. -If there's another source of truth in your infrastructure, Ansible can also plugin to that, such as drawing inventory, group, and variable information from sources like EC2, Rackspace, OpenStack, and more. +If there's another source of truth in your infrastructure, Ansible can also connect to that. Ansible can draw inventory, group, and variable information from sources like EC2, Rackspace, OpenStack, and more. Here's what a plain text inventory file looks like:: @@ -67,8 +76,74 @@ Here's what a simple playbook looks like:: - common - content +.. _ansible_search_path: + +The Ansible search path +======================= + +Modules, module utilities, plugins, playbooks, and roles can live in multiple locations. If you +write your own code to extend Ansible's core features, you may have multiple files with similar or the same names in different locations on your Ansible control node. The search path determines which of these files Ansible will discover and use on any given playbook run. + +Ansible's search path grows incrementally over a run. As +Ansible finds each playbook and role included in a given run, it appends +any directories related to that playbook or role to the search path. Those +directories remain in scope for the duration of the run, even after the playbook or role +has finished executing. Ansible loads modules, module utilities, and plugins in this order: + +1. Directories adjacent to a playbook specified on the command line. If you run Ansible with ``ansible-playbook /path/to/play.yml``, Ansible appends these directories if they exist: + + .. code-block:: bash + + /path/to/modules + /path/to/module_utils + /path/to/plugins + +2. Directories adjacent to a playbook that is statically imported by a + playbook specified on the command line. If ``play.yml`` includes + ``- import_playbook: /path/to/subdir/play1.yml``, Ansible appends these directories if they exist: + + .. code-block:: bash + + /path/to/subdir/modules + /path/to/subdir/module_utils + /path/to/subdir/plugins + +3. Subdirectories of a role directory referenced by a playbook. If + ``play.yml`` runs ``myrole``, Ansible appends these directories if they exist: + + .. code-block:: bash + + /path/to/roles/myrole/modules + /path/to/roles/myrole/module_utils + /path/to/roles/myrole/plugins + +4. Directories specified as default paths in ``ansible.cfg`` or by the related + environment variables, including the paths for the various plugin types. See :ref:`ansible_configuration_settings` for more information. + Sample ``ansible.cfg`` fields: + + .. code-block:: bash + + DEFAULT_MODULE_PATH + DEFAULT_MODULE_UTILS_PATH + DEFAULT_CACHE_PLUGIN_PATH + DEFAULT_FILTER_PLUGIN_PATH + + Sample environment variables: + + .. code-block:: bash + + ANSIBLE_LIBRARY + ANSIBLE_MODULE_UTILS + ANSIBLE_CACHE_PLUGINS + ANSIBLE_FILTER_PLUGINS + +5. The standard directories that ship as part of the Ansible distribution. + +.. caution:: -Extending Ansible with plug-ins and the API -=========================================== + Modules, module utilities, and plugins in user-specified directories will + override the standard versions. This includes some files with generic names. + For example, if you have a file named ``basic.py`` in a user-specified + directory, it will override the standard ``ansible.module_utils.basic``. -Should you want to write your own, Ansible modules can be written in any language that can return JSON (Ruby, Python, bash, etc). Inventory can also plug in to any datasource by writing a program that speaks to that datasource and returns JSON. There's also various Python APIs for extending Ansible's connection types (SSH is not the only transport possible), callbacks (how Ansible logs, etc), and even for adding new server side behaviors. + If you have more than one module, module utility, or plugin with the same name in different user-specified directories, the order of commands at the command line and the order of includes and roles in each play will affect which one is found and used on that particular play. diff --git a/docs/docsite/rst/dev_guide/testing_documentation.rst b/docs/docsite/rst/dev_guide/testing_documentation.rst index c9aca21d604c0b..ff58836524604d 100644 --- a/docs/docsite/rst/dev_guide/testing_documentation.rst +++ b/docs/docsite/rst/dev_guide/testing_documentation.rst @@ -10,17 +10,27 @@ Before you submit a module for inclusion in the main Ansible repo, you must test To check the HTML output of your module documentation: -#. Save your completed module file into the correct directory: ``lib/ansible/modules/$CATEGORY/my_code.py``. -#. Move to the docsite directory: ``cd /path/to/ansible/docs/docsite/``. -#. Run the command to build the docs for your module: ``MODULES=my_code make webdocs``. -#. View the HTML page at ``file:///path/to/ansible/docs/docsite/_build/html/my_code_module.html``. +#. Ensure working :ref:`development environment `. +#. Install required Python packages (drop '--user' in venv/virtualenv): -To build the HTML documentation for multiple modules, use a comma-separated list of module names: ``MODULES=my_code,my_other_code make webdocs``. + .. code-block:: bash -To ensure that your documentation matches your ``argument_spec``, run the ``validate-modules`` test. + pip install --user -r requirements.txt + pip install --user -r docs/docsite/requirements.txt -.. code-block:: bash +#. Ensure your module is in the correct directory: ``lib/ansible/modules/$CATEGORY/mymodule.py``. +#. Build HTML from your module documentation: ``MODULES=mymodule make webdocs``. +#. To build the HTML documentation for multiple modules, use a comma-separated list of module names: ``MODULES=mymodule,mymodule2 make webdocs``. +#. View the HTML page at ``file:///path/to/docs/docsite/_build/html/modules/mymodule_module.html``. - # If you don't already, ensure you are using your local checkout - source hacking/env-setup - ./test/sanity/validate-modules/validate-modules --arg-spec --warnings lib/ansible/modules/$CATEGORY/my_code.py +To ensure that your module documentation matches your ``argument_spec``: + +#. Install required Python packages (drop '--user' in venv/virtualenv): + + .. code-block:: bash + + pip install --user -r test/runner/requirements/sanity.txt + +#. run the ``validate-modules`` test:: + + ./test/sanity/validate-modules/validate-modules --arg-spec --warnings lib/ansible/modules/$CATEGORY/mymodule.py diff --git a/docs/docsite/rst/installation_guide/intro_installation.rst b/docs/docsite/rst/installation_guide/intro_installation.rst index 0beba6b7734b4a..05e9a3d4cc44c7 100644 --- a/docs/docsite/rst/installation_guide/intro_installation.rst +++ b/docs/docsite/rst/installation_guide/intro_installation.rst @@ -150,7 +150,7 @@ To configure the PPA on your machine and install ansible run these commands: $ sudo apt-add-repository --yes --update ppa:ansible/ansible $ sudo apt install ansible -.. note:: On older Ubuntu distributions, "software-properties-common" is called "python-software-properties". You may want to use ``apt-get`` instead of ``apt`` in older versions. +.. note:: On older Ubuntu distributions, "software-properties-common" is called "python-software-properties". You may want to use ``apt-get`` instead of ``apt`` in older versions. Also, be aware that only newer distributions (i.e. 18.04, 18.10, etc.) have a ``-u`` or ``--update`` flag, so adjust your script accordingly. Debian/Ubuntu packages can also be built from the source checkout, run: diff --git a/docs/docsite/rst/porting_guides/porting_guide_2.5.rst b/docs/docsite/rst/porting_guides/porting_guide_2.5.rst index 3fdeaf633cb12a..b193632a1e21fc 100644 --- a/docs/docsite/rst/porting_guides/porting_guide_2.5.rst +++ b/docs/docsite/rst/porting_guides/porting_guide_2.5.rst @@ -143,7 +143,7 @@ Ansible fact namespacing Ansible facts, which have historically been written to names like ``ansible_*`` in the main facts namespace, have been placed in their own new namespace, ``ansible_facts.*`` For example, the fact ``ansible_distribution`` is now best -queried through the variable structure ``ansible_facts.distribution``. +queried through the variable structure ``ansible_facts.distribution``. A new configuration variable, ``inject_facts_as_vars``, has been added to ansible.cfg. Its default setting, 'True', keeps the 2.4 behavior of facts @@ -389,4 +389,4 @@ If your module uses shared module utilities, you must update all references. For from ansible.module_utils.network.vyos.vyos import get_config, load_config -See the module utilities developer guide see :ref:`appendix_module_utilities` for more information. +See the module utilities developer guide see :ref:`developing_module_utilities` for more information. diff --git a/docs/docsite/rst/reference_appendices/faq.rst b/docs/docsite/rst/reference_appendices/faq.rst index 8e03f4c9f5fd50..7a4c91bae0ebe7 100644 --- a/docs/docsite/rst/reference_appendices/faq.rst +++ b/docs/docsite/rst/reference_appendices/faq.rst @@ -96,6 +96,13 @@ With earlier versions of Ansible, it was necessary to configure a suitable `ProxyCommand` for one or more hosts in `~/.ssh/config`, or globally by setting `ssh_args` in `ansible.cfg`. +.. _ssh_serveraliveinterval: + +How do I get Ansible to notice a dead target in a timely manner? +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +You can add ``-o ServerAliveInterval=NumberOfSeconds`` in ``ssh_args`` from ``ansible.cfg``. Without this option, SSH and therefore Ansible will wait until the TCP connection times out. Another solution is to add ``ServerAliveInterval`` into your global SSH configuration. A good value for ``ServerAliveInterval`` is up to you to decide; keep in mind that ``ServerAliveCountMax=3`` is the SSH default so any value you set will be tripled before terminating the SSH session. + .. _ec2_cloud_performance: How do I speed up management inside EC2? diff --git a/docs/docsite/rst/scenario_guides/guide_aws.rst b/docs/docsite/rst/scenario_guides/guide_aws.rst index 8485f4eb0d9daa..f0aabaccd7b629 100644 --- a/docs/docsite/rst/scenario_guides/guide_aws.rst +++ b/docs/docsite/rst/scenario_guides/guide_aws.rst @@ -132,6 +132,37 @@ With the host group now created, a second play at the bottom of the same provisi - name: Check NTP service service: name=ntpd state=started +.. _aws_security_groups: + +Security Groups +``````````````` + +Security groups on AWS are stateful. The response of a request from your instance is allowed to flow in regardless of inbound security group rules and vice-versa. +In case you only want allow traffic with AWS S3 service, you need to fetch the current IP ranges of AWS S3 for one region and apply them as an egress rule.:: + + - name: fetch raw ip ranges for aws s3 + set_fact: + raw_s3_ranges: "{{ lookup('aws_service_ip_ranges', region='eu-central-1', service='S3', wantlist=True) }}" + + - name: prepare list structure for ec2_group module + set_fact: + s3_ranges: "{{ s3_ranges | default([]) + [{'proto': 'all', 'cidr_ip': item, 'rule_desc': 'S3 Service IP range'}] }}" + with_items: "{{ raw_s3_ranges }}" + + - name: set S3 IP ranges to egress rules + ec2_group: + name: aws_s3_ip_ranges + description: allow outgoing traffic to aws S3 service + region: eu-central-1 + state: present + vpc_id: vpc-123456 + purge_rules: true + purge_rules_egress: true + rules: [] + rules_egress: "{{ s3_ranges }}" + tags: + Name: aws_s3_ip_ranges + .. _aws_host_inventory: Host Inventory diff --git a/docs/docsite/rst/scenario_guides/guide_gce.rst b/docs/docsite/rst/scenario_guides/guide_gce.rst index 4dd678bdb79b2d..39ac95b7748f6e 100644 --- a/docs/docsite/rst/scenario_guides/guide_gce.rst +++ b/docs/docsite/rst/scenario_guides/guide_gce.rst @@ -29,7 +29,7 @@ used, but you may experience issues trying to use them together. While the community GCP modules are not going away, Google is investing effort into the new "gcp_*" modules. Google is committed to ensuring the Ansible -community has a great experience with GCP and therefore recommends adopting +community has a great experience with GCP and therefore recommends adopting these new modules if possible. @@ -42,7 +42,7 @@ The Google Cloud Platform (GCP) modules require both the ``requests`` and the $ pip install requests google-auth -Alternatively for RHEL / CentOS, the ``python-requests`` package is also +Alternatively for RHEL / CentOS, the ``python-requests`` package is also available to satisfy ``requests`` libraries. .. code-block:: bash @@ -169,9 +169,9 @@ rest. hosts: localhost gather_facts: no vars: - project: my-project - auth_kind: serviceaccount - service_account_file: /home/my_account.json + gcp_project: my-project + gcp_cred_kind: serviceaccount + gcp_cred_file: /home/my_account.json zone: "us-central1-a" region: "us-central1" @@ -234,10 +234,10 @@ rest. register: instance - name: Wait for SSH to come up - wait_for: host={{ instance.address }} port=22 delay=10 timeout=60 + wait_for: host={{ address.address }} port=22 delay=10 timeout=60 - name: Add host to groupname - add_host: hostname={{ instance.address }} groupname=new_instances + add_host: hostname={{ address.address }} groupname=new_instances - name: Manage new instances @@ -268,7 +268,7 @@ module (and more!). Below is a mapping of ``gce`` fields over to ``gcp_compute_instance`` fields. ============================ ========================================== ====================== - gce.py gcp_compute_instance.py Notes + gce.py gcp_compute_instance.py Notes ============================ ========================================== ====================== state state/status State on gce has multiple values: "present", "absent", "stopped", "started", "terminated". State on gcp_compute_instance is used to describe if the instance exists (present) or does not (absent). Status is used to describe if the instance is "started", "stopped" or "terminated". image disks[].initialize_params.source_image You'll need to create a single disk using the disks[] parameter and set it to be the boot disk (disks[].boot = true) diff --git a/docs/docsite/rst/user_guide/playbooks_async.rst b/docs/docsite/rst/user_guide/playbooks_async.rst index b17cfb5d723ec3..b385749e19e4b8 100644 --- a/docs/docsite/rst/user_guide/playbooks_async.rst +++ b/docs/docsite/rst/user_guide/playbooks_async.rst @@ -9,9 +9,19 @@ be running operations that take longer than the SSH timeout. To avoid blocking or timeout issues, you can use asynchronous mode to run all of your tasks at once and then poll until they are done. +The behaviour of asynchronous mode depends on the value of `poll`. + + +Avoid connection timeouts: poll > 0 +----------------------------------- + +When ``poll`` is a positive value, the playbook will *still* block on the task until it either completes, fails or times out. + +In this case, however, `async` explicitly sets the timeout you wish to apply to this task rather than being limited by the connection method timeout. + To launch a task asynchronously, specify its maximum runtime and how frequently you would like to poll for status. The default -poll value is 10 seconds if you do not specify a value for `poll`:: +poll value is 15 seconds if you do not specify a value for `poll`:: --- @@ -35,8 +45,21 @@ poll value is 10 seconds if you do not specify a value for `poll`:: task when run in check mode. See :doc:`playbooks_checkmode` on how to skip a task in check mode. -Alternatively, if you do not need to wait on the task to complete, you may -run the task asynchronously by specifying a poll value of 0:: + +Concurrent tasks: poll = 0 +-------------------------- + +When ``poll`` is 0, Ansible will start the task and immediately move on to the next one without waiting for a result. + +From the point of view of sequencing this is asynchronous programming: tasks may now run concurrently. + +The playbook run will end without checking back on async tasks. + +The async tasks will run until they either complete, fail or timeout according to their `async` value. + +If you need a synchronization point with a task, register it to obtain its job ID and use the :ref:`async_status ` module to observe it. + +You may run a task asynchronously by specifying a poll value of 0:: --- diff --git a/docs/docsite/rst/user_guide/playbooks_error_handling.rst b/docs/docsite/rst/user_guide/playbooks_error_handling.rst index 2cdfc691c5fa78..d77482003a6d79 100644 --- a/docs/docsite/rst/user_guide/playbooks_error_handling.rst +++ b/docs/docsite/rst/user_guide/playbooks_error_handling.rst @@ -63,11 +63,9 @@ the handler from running, such as a host becoming unreachable.) Controlling What Defines Failure ```````````````````````````````` -Suppose the error code of a command is meaningless and to tell if there -is a failure what really matters is the output of the command, for instance -if the string "FAILED" is in the output. +Ansible lets you define what "failure" means in each task using the ``failed_when`` conditional. As with all conditionals in Ansible, lists of multiple ``failed_when`` conditions are joined with an implicit ``and``, meaning the task only fails when *all* conditions are met. If you want to trigger a failure when any of the conditions is met, you must define the conditions in a string with an explicit ``or`` operator. -Ansible provides a way to specify this behavior as follows:: +You may check for failure by searching for a word or phrase in the output of a command:: - name: Fail task when the command error output prints FAILED command: /usr/bin/example-command -x -y -z @@ -93,14 +91,29 @@ In previous version of Ansible, this can still be accomplished as follows:: msg: "the command failed" when: "'FAILED' in command_result.stderr" -You can also combine multiple conditions to specify this behavior as follows:: +You can also combine multiple conditions for failure. This task will fail if both conditions are true:: - name: Check if a file exists in temp and fail task if it does command: ls /tmp/this_should_not_be_here register: result failed_when: - - '"No such" not in result.stdout' - result.rc == 0 + - '"No such" not in result.stdout' + +If you want the task to fail when only one condition is satisfied, change the ``failed_when`` definition to:: + + failed_when: result.rc == 0 or "No such" not in result.stdout + +If you have too many conditions to fit neatly into one line, you can split it into a multi-line yaml value with ``>``:: + + + - name: example of many failed_when conditions with OR + shell: "./myBinary" + register: ret + failed_when: > + ("No such file or directory" in ret.stdout) or + (ret.stderr != '') or + (ret.rc == 10) .. _override_the_changed_result: @@ -168,7 +181,7 @@ Blocks only deal with 'failed' status of a task. A bad task definition or an unr - debug: msg: 'I caught an error, can do stuff here to fix it, :-)' -This will 'revert' the failed status of the outer ``block`` task for the run and the play will continue as if it had succeeded. +This will 'revert' the failed status of the outer ``block`` task for the run and the play will continue as if it had succeeded. See :ref:`block_error_handling` for more examples. .. seealso:: @@ -185,5 +198,3 @@ See :ref:`block_error_handling` for more examples. Have a question? Stop by the google group! `irc.freenode.net `_ #ansible IRC chat channel - - diff --git a/docs/docsite/rst/user_guide/playbooks_loops.rst b/docs/docsite/rst/user_guide/playbooks_loops.rst index 8b09330e4ccb3f..551384205bb15d 100644 --- a/docs/docsite/rst/user_guide/playbooks_loops.rst +++ b/docs/docsite/rst/user_guide/playbooks_loops.rst @@ -376,6 +376,8 @@ You can specify the name of the variable for each loop using ``loop_var`` with ` .. note:: If Ansible detects that the current loop is using a variable which has already been defined, it will raise an error to fail the task. +Extended loop variables +----------------------- .. versionadded:: 2.8 As of Ansible 2.8 you can get extended loop information using the ``extended`` option to loop control. This option will expose the following information. @@ -400,6 +402,8 @@ Variable Description loop_control: extended: yes +Accessing the name of your loop_var +----------------------------------- .. versionadded:: 2.8 As of Ansible 2.8 you can get the name of the value provided to ``loop_control.loop_var`` using the ``ansible_loop_var`` variable diff --git a/docs/docsite/rst/user_guide/playbooks_tests.rst b/docs/docsite/rst/user_guide/playbooks_tests.rst index b52dd09a557539..5a2b430cd1870a 100644 --- a/docs/docsite/rst/user_guide/playbooks_tests.rst +++ b/docs/docsite/rst/user_guide/playbooks_tests.rst @@ -59,8 +59,7 @@ To match strings against a substring or a regular expression, use the "match", " msg: "matched pattern 4" when: url is regex("example.com/\w+/foo") -'match' requires zero or more characters at the beginning of the string, while 'search' only requires matching a subset of the string. - +'match' requires zero or more characters at the beginning of the string, while 'search' only requires matching a subset of the string. By default, 'regex' works like `search`, but `regex` can be configured to perform other tests as well. .. _testing_versions: @@ -199,7 +198,7 @@ The following tests can provide information about a path on the controller:: - debug: msg: "path is {{ (mypath is abs)|ternary('absolute','relative')}}" - - debug: + - debug: msg: "path is the same file as path2" when: mypath is same_file(path2) @@ -266,5 +265,3 @@ The following tasks are illustrative of the tests meant to check the status of t Have a question? Stop by the google group! `irc.freenode.net `_ #ansible IRC chat channel - - diff --git a/docs/docsite/rst/user_guide/playbooks_vault.rst b/docs/docsite/rst/user_guide/playbooks_vault.rst index 3aae832780407a..6d9d8aebc1c2ea 100644 --- a/docs/docsite/rst/user_guide/playbooks_vault.rst +++ b/docs/docsite/rst/user_guide/playbooks_vault.rst @@ -5,13 +5,30 @@ Using Vault in playbooks .. contents:: Topics -The "Vault" is a feature of Ansible that allows you to keep sensitive data such as passwords or keys in encrypted files, rather than as plaintext in playbooks or roles. These vault files can then be distributed or placed in source control. +The "Vault" is a feature of Ansible that allows you to keep sensitive data such as passwords or keys protected at rest, rather than as plaintext in playbooks or roles. These vaults can then be distributed or placed in source control. + +There are 2 types of vaulted content and each has their own uses and limitations: + +:Vaulted files: + * The full file is encrypted in the vault, this can contain Ansible variables or any other type of content. + * It will always be decrypted when loaded or referenced, Ansible cannot know if it needs the content unless it decrypts it. + * It can be used for inventory, anything that loads variables (i.e vars_files, group_vars, host_vars, include_vars, etc) + and some actions that deal with files (i.e M(copy), M(assemble), M(script), etc). + +:Single encrypted variable: + * Only specific variables are encrypted inside a normal 'variable file'. + * Does not work for other content, only variables. + * Decrypted on demand, so you can have vaulted variables with different vault secrets and only provide those needed. + * You can mix vaulted and non vaulted variables in the same file, even inline in a play or role. + +.. warning:: + * Vault ONLY protects data 'at rest', once decrypted play and plugin authors are responsible of avoiding any secrets discolsure, + see ``no_log`` for details on hiding output. To enable this feature, a command line tool, :ref:`ansible-vault` is used to edit files, and a command line flag :option:`--ask-vault-pass `, :option:`--vault-password-file ` or :option:`--vault-id ` is used. You can also modify your ``ansible.cfg`` file to specify the location of a password file or configure Ansible to always prompt for the password. These options require no command line flag usage. For best practices advice, refer to :ref:`best_practices_for_variables_and_vaults`. - Running a Playbook With Vault ````````````````````````````` diff --git a/docs/docsite/rst/user_guide/vault.rst b/docs/docsite/rst/user_guide/vault.rst index 5a977fe8cf1b11..08ed4a62525fda 100644 --- a/docs/docsite/rst/user_guide/vault.rst +++ b/docs/docsite/rst/user_guide/vault.rst @@ -16,7 +16,12 @@ For best practices advice, refer to :ref:`best_practices_for_variables_and_vault What Can Be Encrypted With Vault ```````````````````````````````` -Ansible Vault can encrypt any structured data file used by Ansible. This can include "group_vars/" or "host_vars/" inventory variables, variables loaded by "include_vars" or "vars_files", or variable files passed on the ansible-playbook command line with ``-e @file.yml`` or ``-e @file.json``. Role variables and defaults are also included. +File-level encryption +^^^^^^^^^^^^^^^^^^^^^ + +Ansible Vault can encrypt any structured data file used by Ansible. + +This can include "group_vars/" or "host_vars/" inventory variables, variables loaded by "include_vars" or "vars_files", or variable files passed on the ansible-playbook command line with ``-e @file.yml`` or ``-e @file.json``. Role variables and defaults are also included. Ansible tasks, handlers, and so on are also data so these can be encrypted with vault as well. To hide the names of variables that you're using, you can encrypt the task files in their entirety. @@ -26,7 +31,19 @@ given as the ``src`` argument to the :ref:`copy `, :ref:`template < ` modules, the file will be placed at the destination on the target host decrypted (assuming a valid vault password is supplied when running the play). -As of version 2.3, Ansible supports encrypting single values inside a YAML file, using the `!vault` tag to let YAML and Ansible know it uses special processing. This feature is covered in more details below. +.. note:: + The advantages of file-level encryption are that it is easy to use and that password rotation is straightforward with :ref:`rekeying `. + The drawback is that the contents of files are no longer easy to access and read. This may be problematic if it is a list of tasks (when encrypting a variables file, :ref:`best practice ` is to keep references to these variables in a non-encrypted file). + + +Variable-level encryption +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Ansible also supports encrypting single values inside a YAML file, using the `!vault` tag to let YAML and Ansible know it uses special processing. This feature is covered in more detail :ref:`below `. + +.. note:: + The advantage of variable-level encryption is that files are still easily legible even if they mix plaintext and encrypted variables. + The drawback is that password rotation is not as simple as with file-level encryption: the :ref:`rekey ` command does not work with this method. .. _vault_ids: @@ -34,23 +51,28 @@ As of version 2.3, Ansible supports encrypting single values inside a YAML file, Vault IDs and Multiple Vault Passwords `````````````````````````````````````` -*Available since Ansible 2.4* -A vault ID is an identifier for one or more vault secrets. Since Ansible 2.4, -Ansible supports multiple vault passwords. Vault IDs provide -labels for individual vault passwords. +A vault ID is an identifier for one or more vault secrets; +Ansible supports multiple vault passwords. -Vault-encrypted content can specify which vault ID it was encrypted with. +Vault IDs provide labels to distinguish between individual vault passwords. -Prior to Ansible 2.4, only one vault password could be used at a time, So any -vault files or vars that needed to be decrypted all had to use the same password. +To use vault IDs, you must provide an ID *label* of your choosing and a *source* to obtain its password (either ``prompt`` or a file path): -Since Ansible 2.4, vault files or vars that are encrypted with different -passwords can be used at the same time. +.. code-block:: bash + + --vault-id label@source + +This switch is available for all Ansible commands that can interact with vaults: :ref:`ansible-vault`, :ref:`ansible-playbook`, etc. + +Vault-encrypted content can specify which vault ID it was encrypted with. For example, a playbook can now include a vars file encrypted with a 'dev' vault ID and a 'prod' vault ID. +.. note: + Older versions of Ansible, before 2.4, only supported using one single vault password at a time. + .. _creating_files: @@ -67,6 +89,12 @@ First you will be prompted for a password. After providing a password, the tool The default cipher is AES (which is shared-secret based). +To create a new encrypted data file with the Vault ID 'password1' assigned to it and be prompted for the password, run: + +.. code-block:: bash + + ansible-vault create --vault-id password1@prompt foo.yml + .. _editing_encrypted_files: @@ -81,6 +109,12 @@ the file, saving it back when done and removing the temporary file: ansible-vault edit foo.yml +To edit a file encrypted with the 'vault2' password file and assigned the 'pass2' vault ID: + +.. code-block:: bash + + ansible-vault edit --vault-id pass2@vault2 foo.yml + .. _rekeying_files: @@ -96,6 +130,13 @@ Should you wish to change your password on a vault-encrypted file or files, you This command can rekey multiple data files at once and will ask for the original password and also the new password. +To rekey files encrypted with the 'preprod2' vault ID and the 'ppold' file and be prompted for the new password: + +.. code-block:: bash + + ansible-vault rekey --vault-id preprod2@ppold --new-vault-id preprod2@prompt foo.yml bar.yml baz.yml + +A different ID could have been set for the rekeyed files by passing it to ``--new-vault-id``. .. _encrypting_files: @@ -109,6 +150,18 @@ the :ref:`ansible-vault encrypt ` command. This command ansible-vault encrypt foo.yml bar.yml baz.yml +To encrypt existing files with the 'project' ID and be prompted for the password: + +.. code-block:: bash + + ansible-vault encrypt --vault-id project@prompt foo.yml bar.yml baz.yml + +.. note:: + + It is technically possible to separately encrypt files or strings with the *same* vault ID but *different* passwords, if different password files or prompted passwords are provided each time. + This could be desirable if you use vault IDs as references to classes of passwords (rather than a single password) and you always know which specific password or file to use in context. However this may be an unnecessarily complex use-case. + If two files are encrypted with the same vault ID but different passwords by accident, you can use the :ref:`rekey ` command to fix the issue. + .. _decrypting_files: diff --git a/docs/docsite/rst/user_guide/windows_setup.rst b/docs/docsite/rst/user_guide/windows_setup.rst index 13f2fb7158fb4e..2c6e9dcee9e780 100644 --- a/docs/docsite/rst/user_guide/windows_setup.rst +++ b/docs/docsite/rst/user_guide/windows_setup.rst @@ -101,6 +101,8 @@ The following PowerShell command will install the hotfix: (New-Object -TypeName System.Net.WebClient).DownloadFile($url, $file) powershell.exe -ExecutionPolicy ByPass -File $file -Verbose +For more details, please refer to the `Hotfix document `_ from Microsoft. + WinRM Setup ``````````` Once Powershell has been upgraded to at least version 3.0, the final step is for the @@ -381,7 +383,7 @@ target Windows host: winrs -r:http://server:5985/wsman -u:Username -p:Password ipconfig # Test out HTTPS (will fail if the cert is not verifiable) - winrs -r:http://server:5985/wsman -u:Username -p:Password -ssl ipconfig + winrs -r:https://server:5986/wsman -u:Username -p:Password -ssl ipconfig # Test out HTTPS, ignoring certificate verification $username = "Username"