Update info on python support (#38855)

* Update the documentation to list Python 3 as official
* Add some reference targets for inventory variables so we can link to docs
* Add a platform FAQ section
  Populate it with

  * virtualenv info (previously on the python3 support page)
  * BSD (Link to the working with BSD page)
  * Solaris (Document how to work around the non-POSIX shell on some
    Solaris hosts)

  Fixes #21594

* Fix some refs in the release_and_maintenance document

* Fix unindent error in module template

Fix for the module/plugin template unintentionally unindented inside of
a raw block, leading to errors like:

ERROR: docs/docsite/rst/modules/redshift_facts_module.rst:289:0: Explicit markup ends without a blank line; unexpected unindent.

* Make wording for Solaris troubleshooting better.

docs/docsite/rst/installation_guide/intro_installation.rst

@@ -43,9 +43,6 @@ Control Machine Requirements
 
 Currently Ansible can be run from any machine with Python 2 (versions 2.6 or 2.7) or Python 3 (versions 3.5 and higher) installed (Windows isn't supported for the control machine).
 
-.. note::
-  Ansible 2.2 introduces a tech preview of support for Python 3 (versions 3.5 and higher). For more information, see `Python 3 Support <http://docs.ansible.com/ansible/python_3_support.html>`_.
-
 This includes Red Hat, Debian, CentOS, OS X, any of the BSDs, and so on.
 
 .. note::
@@ -64,40 +61,36 @@ Managed Node Requirements
 
 On the managed nodes, you need a way to communicate, which is normally ssh. By
 default this uses sftp. If that's not available, you can switch to scp in
-:file:`ansible.cfg`.  You also need Python 2.6 or later.
+:file:`ansible.cfg`.  You also need Python 2 (version 2.6 or later) or Python 3 (version 3.5 or
+later).
 
 .. note::
 
-   Ansible's "raw" module (for executing commands in a quick and dirty
-   way) and the script module don't even need that.  So technically, you can use
-   Ansible to install python-simplejson using the raw module, which
-   then allows you to use everything else.  (That's jumping ahead
-   though.)
-
-.. note::
+   * If you have SELinux enabled on remote nodes, you will also want to install
+     libselinux-python on them before using any copy/file/template related functions in Ansible. You
+     can use the :ref:`yum` or :ref`dnf` module in Ansible to install this package on remote systems
+     that do not have it.
 
-   If you have SELinux enabled on remote nodes, you will also want to install
-   libselinux-python on them before using any copy/file/template related functions in
-   Ansible. You can of course still use the yum module in Ansible to install this package on
-   remote systems that do not have it.
+   * By default, Ansible uses the python interpreter located at :file:`/usr/bin/python` to run its
+     modules.  However, some Linux distributions may only have a Python 3 interpreter installed to
+     :file:`/usr/bin/python3` by default.  On those systems, you may see an error like::
 
-.. note::
+        "module_stdout": "/bin/sh: /usr/bin/python: No such file or directory\r\n"
 
-   Ansible 2.2 introduces a tech preview of support for Python 3. For more information, see `Python 3 Support <http://docs.ansible.com/ansible/python_3_support.html>`_.
+     you can either set the :ref:`ansible_python_interpreter` inventory variable (see
+     :ref:`inventory`) to point at your interpreter or you can install a Python 2 interpreter for
+     modules to use. You will still need to set :ref:`ansible_python_interpreter` if the Python
+     2 interpreter is not installed to :command:`/usr/bin/python`.
 
-   By default, Ansible uses Python 2 in order to maintain compatibility with older distributions
-   such as RHEL 6. However, some Linux distributions (Gentoo, Arch) may not have a
-   Python 2.X interpreter installed by default.  On those systems, you should install one, and set
-   the ``ansible_python_interpreter`` variable in inventory (see :ref:`inventory`) to point at your 2.X Python.  Distributions
-   like Red Hat Enterprise Linux, CentOS, Fedora, and Ubuntu all have a 2.X interpreter installed
-   by default and this does not apply to those distributions.  This is also true of nearly all
-   Unix systems.
+   * Ansible's "raw" module (for executing commands in a quick and dirty way) and the script module
+     don't even need Python installed.  So technically, you can use Ansible to install a compatible
+     version of Python using the :ref:`raw` module, which then allows you to use everything else.
+     For example, if you need to bootstrap Python 2 onto a RHEL-based system, you can install it
+     via
 
+     .. code-block:: shell
 
-   If you need to bootstrap these remote systems by installing Python 2.X,
-   using the 'raw' module will be able to do it remotely. For example,
-   ``ansible myhost --sudo -m raw -a "yum install -y python2 python-simplejson"``
-   would install Python 2.X and the simplejson module needed to run ansible and its modules.
+        ansible myhost --sudo -m raw -a "yum install -y python2"
 
 .. _installing_the_control_machine:
 
@@ -204,8 +197,9 @@ To install the newest version, you may need to unmask the ansible package prior
 
 .. note::
 
-   If you have Python 3 as a default Python slot on your Gentoo nodes (default setting), then you
-   must set ``ansible_python_interpreter = /usr/bin/python2`` in your group or inventory variables.
+    The current default Python slot on Gentoo is version 3.4.  Ansible needs Python-3.5 or higher so
+    you will need to `:ref:`bootstrap <managed_node_requirements>` a compatible version onto the
+    machines.
 
 Latest Releases Via pkg (FreeBSD)
 +++++++++++++++++++++++++++++++++
@@ -254,11 +248,6 @@ The AUR has a PKGBUILD for pulling directly from Github called `ansible-git <htt
 
 Also see the `Ansible <https://wiki.archlinux.org/index.php/Ansible>`_ page on the ArchWiki.
 
-.. note::
-
-   If you have Python 3 as a default Python slot on your Arch nodes (default setting), then you
-   must set ``ansible_python_interpreter = /usr/bin/python2`` in your group or inventory variables.
-
 .. _from_pip:
 
 Latest Releases Via Pip

docs/docsite/rst/reference_appendices/faq.rst

@@ -105,26 +105,104 @@ and run Ansible from there.
 
 .. _python_interpreters:
 
-How do I handle python pathing not having a Python 2.X in /usr/bin/python on a remote machine?
+How do I handle python not having a Python interpreter at /usr/bin/python on a remote machine?
 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
-While you can write ansible modules in any language, most ansible modules are written in Python, and some of these
-are important core ones.
+While you can write Ansible modules in any language, most Ansible modules are written in Python,
+including the ones central to letting Ansible work.
 
-By default, Ansible assumes it can find a /usr/bin/python on your remote system that is a 2.X version of Python, specifically
-2.6 or higher.
+By default, Ansible assumes it can find a :command:`/usr/bin/python` on your remote system that is
+either Python2, version 2.6 or higher or Python3, 3.5 or higher.
 
-Setting the inventory variable 'ansible_python_interpreter' on any host will allow Ansible to auto-replace the interpreter
-used when executing python modules.   Thus, you can point to any python you want on the system if /usr/bin/python on your
-system does not point to a Python 2.X interpreter.
+Setting the inventory variable ``ansible_python_interpreter`` on any host will tell Ansible to
+auto-replace the Python interpreter with that value instead. Thus, you can point to any Python you
+want on the system if :command:`/usr/bin/python` on your system does not point to a compatible
+Python interpreter.
 
-Some Linux operating systems, such as Arch, may only have Python 3 installed by default.  This is not sufficient and you will
-get syntax errors trying to run modules with Python 3.  Python 3 is essentially not the same language as Python 2.  Python 3
-support is being worked on but some Ansible modules are not yet ported to run under Python 3.0.  This is not a problem though
-as you can just install Python 2 also on a managed host.
+Some platforms may only have Python 3 installed by default. If it is not installed as
+:command:`/usr/bin/python`, you will need to configure the path to the interpreter via
+``ansible_python_interpreter``. Although most core modules will work with Python 3, there may be some
+special purpose ones which do not or you may encounter a bug in an edge case. As a temporary
+workaround you can install Python 2 on the managed host and configure Ansible to use that Python via
+``ansible_python_interpreter``. If there's no mention in the module's documentation that the module
+requires Python 2, you can also report a bug on our `bug tracker
+<https://github.com/ansible/ansible/issues>`_ so that the incompatibility can be fixed in a future release.
 
 Do not replace the shebang lines of your python modules.  Ansible will do this for you automatically at deploy time.
 
+Common Platform Issues
+++++++++++++++++++++++
+
+Running in a virtualenv
+-----------------------
+
+You can install Ansible into a virtualenv on the controller quite simply:
+
+.. code-block:: shell
+
+    $ virtualenv ansible
+    $ source ./ansible/bin/activate
+    $ pip install ansible
+
+If you want to run under Python 3 instead of Python 2 you may want to change that slightly:
+
+.. code-block:: shell
+
+    $ virtualenv ansible
+    $ source ./ansible/bin/activate
+    $ pip3 install ansible
+
+If you need to use any libraries which are not available via pip (for instance, SELinux Python
+bindings on systems such as Red Hat Enterprise Linux or Fedora that have SELinux enabled) then you
+need to install them into the virtualenv.  There are two methods:
+
+* When you create the virtualenv, specify ``--system-site-packages`` to make use of any libraries
+  installed in the system's Python:
+
+  .. code-block:: shell
+
+      $ virtualenv ansible --system-site-packages
+
+* Copy those files in manually from the system.  For instance, for SELinux bindings you might do:
+
+  .. code-block:: shell
+
+      $ virtualenv ansible --system-site-packages
+      $ cp -r -v /usr/lib64/python3.*/site-packages/selinux/ ./py3-ansible/lib64/python3.*/site-packages/
+      $ cp -v /usr/lib64/python3.*/site-packages/*selinux*.so ./py3-ansible/lib64/python3.*/site-packages/
+
+
+Running on BSD
+--------------
+
+.. seealso:: :ref:`working_with_bsd`
+
+
+Running on Solaris
+------------------
+
+By default, Solaris 10 and earlier run a non-POSIX shell which does not correctly expand the default
+tmp directory Ansible uses ( :file:`~/.ansible/tmp`). If you see module failures on Solaris machines, this
+is likely the problem. There are several workarounds:
+
+* You can set :ref:`remote_tmp` to a path that will expand correctly with the Solaris shell.  For
+  example, in the ansible config file you can set::
+
+    remote_tmp=$HOME/.ansible/tmp
+
+  In Ansible 2.5 and later, you can also set it per-host like this::
+
+    solaris1 ansible_remote_tmp=$HOME/.ansible/tmp
+
+* You can set :ref:`ansible_shell_executable` to the path to a POSIX compatible shell.  For
+  instance, many Solaris hosts have a POSIX shell located at :file:`/usr/xpg4/bin/sh` so you can set
+  this in inventory like so::
+
+    solaris1 ansible_shell_executable=/usr/xpg4/bin/sh
+
+  (bash, ksh, and zsh should also be POSIX compatible if you have any of those installed).
+
+
 .. _use_roles:
 
 What is the best way to make content reusable/redistributable?

docs/docsite/rst/reference_appendices/python_3_support.rst

@@ -2,45 +2,32 @@
 Python 3 Support
 ================
 
-Ansible 2.2, 2.3, and 2.4 feature a tech preview of Python 3 support. This topic discusses how you can test to make sure your modules and playbooks work with Python 3.
+Ansible 2.5 and above have support for Python 3.  Previous to 2.5, the Python 3 support was
+considered a tech preview.  This topic discusses how to setup your controller and managed machines
+to use Python 3.
 
 .. note:: Ansible supports Python version 3.5 and above only.
 
+On the controller side
+----------------------
 
-.. note:: Technology preview features provide early access to upcoming product innovations,
-   enabling you to test functionality and provide feedback during the development process.
-   Please be aware that tech preview features may not be functionally complete and are not
-   intended for production use. To report a Python 3 bug, please see `Community Information & Contributing <http://docs.ansible.com/ansible/community.html#i-d-like-to-report-a-bug>`_.
-
-Testing Python 3 with commands and playbooks
---------------------------------------------
-
-* Run Ansible 2.2+ - See :ref:`from_source`
-* To test Python 3 on the controller, run your ansible command via
-  ``python3``. For example:
-
-.. code-block:: shell
-
-    $ python3 /usr/bin/ansible localhost -m ping
-    $ python3 /usr/bin/ansible-playbook sample-playbook.yml
-
-You can also install Ansible using :program:`pip` for Python3 which will make the default
-:command:`/usr/bin/ansible` run with Python3:
+The easiest way to run :command:`/usr/bin/ansible` under Python 3 is to install it with the Python3
+version of pip.  This will make the default :command:`/usr/bin/ansible` run with Python3:
 
 .. code-block:: shell
 
-    $ virtualenv --python=python3 py3-ansible
-    $ source ./py3-ansible/bin/activate
     $ pip3 install ansible
     $ ansible --version | grep "python version"
       python version = 3.6.2 (default, Sep 22 2017, 08:28:09) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)]
 
-On systems with SELinux installed, such as Red Hat Enterprise Linux or Fedora, the SELinux Python libraries also need to be copied over.
+If you are running Ansible :ref:`from_source` and want to use Python 3 with your source checkout, run your
+command via ``python3``.  For example:
 
 .. code-block:: shell
 
-    $ cp -r -v /usr/lib64/python3.*/site-packages/selinux/ ./py3-ansible/lib64/python3.*/site-packages/
-    $ cp -v /usr/lib64/python3.*/site-packages/*selinux*.so ./py3-ansible/lib64/python3.*/site-packages/
+    $ source ./hacking/env-setup
+    $ python3 $(which ansible) localhost -m ping
+    $ python3 $(which ansible-playbook) sample-playbook.yml
 
 .. note:: Individual Linux distribution packages may be packaged for Python2 or Python3.  When running from
     distro packages you'll only be able to use Ansible with the Python version for which it was
@@ -49,23 +36,27 @@ On systems with SELinux installed, such as Red Hat Enterprise Linux or Fedora, t
     with your distro to see if that applies in your case.
 
 
-Testing Python 3 module support
---------------------------------
+Using Python 3 on the managed machines with commands and playbooks
+------------------------------------------------------------------
 
-* Set the ansible_python_interpreter configuration option to
-  :command:`/usr/bin/python3`. The ``ansible_python_interpreter`` configuration option is
-  usually set per-host as an inventory variable associated with a host or group of hosts:
+* Set the ``ansible_python_interpreter`` configuration option to :command:`/usr/bin/python3`. The
+  ``ansible_python_interpreter`` configuration option is usually set as an inventory
+  variable associated with a host or group of hosts:
 
 .. code-block:: ini
 
-    # Example inventory that makes an alias for localhost that uses python3
+    # Example inventory that makes an alias for localhost that uses Python3
+    localhost-py3 ansible_host=localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python3
+
+    # Example of setting a group of hosts to use Python3
     [py3-hosts]
-    localhost-py3 ansible_host=localhost ansible_connection=local
+    ubuntu16
+    fedora27
 
     [py3-hosts:vars]
     ansible_python_interpreter=/usr/bin/python3
 
-See :ref:`intro_inventory` for more information.
+.. seealso:: :ref:`intro_inventory` for more information.
 
 * Run your command or playbook:
 
@@ -76,7 +67,8 @@ See :ref:`intro_inventory` for more information.
 
 
 Note that you can also use the `-e` command line option to manually
-set the python interpreter when you run a command. For example:
+set the python interpreter when you run a command.   This can be useful if you want to test whether
+a specific module or playbook has any bugs under Python 3.  For example:
 
 .. code-block:: shell
 
@@ -86,10 +78,14 @@ set the python interpreter when you run a command. For example:
 What to do if an incompatibility is found
 -----------------------------------------
 
-If you find a bug while testing modules with Python3 you can submit a bug
-report on `Ansible's GitHub project
-<https://github.com/ansible/ansible/issues/>`_.  Be sure to mention Python3 in
-the bug report so that the right people look at it.
+We have spent several releases squashing bugs and adding new tests so that Ansible's core feature
+set runs under both Python 2 and Python 3.  However, bugs may still exist in edge cases and many of
+the modules shipped with Ansible are maintained by the community and not all of those may be ported
+yet.
+
+If you find a bug running under Python 3 you can submit a bug report on `Ansible's GitHub project
+<https://github.com/ansible/ansible/issues/>`_.  Be sure to mention Python3 in the bug report so
+that the right people look at it.
 
 If you would like to fix the code and submit a pull request on github, you can
 refer to :ref:`developing_python3` for information on how we fix

docs/docsite/rst/reference_appendices/release_and_maintenance.rst

@@ -16,8 +16,8 @@ This cycle can be extended in order to allow for larger changes to be properly
 implemented and tested before a new release is made available.
 
 Ansible has a graduated support structure that extends to three major releases.
-For more information, read about the :ref: `_development_and_stable_versiona_maintenance_workflow` or see
-the chart in :ref:`schedule` for the degrees to which current releases are supported.
+For more information, read about the :ref:`development_and_stable_version_maintenance_workflow` or
+see the chart in :ref:`release_schedule` for the degrees to which current releases are supported.
 
 .. note:: Support for three major releases began with Ansible-2.4. Ansible-2.3 and older versions
     are only supported for two releases.
@@ -70,7 +70,7 @@ devel             `2.6` (unreleased, trunk)    In development
 .. _support_life:
 .. _methods:
 
-.. _development_and_stable_versiona_maintenance_workflow:
+.. _development_and_stable_version_maintenance_workflow:
 
 Development and stable version maintenance workflow
 ```````````````````````````````````````````````````

docs/docsite/rst/user_guide/intro_inventory.rst

@@ -389,21 +389,28 @@ ansible_become_flags
 Remote host environment parameters:
 
 ansible_shell_type
-    The shell type of the target system. You should not use this setting unless you have set the ``ansible_shell_executable`` to a non-Bourne (sh) compatible shell.
-    By default commands are formatted using ``sh``-style syntax.
-    Setting this to ``csh`` or ``fish`` will cause commands executed on target systems to follow those shell's syntax instead.
+    The shell type of the target system. You should not use this setting unless you have set the
+    :ref:`ansible_shell_executable` to a non-Bourne (sh) compatible shell.  By default commands are
+    formatted using ``sh``-style syntax.  Setting this to ``csh`` or ``fish`` will cause commands
+    executed on target systems to follow those shell's syntax instead.
+
+.. _ansible_python_interpreter:
+
 ansible_python_interpreter
     The target host python path. This is useful for systems with more
     than one Python or not located at :command:`/usr/bin/python` such as \*BSD, or where :command:`/usr/bin/python`
     is not a 2.X series Python.  We do not use the :command:`/usr/bin/env` mechanism as that requires the remote user's
     path to be set right and also assumes the :program:`python` executable is named python, where the executable might
     be named something like :program:`python2.6`.
+
 ansible_*_interpreter
-    Works for anything such as ruby or perl and works just like ``ansible_python_interpreter``.
+    Works for anything such as ruby or perl and works just like :ref:`ansible_python_interpreter`.
     This replaces shebang of modules which will run on that host.
 
 .. versionadded:: 2.1
 
+.. _ansible_shell_executable:
+
 ansible_shell_executable
     This sets the shell the ansible controller will use on the target machine,
     overrides ``executable`` in :file:`ansible.cfg` which defaults to