Transition inventory into plugins (ansible#23001)

* draft new inventory plugin arch, yaml sample

 - split classes, moved out of init
 - extra debug statements
 - allow mulitple invenotry files
 - dont add hosts more than once
 - simplified host vars
 - since now we can have multiple, inventory_dir/file needs to be per host
 - ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
 - centralized localhost handling
 - added plugin docs
 - leaner meaner inventory (split to data + manager)
 - moved noop vars plugin
 - added 'postprocessing' inventory plugins
 - fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
 - grouphost_vars loading as inventory plugin (postprocessing)
 - playbook_dir allways full path
 - use bytes for file operations
 - better handling of empty/null sources
 - added test target that skips networking modules
 - now var manager loads play group/host_vars independant from inventory
 - centralized play setup repeat code
 - updated changelog with inv features
 - asperioribus verbis spatium album
 - fixed dataloader to new sig
 - made yaml plugin more resistant to bad data
 - nicer error msgs
 - fixed undeclared group detection
 - fixed 'ungrouping'
 - docs updated s/INI/file/ as its not only format
 - made behaviour of var merge a toggle
 - made 'source over group' path follow existing rule for var precedence
 - updated add_host/group from strategy
 - made host_list a plugin and added it to defaults
 - added advanced_host_list as example variation
 - refactored 'display' to be availbe by default in class inheritance
 - optimized implicit handling as per @pilou's feedback
 - removed unused code and tests
 - added inventory cache and vbox plugin now uses it
 - added _compose method for variable expressions in plugins
 - vbox plugin now uses 'compose'
 - require yaml extension for yaml
 - fix for plugin loader to always add original_path, even when not using all()
 - fix py3 issues
 - added --inventory as clearer option
 - return name when stringifying host objects
 - ajdust checks to code moving

* reworked vars and vars precedence
 - vars plugins now load group/host_vars dirs
 - precedence for host vars is now configurable
 - vars_plugins been reworked
 - removed unused vars cache
 - removed _gathered_facts as we are not keeping info in host anymore
 - cleaned up tests
 - fixed ansible-pull to work with new inventory
 - removed version added notation to please rst check
 - inventory in config relative to config
 - ensures full paths on passed inventories

* implicit localhost connection local

CHANGELOG.md

@@ -8,10 +8,18 @@ Ansible Changes By Release
 * Added fact namespacing, from now on facts will be available under 'ansible_facts' namespace (i.e. `ansible_facts.ansible_os_distribution`), they will still also be added into the main namespace directly but now also having a configuration toggle to disable this. Eventually this will be on by default. This is done to avoid collisions and possible security issues as facts come from the remote targets and they might be compromised.
 * new 'order' play level keyword that allows the user to change the order in which Ansible processes hosts when dispatching tasks.
 * Users can now set group merge priority for groups of the same depth (parent child relationship), using the new `ansible_group_priority` variable, when values are the same or don't exist it will fallback to the previous 'sorting by name'.
-* Support for Python-2.4 and Python-2.5 on the managed system's side was
-  dropped.  If you need to manage a system that ships with Python-2.4 or
-  Python-2.5 you'll need to install Python-2.6 or better there or run
-  Ansible-2.3 until you can upgrade the system.
+* Support for Python-2.4 and Python-2.5 on the managed system's side was dropped. If you need to manage a system that ships with Python-2.4 or Python-2.5, you'll need to install Python-2.6 or better on the managed system or run Ansible-2.3 until you can upgrade the system.
+* Inventory has been revamped:
+	- Inventory classes have been split to allow for better management and deduplication
+	- Logic that each inventory source duplicated is now common and pushed up to reconciliation
+	- VariableManager has been updated for better interaction with inventory
+	- Updated CLI with helper method to initialize base objects for plays
+	- Inventory plugins are a new type of plugin (can generate and/or update inventory)
+    - Old inventory formats are still supported via plugins
+    - The vars_plugins have been eliminated in favor of inventory_plugins #TODO: repurpose/implement inv plugin that loads vars ones?
+	- Loading group_vars/host_vars is now a plugin and can be overridden (for inventory)
+	- It is now possible to specify mulitple inventory sources in the command line (-i /etc/hosts1 -i /opt/hosts2)
+	- Inventory plugins can use the cache plugin (i.e. virtualbox) and is affected by `meta: refresh_inventory`
 
 ### Deprecations
 * The behaviour when specifying --tags (or --skip-tags) multiple times on the command line

Makefile

@@ -121,6 +121,9 @@ tests:
 tests-py3:
 	$(ANSIBLE_TEST) units -v --python $(PYTHON3_VERSION) $(TEST_FLAGS)
 
+tests-nonet:
+	$(ANSIBLE_TEST) units -v --python $(PYTHON_VERSION) $(TEST_FLAGS)  --exclude test/units/modules/network/
+
 integration:
 	$(ANSIBLE_TEST) integration -v --docker $(IMAGE) $(TARGET) $(TEST_FLAGS)
 
@@ -179,6 +182,7 @@ clean:
 	@echo "Cleaning up docsite"
 	$(MAKE) -C docs/docsite clean
 	$(MAKE) -C docs/api clean
+	find test/ -type f -name '*.retry' -delete
 
 python:
 	$(PYTHON) setup.py build

bin/ansible

@@ -37,9 +37,6 @@ import shutil
 import sys
 import traceback
 
-# for debug
-from multiprocessing import Lock
-
 import ansible.constants as C
 from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError
 from ansible.utils.display import Display

docs/docsite/rst/intro_inventory.rst

@@ -5,22 +5,22 @@ Inventory
 
 .. contents:: Topics
 
-Ansible works against multiple systems in your infrastructure at the
-same time.  It does this by selecting portions of systems listed in
-Ansible's inventory file, which defaults to being saved in
-the location ``/etc/ansible/hosts``. You can specify a different inventory file using the
-``-i <path>`` option on the command line.
-
-Not only is this inventory configurable, but you can also use
-multiple inventory files at the same time (explained below) and also
+Ansible works against multiple systems in your infrastructure at the same time.
+It does this by selecting portions of systems listed in Ansible's inventory,
+which defaults to being saved in the location ``/etc/ansible/hosts``.
+You can specify a different inventory file using the ``-i <path>`` option on the command line.
+
+Not only is this inventory configurable, but you can also use multiple inventory files at the same time and
 pull inventory from dynamic or cloud sources, as described in :doc:`intro_dynamic_inventory`.
+Introduced in version 2.4, Ansible has inventory plugins to make this flexible and customizable.
 
 .. _inventoryformat:
 
 Hosts and Groups
 ++++++++++++++++
 
-The format for ``/etc/ansible/hosts`` is an INI-like format and looks like this:
+The inventory file can be in one of many formats, depending on the inventory plugins you have.
+For this example, the format for ``/etc/ansible/hosts`` is an INI-like (one of Ansible's defaults) and looks like this::
 
 .. code-block:: ini
 
@@ -118,6 +118,8 @@ Variables can also be applied to an entire group at once::
    ntp_server=ntp.atlanta.example.com
    proxy=proxy.atlanta.example.com
 
+Be aware that this is only a convenient way to apply variables to multiple hosts at once; even though you can target hosts by group, variables are always flattened to the host level before a play is executed.
+
 .. _subgroups:
 
 Groups of Groups, and Group Variables
@@ -149,8 +151,11 @@ It is also possible to make groups of groups using the ``:children`` suffix. Jus
    southwest
    northwest
 
-If you need to store lists or hash data, or prefer to keep host and group specific variables
-separate from the inventory file, see the next section.
+If you need to store lists or hash data, or prefer to keep host and group specific variables separate from the inventory file, see the next section.
+Child groups have a couple of properties to note:
+
+ - First, any host that is member of a child group is automatically a member of the parent group.
+ - Second, a child group's variables will have higher precedence (override) a parent group's variables.
 
 .. _default_groups:
 
@@ -228,14 +233,18 @@ ansible_connection
 
 .. include:: ../rst_common/ansible_ssh_changes_note.rst
 
-SSH connection:
+General for all connections:
 
 ansible_host
     The name of the host to connect to, if different from the alias you wish to give to it.
 ansible_port
     The ssh port number, if not 22
 ansible_user
     The default ssh user name to use.
+
+
+Specific to the SSH connection:
+
 ansible_ssh_pass
     The ssh password to use (never store this variable in plain text; always use a vault. See :ref:`best_practices_for_variables_and_vaults`)
 ansible_ssh_private_key_file
@@ -252,10 +261,7 @@ ansible_ssh_extra_args
     This setting is always appended to the default :command:`ssh` command line.
 ansible_ssh_pipelining
     Determines whether or not to use SSH pipelining. This can override the ``pipelining`` setting in :file:`ansible.cfg`.
-
-.. versionadded:: 2.2
-
-ansible_ssh_executable
+ansible_ssh_executable (added in version 2.2)
     This setting overrides the default behavior to use the system :command:`ssh`. This can override the ``ssh_executable`` setting in :file:`ansible.cfg`.
 
 
@@ -295,7 +301,7 @@ ansible_shell_executable
     to use :command:`/bin/sh` (i.e. :command:`/bin/sh` is not installed on the target
     machine or cannot be run from sudo.).
 
-Examples from a host file::
+Examples from an Ansible-INI host file::
 
   some_host         ansible_port=2222     ansible_user=manager
   aws_host          ansible_ssh_private_key_file=/home/example/.ssh/aws.pem
@@ -360,3 +366,4 @@ Here is an example of how to instantly deploy to created containers::
        Questions? Help? Ideas?  Stop by the list on Google Groups
    `irc.freenode.net <http://irc.freenode.net>`_
        #ansible IRC chat channel
+

docs/docsite/rst/playbooks_variables.rst

@@ -835,12 +835,12 @@ In 1.x, the precedence is as follows (with the last listed variables winning pri
 In 2.x, we have made the order of precedence more specific (with the last listed variables winning prioritization):
 
   * role defaults [1]_
-  * inventory INI or script group vars [2]_
+  * inventory file or script group vars [2]_
   * inventory group_vars/all
   * playbook group_vars/all
   * inventory group_vars/*
   * playbook group_vars/*
-  * inventory INI or script host vars [2]_
+  * inventory file or script host vars [2]_
   * inventory host_vars/*
   * playbook host_vars/*
   * host facts

examples/ansible.cfg

@@ -60,11 +60,21 @@
 # uncomment this to disable SSH key host checking
 #host_key_checking = False
 
-# change the default callback
+# change the default callback, you can only have one 'stdout' type  enabled at a time.
 #stdout_callback = skippy
-# enable additional callbacks
+
+
+## Ansible ships with some plugins that require whitelisting,
+## this is done to avoid running all of a type by default.
+## These setting lists those that you want enabled for your system.
+## Custom plugins should not need this unless plugin author specifies it.
+
+# enable callback plugins, they can output to stdout but cannot be 'stdout' type.
 #callback_whitelist = timer, mail
 
+# enable inventory plugins, default: 'host_list', 'script', 'ini', 'yaml'
+#inventory_enabled = host_list, aws, openstack, docker
+
 # Determine whether includes in tasks and handlers are "static" by
 # default. As of 2.0, includes are dynamic by default. Setting these
 # values to True will make includes behave more like they did in the

lib/ansible/cli/__init__.py

@@ -35,9 +35,13 @@
 from ansible.release import __version__
 from ansible import constants as C
 from ansible.errors import AnsibleError, AnsibleOptionsError
-from ansible.module_utils.six import with_metaclass
+from ansible.inventory.manager import InventoryManager
+from ansible.module_utils.six import with_metaclass, string_types
 from ansible.module_utils._text import to_bytes, to_text
+from ansible.parsing.dataloader import DataLoader
 from ansible.utils.path import unfrackpath
+from ansible.utils.vars import load_extra_vars, load_options_vars
+from ansible.vars.manager import VariableManager
 
 try:
     from __main__ import display
@@ -49,8 +53,6 @@
 class SortedOptParser(optparse.OptionParser):
     '''Optparser which sorts the options by opt before outputting --help'''
 
-    #FIXME: epilog parsing: OptionParser.format_epilog = lambda self, formatter: self.epilog
-
     def format_help(self, formatter=None, epilog=None):
         self.option_list.sort(key=operator.methodcaller('get_opt_string'))
         return optparse.OptionParser.format_help(self, formatter=None)
@@ -294,9 +296,8 @@ def base_parser(usage="", output_opts=False, runas_opts=False, meta_opts=False,
             help="verbose mode (-vvv for more, -vvvv to enable connection debugging)")
 
         if inventory_opts:
-            parser.add_option('-i', '--inventory-file', dest='inventory',
-                help="specify inventory host path (default=%s) or comma separated host list." % C.DEFAULT_HOST_LIST,
-                default=C.DEFAULT_HOST_LIST, action="callback", callback=CLI.expand_tilde, type=str)
+            parser.add_option('-i', '--inventory', '--inventory-file', dest='inventory', action="append",
+                help="specify inventory host path (default=[%s]) or comma separated host list. --inventory-file is deprecated" % C.DEFAULT_HOST_LIST)
             parser.add_option('--list-hosts', dest='listhosts', action='store_true',
                 help='outputs a list of matching hosts; does not execute anything else')
             parser.add_option('-l', '--limit', default=C.DEFAULT_SUBSET, dest='subset',
@@ -318,12 +319,12 @@ def base_parser(usage="", output_opts=False, runas_opts=False, meta_opts=False,
             parser.add_option('--ask-vault-pass', default=C.DEFAULT_ASK_VAULT_PASS, dest='ask_vault_pass', action='store_true',
                 help='ask for vault password')
             parser.add_option('--vault-password-file', default=C.DEFAULT_VAULT_PASSWORD_FILE, dest='vault_password_file',
-                help="vault password file", action="callback", callback=CLI.expand_tilde, type=str)
+                help="vault password file", action="callback", callback=CLI.expand_tilde, type='string')
             parser.add_option('--new-vault-password-file', dest='new_vault_password_file',
-                help="new vault password file for rekey", action="callback", callback=CLI.expand_tilde, type=str)
+                help="new vault password file for rekey", action="callback", callback=CLI.expand_tilde, type='string')
             parser.add_option('--output', default=None, dest='output_file',
                 help='output file name for encrypt or decrypt; use - for stdout',
-                action="callback", callback=CLI.expand_tilde, type=str)
+                action="callback", callback=CLI.expand_tilde, type='string')
 
         if subset_opts:
             parser.add_option('-t', '--tags', dest='tags', default=[], action='append',
@@ -342,8 +343,7 @@ def base_parser(usage="", output_opts=False, runas_opts=False, meta_opts=False,
             connect_group.add_option('-k', '--ask-pass', default=C.DEFAULT_ASK_PASS, dest='ask_pass', action='store_true',
                 help='ask for connection password')
             connect_group.add_option('--private-key','--key-file', default=C.DEFAULT_PRIVATE_KEY_FILE, dest='private_key_file',
-                help='use this file to authenticate the connection',
-                action="callback", callback=CLI.unfrack_path, type=str)
+                help='use this file to authenticate the connection', action="callback", callback=CLI.unfrack_path, type='string')
             connect_group.add_option('-u', '--user', default=C.DEFAULT_REMOTE_USER, dest='remote_user',
                 help='connect as this user (default=%s)' % C.DEFAULT_REMOTE_USER)
             connect_group.add_option('-c', '--connection', dest='connection', default=C.DEFAULT_TRANSPORT,
@@ -439,7 +439,10 @@ def parse(self):
                 # If some additional transformations are needed for the
                 # arguments and options, do it here.
         """
+
         self.options, self.args = self.parser.parse_args(self.args[1:])
+
+        # process tags
         if hasattr(self.options, 'tags') and not self.options.tags:
             # optparse defaults does not do what's expected
             self.options.tags = ['all']
@@ -457,6 +460,7 @@ def parse(self):
                     tags.add(tag.strip())
             self.options.tags = list(tags)
 
+        # process skip_tags
         if hasattr(self.options, 'skip_tags') and self.options.skip_tags:
             if not C.MERGE_MULTIPLE_CLI_TAGS:
                 if len(self.options.skip_tags) > 1:
@@ -471,6 +475,23 @@ def parse(self):
                     skip_tags.add(tag.strip())
             self.options.skip_tags = list(skip_tags)
 
+        # process inventory options
+        if hasattr(self.options, 'inventory'):
+
+            if self.options.inventory:
+
+                # should always be list
+                if isinstance(self.options.inventory, string_types):
+                    self.options.inventory = [self.options.inventory]
+
+                # Ensure full paths when needed
+                self.options.inventory = [unfrackpath(opt) if ',' not in opt else opt for opt in self.options.inventory]
+
+            else:
+                # set default if it exists
+                if os.path.exists(C.DEFAULT_HOST_LIST):
+                    self.options.inventory = [ C.DEFAULT_HOST_LIST ]
+
     @staticmethod
     def version(prog):
         ''' return ansible version '''
@@ -654,18 +675,33 @@ def read_vault_password_file(vault_password_file, loader):
 
         return vault_pass
 
-    def get_opt(self, k, defval=""):
-        """
-        Returns an option from an Optparse values instance.
-        """
-        try:
-            data = getattr(self.options, k)
-        except:
-            return defval
-        # FIXME: Can this be removed if cli and/or constants ensures it's a
-        # list?
-        if k == "roles_path":
-            if os.pathsep in data:
-                data = data.split(os.pathsep)[0]
-        return data
+    @staticmethod
+    def _play_prereqs(options):
+
+        # all needs loader
+        loader = DataLoader()
+
+        # vault
+        b_vault_pass = None
+        if options.vault_password_file:
+            # read vault_pass from a file
+            b_vault_pass = CLI.read_vault_password_file(options.vault_password_file, loader=loader)
+        elif options.ask_vault_pass:
+            b_vault_pass = CLI.ask_vault_passwords()
+
+        if b_vault_pass is not None:
+            loader.set_vault_password(b_vault_pass)
+
+        # create the inventory, and filter it based on the subset specified (if any)
+        inventory = InventoryManager(loader=loader, sources=options.inventory)
+
+        # create the variable manager, which will be shared throughout
+        # the code, ensuring a consistent view of global variables
+        variable_manager = VariableManager(loader=loader, inventory=inventory)
+
+        # load vars from cli options
+        variable_manager.extra_vars = load_extra_vars(loader=loader, options=options)
+        variable_manager.options_vars = load_options_vars(options, CLI.version_info(gitinfo=False))
+
+        return loader, inventory, variable_manager
 

lib/ansible/cli/adhoc.py

@@ -26,15 +26,10 @@
 from ansible.cli import CLI
 from ansible.errors import AnsibleError, AnsibleOptionsError
 from ansible.executor.task_queue_manager import TaskQueueManager
-from ansible.inventory import Inventory
 from ansible.module_utils._text import to_text
-from ansible.parsing.dataloader import DataLoader
 from ansible.parsing.splitter import parse_kv
 from ansible.playbook.play import Play
 from ansible.plugins import get_all_plugin_loaders
-from ansible.utils.vars import load_extra_vars
-from ansible.utils.vars import load_options_vars
-from ansible.vars import VariableManager
 
 try:
     from __main__ import display
@@ -105,29 +100,12 @@ def run(self):
 
         sshpass    = None
         becomepass = None
-        b_vault_pass = None
 
         self.normalize_become_options()
         (sshpass, becomepass) = self.ask_passwords()
         passwords = { 'conn_pass': sshpass, 'become_pass': becomepass }
 
-        loader = DataLoader()
-
-        if self.options.vault_password_file:
-            # read vault_pass from a file
-            b_vault_pass = CLI.read_vault_password_file(self.options.vault_password_file, loader=loader)
-            loader.set_vault_password(b_vault_pass)
-        elif self.options.ask_vault_pass:
-            b_vault_pass = self.ask_vault_passwords()
-            loader.set_vault_password(b_vault_pass)
-
-        variable_manager = VariableManager()
-        variable_manager.extra_vars = load_extra_vars(loader=loader, options=self.options)
-
-        variable_manager.options_vars = load_options_vars(self.options)
-
-        inventory = Inventory(loader=loader, variable_manager=variable_manager, host_list=self.options.inventory)
-        variable_manager.set_inventory(inventory)
+        loader, inventory, variable_manager = self._play_prereqs(self.options)
 
         no_hosts = False
         if len(inventory.list_hosts()) == 0: