Fix variable precedence of INI/script vars to be in-line with docs.

This commit also adds a new test script (ansible-var-precedence-check.py in code-smell/)
to provide us with another line of defense against precedence bugs going forward.

The precedence docs state that the INI vars have a lower precedence than group/host
vars files for inventory and playbooks, however that has not been the case since 2.0
was released. This change fixes that in one way, though not exactly as the docs say.
The rules are:

1) INI/script < inventory dir < playbook dir
2) "all" group vars < other group_vars < host_vars

So the new order will be (from the test script mentioned above):

8. pb_host_vars_file - var in playbook/host_vars/host
9. ini_host_vars_file - var in inventory/host_vars/host
10. ini_host - host var inside the ini
11. pb_group_vars_file_child - var in playbook/group_vars/child
12. ini_group_vars_file_child - var in inventory/group_vars/child
13. pb_group_vars_file_parent - var in playbook/group_vars/parent
14. ini_group_vars_file_parent - var in inventory/group_vars/parent
15. pb_group_vars_file_all - var in playbook/group_vars/all
16. ini_group_vars_file_all - var in inventory/group_vars/all
17. ini_child - child group var inside the ini
18. ini_parent - parent group var inside the ini
19. ini_all - all group var inside the ini

Fixes #21845

Author: jimi-c

docs/docsite/rst/playbooks_variables.rst

@@ -830,20 +830,25 @@ In 1.x, the precedence is as follows (with the last listed variables winning pri
 In 2.x, we have made the order of precedence more specific (with the last listed variables winning prioritization):
 
   * role defaults [1]_
-  * inventory vars [2]_
-  * inventory group_vars
-  * inventory host_vars
-  * playbook group_vars
-  * playbook host_vars
+  * inventory INI or script group vars [2]_
+  * inventory group_vars/all
+  * playbook group_vars/all
+  * inventory group_vars/*
+  * playbook group_vars/*
+  * inventory INI or script host vars [2]_
+  * inventory host_vars/*
+  * playbook host_vars/*
   * host facts
   * play vars
   * play vars_prompt
   * play vars_files
-  * registered vars
-  * set_facts
-  * role and include vars
+  * role vars (defined in role/vars/main.yml)
   * block vars (only for tasks in block)
   * task vars (only for the task)
+  * role (and include_role) params
+  * include params
+  * include_vars
+  * set_facts / registered vars
   * extra vars (always win precedence)
 
 Basically, anything that goes into "role defaults" (the defaults folder inside the role) is the most malleable and easily overridden. Anything in the vars directory of the role overrides previous versions of that variable in namespace.  The idea here to follow is that the more explicit you get in scope, the more precedence it takes with command line ``-e`` extra vars always winning.  Host and/or inventory variables can win over role defaults, but not explicit includes like the vars directory or an ``include_vars`` task.

lib/ansible/vars/__init__.py

@@ -249,19 +249,17 @@ def get_vars(self, loader, play=None, host=None, task=None, include_hostvars=Tru
             all_vars = combine_vars(all_vars, task._role.get_default_vars(dep_chain=task.get_dep_chain()))
 
         if host:
-            # next, if a host is specified, we load any vars from group_vars
-            # files and then any vars from host_vars files which may apply to
-            # this host or the groups it belongs to
+            # first we merge in vars from groups specified in the inventory (INI or script)
+            all_vars = combine_vars(all_vars, host.get_group_vars())
 
-            # we merge in the special 'all' group_vars first, if they exist
+            # next, we load any vars from group_vars files and then any vars from host_vars
+            # files which may apply to this host or the groups it belongs to. We merge in the
+            # special 'all' group_vars first, if they exist
             if 'all' in self._group_vars_files:
                 data = preprocess_vars(self._group_vars_files['all'])
                 for item in data:
                     all_vars = combine_vars(all_vars, item)
 
-            # we merge in vars from groups specified in the inventory (INI or script)
-            all_vars = combine_vars(all_vars, host.get_group_vars())
-
             for group in sorted(host.get_groups(), key=lambda g: (g.depth, g.name)):
                 if group.name in self._group_vars_files and group.name != 'all':
                     for data in self._group_vars_files[group.name]: