Ansible 2.4 group_vars not parsed lexicographically

[Saloshp]

ISSUE TYPE

• Bug Report
• Feature Idea

COMPONENT NAME

group_vars

ANSIBLE VERSION

ansible 2.4.0.0
  config file = /opt/ansible/ansible.cfg
  configured module search path = [u'/usr/lib/pymodules/python2.7/ansible/module_utils']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.14 (default, Sep 20 2017, 01:25:59) [GCC 7.2.0]

CONFIGURATION

[root@local test]# ansible-config dump --only-changed
[root@local test]# 

OS / ENVIRONMENT

[root@local test]# uname -a
Linux local 4.13.4-1-ARCH #1 SMP PREEMPT Thu Sep 28 08:39:52 CEST 2017 x86_64 GNU/Linux

SUMMARY

group variable files of higher lexicographical order do not override lower ones

STEPS TO REPRODUCE

[root@local test]# for file in $(find -type f); do echo "#File: ${file}"; cat $file; echo; done
#File: ./playbooks/debug.yml
- hosts: localhost
  gather_facts: yes
  tasks:
  - debug: var=myvar

#File: ./playbooks/group_vars/all/02_vars
myvar: "02"

#File: ./playbooks/group_vars/all/00_vars
myvar: "00"

[root@local test]# ansible-playbook -i "127.0.0.1," -c local playbooks/debug.yml 

EXPECTED RESULTS

ok: [127.0.0.1] => {
"myvar": "02"
}

ACTUAL RESULTS

PLAY [localhost] *************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
ok: [127.0.0.1]

TASK [debug] *****************************************************************************************************************************************
ok: [127.0.0.1] => {
    "myvar": "00"
}

PLAY RECAP *******************************************************************************************************************************************
127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=0   

[NomAnor]

It seems the group_vars/host_vars files are found in the function _get_dir_files in plugins/vars/host_group_vars.py. This function uses os.listdir. According to the python documentation:

The list is in arbitrary order.

So one can't rely on any ordering. This behaviour should at least be documented.

I think it would be better to load the files in sorted order. This can be benificial for multi-stage environments to specify defaults for all environments as shown in the article How to Manage Multistage Environments with Ansible (The method shown is not reliable in the current version). Playbook group_vars can't be used beacuse they would override the inventory group_vars.

[bcoca]

this was never sorted, previous versions always used os.listdir, this would be a change in behaviour.

[Saloshp]

@bcoca, so your'e saying a random behaviour is better than a predicted one, simply because you got used to it?
Also, I didn't dive into the code but can't reproduce under Ansible 2.3 (works fine)

[bcoca]

@Saloshp its not random, it is just not determined by our code, there is a big difference there.

The code in 2.4 mostly 'moved' but was not changed when it came to finding files, you can check 2.3 as it also uses os.listdir and there is no sorting after that.

I'm not saying i'm rejecting the feature, I'm saying we need to add a toggle for those depending on the previous behaviour.

[ansibot]

cc @jhoekx
click here for bot help

[logan2211]

Actually, they were sorted, and 2.4 broke this behavior.

Here is the snippet from 2.3.1.0 which previously sorted the list of files:

ansible/lib/ansible/vars/__init__.py

Lines 559 to 566 in c7052b4

	try:
	names = loader.list_directory(path)
	except os.error as err:
	raise AnsibleError("This folder cannot be listed: %s: %s." % (path, err.strerror))
	# evaluate files in a stable order rather than whatever
	# order the filesystem lists them.
	names.sort()

This is a valid bug and the fix in the PR is valid imo. This regression is preventing a smooth upgrade to 2.4 in one of my projects currently.

[bcoca]

hmm, seems we used to do 1/2 the vars one way, the other 1/2 the other ... sorted does seem more predictable so I'll add it w/o requiring a toggle.