Host order in groups variable is not 'as provided' when using yaml inventory

[YoninL]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• inventory

ANSIBLE VERSION

ansible 2.3.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]

OS / ENVIRONMENT

• Ubuntu 14.04 LTS

SUMMARY

The order of elements in hostvars[groups['group_name']] list doesn't follow the order in yaml format host inventory file.

Host file:

root@myci:~# cat dev
mysql:
    hosts:
        192.168.11.2:
        192.168.11.1:

Commands run against the host file:

root@myci:~# ansible -i dev localhost -m debug -a "var=hostvars[groups['mysql'][0]]"

EXPECTED RESULTS

192.168.11.1

ACTUAL RESULTS

192.168.11.2

STEPS TO REPRODUCE

Let's say we have two host files, host.yaml and host.ini.

host.yaml:

mysql:
    hosts:
        192.168.11.2:
        192.168.11.1:

host.ini:

[mysql]
192.168.11.2
192.168.11.1

I want to run ansible -i host.yaml localhost -m debug -a "var=hostvars[groups['mysql'][0]]" to get the first host, but got the second one.

localhost | SUCCESS => {
    "hostvars[groups['mysql'][0]]": {
        "ansible_check_mode": false,
        "ansible_playbook_python": "/usr/bin/python",
        "ansible_version": {
            "full": "2.3.1.0",
            "major": 2,
            "minor": 3,
            "revision": 1,
            "string": "2.3.1.0"
        },
        "group_names": [
            "mysql"
        ],
        "groups": {
            "all": [
                "192.168.11.1",
                "192.168.11.2"
            ],
            "mysql": [
                "192.168.11.1",
                "192.168.11.2"
            ],
            "ungrouped": [
                "localhost"
            ]
        },
        "inventory_dir": "/root",
        "inventory_file": "/root/dev",
        "inventory_hostname": "192.168.11.1",
        "inventory_hostname_short": "192",
        "omit": "__omit_place_holder__9cc33acdf1e4828707e5668d453dbe2a95a98243",
        "playbook_dir": "."
    }
}

While ansible -i host.ini localhost -m debug -a "var=hostvars[groups['mysql'][0]]" gives me correct result.

localhost | SUCCESS => {
    "hostvars[groups['mysql'][0]]": {
        "ansible_check_mode": false,
        "ansible_playbook_python": "/usr/bin/python",
        "ansible_version": {
            "full": "2.3.1.0",
            "major": 2,
            "minor": 3,
            "revision": 1,
            "string": "2.3.1.0"
        },
        "group_names": [
            "mysql"
        ],
        "groups": {
            "all": [
                "192.168.11.2",
                "192.168.11.1"
            ],
            "mysql": [
                "192.168.11.2",
                "192.168.11.1"
            ],
            "ungrouped": [
                "localhost"
            ]
        },
        "inventory_dir": "/root",
        "inventory_file": "/root/dev2",
        "inventory_hostname": "192.168.11.2",
        "inventory_hostname_short": "192",
        "omit": "__omit_place_holder__6c8e522f74475f8abdd8199cf2c270434b024b1a",
        "playbook_dir": "."
    }
}

You can see the order in host.ini matters, but in host.yaml not.

I need to use yaml format to store some variables which is encrypted by Ansible vault.

[ansibot]

@YoninL Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information.

Here are the items we could not find in your description:

• issue type
• ansible version
• component name

Please set the description of this issue with this template:
https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE.md

click here for bot help

[sivel]

I've update the title, to better reflect the issue. Based on what I have read in this issue, it appears as though the issue is that the host order in the groups variable is inconsistent when using an INI inventory.

When using a YAML inventory, the host order in the groups variable matches inventory order, but when using an INI inventory, the order may not match the inventory order.

[YoninL]

thanks @sivel , you may say the host order is not consistent in ini inventory, but I want the order in yaml inventory matters/'as provided' just like ini inventory, otherwise I don't know which one should be first, which one should be second before test.

[sivel]

Ok, that makes a little more sense then. I was having difficulty following your description of the problem.

Roughly, I know how this needs to be handled, or at minimum what the problem is. In python a dict is not ordered, as compared to a list which is ordered. So when loading any inventory source that represents the hosts as keys in a dictionary, it is likely that order will not be maintained, or at minimum we can say that the order is not guaranteed.

We could use something like collections.OrderedDict, however that was introduced in python2 version 2.7, and we have a compatibility need with py2.6.

I'm hesitant to optionally use collections.OrderedDict if available, because people sharing a playbook could have different outcomes based on whether that module is available.

There is a compatible OrderedDict for older python versions at https://github.com/ActiveState/code/tree/master/recipes/Python/576693_Ordered_Dictionary_for_Py24

But since we just need to go back to 2.6, it might be easier to modify and vendor what lives in 2.7, but we will have to evaluate this.

[willmerae]

First stab at yaml_ordered, a replacement inventory plugin which preserves the ordering of hosts in a group. Instructions are included for using it instead of Ansible's built in plugins. It's based on the yaml inventory plugin from Ansible 2.4.2.

[bcoca]

not a fix that will be going in but #36644 'fixes' this issue, a similar approach just for the yaml inventory plugin might be a solution

[macster84]

hi @sivel. As Ansible 2.7 will require Python 2.7+ on the controller machine, is it an option to revive your idea of using an OrderedDict?

[westsouthnight]

@bcoca please up this issue. We think order of the play hosts must to be by order in inventory without get some host for first to play from some host id in group. Sometimes ansible get host id 0 sometimes get for first to run last host in group, its not normal behaviour if deploy start on random host on group. We want have one stable behaviour for run.

[bcoca]

@westsouthnight it is being worked on, but the problem is not limited to the YAML plugin as restructuring of how groups are calculated to increase performance removed the ordering, so several cascading fixes have to take place.

As a workaround you can use the order: keyword to ensure reproducible sorting (of course, don't set it to 'inventory' which is the default).

[westsouthnight]

@bcoca Thanks! We really awaiting this!

[willmerae]

First stab at yaml_ordered, a replacement inventory plugin which preserves the ordering of hosts in a group.

Updated yaml_ordered to make it compatible with Ansible 2.8.1

[gvfnix]

The same issue here.
ansible 2.9.5 / molecule 3.0.0 / python 3.5

provisioner:
  name: ansible
  inventory:
    hosts:
      all:
        children:
          clickhouse:
            hosts:
              clickhouse1:
                ansible_host: clickhouse1
              clickhouse2:
                ansible_host: clickhouse2

When I try to use {% for host in groups['clickhouse'] %} ... {% endfor %} I end up with different order from time to time I run molecule converge.

I found a workaround for this. The first task in my ansible role is

- set_fact:
    clickhouse_hosts: "{{groups['clickhouse'] | sort}}"

Then I don't use groups['clickhouse'], I use clickhouse_hosts and this works for me.

[rony36]

This issue resolved on ansible 2.9.5 + python 3.6.9.

I have same issue on ansible 2.9.5 + python 3.5.2.
python 3.6 change dict order into "inserted ordered".

[bcoca]

@rony36 any Ansible version with python >=3.6 won't have the issue because Python changed to preserve order in dictionaries, lower versions of Python have either a 'hash order' or even a 'randomized order' to prevent predictability.

Going forward this will now work independent of Python versions via #58000.