Hash variables fail to be loaded with include_vars in a role after a group_by #70598

jean-christophe-manciot · 2020-07-13T13:59:40Z

SUMMARY

When include_vars is used in a role after a group_by, complex variables fail to load whereas simple ones can be loaded.
This issue happens:

only with 2.9.10
not with previous ansible releases
on Debian bullseye
on Ubuntu 20.04
...

ISSUE TYPE

Bug report

COMPONENT NAME

include_vars

ANSIBLE VERSION

ansible 2.9.10
  config file = /etc/ansible/ansible.cfg
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.4rc1 (default, Jul  1 2020, 15:31:45) [GCC 9.3.0]

CONFIGURATION

ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
CACHE_PLUGIN(/etc/ansible/ansible.cfg) = redis
CACHE_PLUGIN_TIMEOUT(/etc/ansible/ansible.cfg) = 3600
DEFAULT_EXECUTABLE(/etc/ansible/ansible.cfg) = /bin/bash
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 1000
DEFAULT_GATHERING(/etc/ansible/ansible.cfg) = explicit
DEFAULT_GATHER_TIMEOUT(/etc/ansible/ansible.cfg) = 30
DEFAULT_HASH_BEHAVIOUR(/etc/ansible/ansible.cfg) = merge
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/etc/ansible/hosts']
DEFAULT_LOG_PATH(/etc/ansible/ansible.cfg) = /var/log/ansible.log
DEFAULT_PRIVATE_ROLE_VARS(/etc/ansible/ansible.cfg) = False
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 180
DEFAULT_TRANSPORT(/etc/ansible/ansible.cfg) = ssh
ENABLE_TASK_DEBUGGER(/etc/ansible/ansible.cfg) = True
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = /usr/bin/python3
PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible.cfg) = 3599
PERSISTENT_CONNECT_RETRY_TIMEOUT(/etc/ansible/ansible.cfg) = 200
PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible.cfg) = 3600
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
SHOW_CUSTOM_STATS(/etc/ansible/ansible.cfg) = True

OS / ENVIRONMENT

controller host: Debian bullseye

STEPS TO REPRODUCE

playbook: include_vars issue.yml

---
- name: Classifying all hosts
  hosts: all
  gather_facts: False
  tasks:
        - set_fact:
                os_family: os
        - set_fact:
                os_version: version
                
        - name: Classifying all hosts depending on their OS & release
          include_role:
                name: os_classify_issue

roles/os_classify_issue/tasks/main.yml

---
- name: Classifying the network host depending on its OS & release & loading the corresponding group variables
  group_by:
        key: "{{ os_family }}_{{ os_version }}"

- name: Including ansible connection variables
  block:
        - include_vars: "{{ role_path }}/vars/all/ssh/ansible_connections.with.private_key_file.yml"
          when: connections.ssh.private_key_file is defined

        - include_vars: "{{ role_path }}/vars/all/ssh/ansible_connections.wo.private_key_file.yml"
          when: connections.ssh.private_key_file is not defined

- debug:
        msg:
                - "ansible_network_os: {{ ansible_network_os }}"
                - "connections.ssh.become_method: {{ connections.ssh.become_method }}"
                - "ansible_become_method: {{ ansible_become_method }}"

group_vars/os_version/connections.yml

---
ansible_network_os: ios

connections:
        ssh:
          become: yes
          become_method: 'enable'
          private_key_file: "~/.ssh/id_rsa_4096"

roles/os_classify_issue/vars/all/ssh/ansible_connections.with.private_key_file.yml

---
ansible_become: "{{ connections.ssh.become }}"
ansible_become_method: "{{ connections.ssh.become_method }}"

ACTUAL RESULTS with ansible 2.9.9: No issue

PLAYBOOK: include_vars issue.yml *****************************************************************************************************************************
Positional arguments: issues/include_vars issue.yml
verbosity: 4
connection: ssh
timeout: 180
become_method: sudo
tags: ('all',)
inventory: ('hosts',)
subset: localhost
forks: 1000
1 plays in issues/include_vars issue.yml

PLAY [Classifying all hosts] *********************************************************************************************************************************
META: ran handlers

TASK [set_fact] **********************************************************************************************************************************************
task path: issues/include_vars issue.yml:6
ok: [localhost] => {
    "ansible_facts": {
        "os_family": "os"
    },
    "changed": false
}

TASK [set_fact] **********************************************************************************************************************************************
task path: issues/include_vars issue.yml:8
ok: [localhost] => {
    "ansible_facts": {
        "os_version": "version"
    },
    "changed": false
}

TASK [Classifying all hosts depending on their OS & release] *************************************************************************************************
task path: issues/include_vars issue.yml:11

TASK [os_classify_issue : Classifying the network host depending on its OS & release & loading the corresponding group variables] ****************************
task path: os_classify_issue/tasks/main.yml:2
ok: [localhost] => {
    "add_group": "os_version",
    "changed": false,
    "parent_groups": [
        "all"
    ]
}

TASK [os_classify_issue : include_vars] **********************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:8
ok: [localhost] => {
    "ansible_facts": {
        "ansible_become": "{{ connections.ssh.become }}",
        "ansible_become_method": "{{ connections.ssh.become_method }}"
    },
    "ansible_included_var_files": [
        "os_classify_issue/vars/all/ssh/ansible_connections.with.private_key_file.yml"
    ],
    "changed": false
}

TASK [os_classify_issue : include_vars] **********************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:11
skipping: [localhost] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}

TASK [os_classify_issue : debug] *****************************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:14
ok: [localhost] => {
    "msg": [
        "ansible_network_os: ios",
        "connections.ssh.become_method: enable",
        "ansible_become_method: enable"
    ]
}
META: ran handlers
META: ran handlers

PLAY RECAP ***************************************************************************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

ACTUAL RESULTS with ansible 2.9.10: issue

The included hash is not interpreted correctly:
"Invalid become method specified, could not find matching plugin: '{{ connections.ssh.become_method }}'

# pip3 uninstall ansible
Found existing installation: ansible 2.9.9
Uninstalling ansible-2.9.9:
  Would remove:
    /usr/local/bin/ansible
    /usr/local/bin/ansible-config
    /usr/local/bin/ansible-connection
    /usr/local/bin/ansible-console
    /usr/local/bin/ansible-doc
    /usr/local/bin/ansible-galaxy
    /usr/local/bin/ansible-inventory
    /usr/local/bin/ansible-playbook
    /usr/local/bin/ansible-pull
    /usr/local/bin/ansible-test
    /usr/local/bin/ansible-vault
    /usr/local/lib/python3.8/dist-packages/ansible-2.9.9.dist-info/*
    /usr/local/lib/python3.8/dist-packages/ansible/*
    /usr/local/lib/python3.8/dist-packages/ansible_test/*
Proceed (y/n)? y
  Successfully uninstalled ansible-2.9.9
# pip3 install ansible==2.9.10
Collecting ansible==2.9.10
  Downloading ansible-2.9.10.tar.gz (14.2 MB)
     |████████████████████████████████| 14.2 MB 4.6 MB/s 
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from ansible==2.9.10) (2.11.2)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.8/dist-packages (from ansible==2.9.10) (5.3.1)
Requirement already satisfied: cryptography in /usr/local/lib/python3.8/dist-packages (from ansible==2.9.10) (2.9.2)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2->ansible==2.9.10) (1.1.1)
Requirement already satisfied: six>=1.4.1 in /usr/local/lib/python3.8/dist-packages (from cryptography->ansible==2.9.10) (1.15.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.8/dist-packages (from cryptography->ansible==2.9.10) (1.14.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi!=1.11.3,>=1.8->cryptography->ansible==2.9.10) (2.20)
Building wheels for collected packages: ansible
  Building wheel for ansible (setup.py) ... done
  Created wheel for ansible: filename=ansible-2.9.10-py3-none-any.whl size=16174944 sha256=d99a55988aebfdb16f3dc2ba342b4d9131f627bd080519c039ae565b8b87aaab
  Stored in directory: /root/.cache/pip/wheels/f0/02/9e/e40841e0c3ab60142092320d6cbe45c699a965e6224dbd1258
Successfully built ansible
Installing collected packages: ansible
Successfully installed ansible-2.9.10

PLAYBOOK: include_vars issue.yml *****************************************************************************************************************************
Positional arguments: issues/include_vars issue.yml
verbosity: 4
connection: ssh
timeout: 180
become_method: sudo
tags: ('all',)
inventory: ('hosts',)
subset: localhost
forks: 1000
1 plays in issues/include_vars issue.yml

PLAY [Classifying all hosts] *********************************************************************************************************************************
META: ran handlers

TASK [set_fact] **********************************************************************************************************************************************
task path: issues/include_vars issue.yml:6
ok: [localhost] => {
    "ansible_facts": {
        "os_family": "os"
    },
    "changed": false
}

TASK [set_fact] **********************************************************************************************************************************************
task path: issues/include_vars issue.yml:8
ok: [localhost] => {
    "ansible_facts": {
        "os_version": "version"
    },
    "changed": false
}

TASK [Classifying all hosts depending on their OS & release] *************************************************************************************************
task path: issues/include_vars issue.yml:11

TASK [os_classify_issue : Classifying the network host depending on its OS & release & loading the corresponding group variables] ****************************
task path: os_classify_issue/tasks/main.yml:2
ok: [localhost] => {
    "add_group": "os_version",
    "changed": false,
    "parent_groups": [
        "all"
    ]
}

TASK [os_classify_issue : include_vars] **********************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:8
ok: [localhost] => {
    "ansible_facts": {
        "ansible_become": "{{ connections.ssh.become }}",
        "ansible_become_method": "{{ connections.ssh.become_method }}"
    },
    "ansible_included_var_files": [
        "os_classify_issue/vars/all/ssh/ansible_connections.with.private_key_file.yml"
    ],
    "chan"Invalid become method specified, could not find matching plugin: '{{ connections.ssh.become_method }}'ged": false
}

TASK [os_classify_issue : include_vars] **********************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:11
skipping: [localhost] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}

TASK [os_classify_issue : debug] *****************************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:14
fatal: [localhost]: FAILED! => {
    "msg": "Invalid become method specified, could not find matching plugin: '{{ connections.ssh.become_method }}'. Use `ansible-doc -t become -l` to list available plugins."
}

The text was updated successfully, but these errors were encountered:

ansibot · 2020-07-13T14:02:24Z

Files identified in the description:

lib/ansible/modules/include_vars.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

s-hertel · 2020-07-14T16:09:03Z

I bisected the first bad commit to 9e00214

jean-christophe-manciot · 2020-07-21T13:19:13Z

@bcoca
This issue does not seem to be solved.

ACTUAL RESULTS with ansible 2.9.11: issue

With ansible from Pypi:

ansible 2.9.11
  config file = /etc/ansible/ansible.cfg
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.4 (default, Jul 13 2020, 21:16:07) [GCC 9.3.0]

The issue is still there although the symptom is quite different: The value '{{ connections.ssh.become }}' is not a valid boolean

TASK [os_classify_issue : debug] *****************************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:14
The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/ansible/executor/task_executor.py", line 147, in run
    res = self._execute()
  File "/usr/local/lib/python3.8/dist-packages/ansible/executor/task_executor.py", line 607, in _execute
    self._connection = self._get_connection(variables=variables, templar=templar)
  File "/usr/local/lib/python3.8/dist-packages/ansible/executor/task_executor.py", line 911, in _get_connection
    if boolean(cvars.get('ansible_become', self._task.become)):
  File "/usr/local/lib/python3.8/dist-packages/ansible/module_utils/parsing/convert_bool.py", line 26, in boolean
    raise TypeError("The value '%s' is not a valid boolean.  Valid booleans include: %s" % (to_text(value), ', '.join(repr(i) for i in BOOLEANS)))
TypeError: The value '{{ connections.ssh.become }}' is not a valid boolean.  Valid booleans include: 'yes', 1, 0, 'off', 'n', 'false', 'true', 'on', 't', '1', 'no', 'f', 'y', '0'
fatal: [localhost]: FAILED! => {
    "msg": "Unexpected failure during module execution."
}

it seems that v2.9.11 version does not include the commits you merged into devel, yet the symptom changed:

git-ansible (devel)# git checkout v2.9.11
...
HEAD is now at 34d1fac39b New release v2.9.11
git-ansible ((v2.9.11))# git log --grep="template connection variables accessed directly before using (#70657)"
git-ansible ((v2.9.11))#

one or more other commits have been applied in the meantime that changed the symptom
this means that there might be some undesired side effects when a new stable version includes your commits and the ones which changed the symptom

ACTUAL RESULTS with ansible 2.11.0.dev0: issue

With ansible from sources (8188bce):

  config file = /etc/ansible/ansible.cfg
  ansible python module location = git-ansible/lib/ansible
  ansible collection location = /home/admin/.ansible/collections:/usr/share/ansible/collections
  executable location = git-ansible/bin/ansible
  python version = 3.8.4 (default, Jul 13 2020, 21:16:07) [GCC 9.3.0]

The issue is still there although the symptom is a little different: could not find matching plugin: 'enable'

TASK [os_classify_issue : debug] *****************************************************************************************************************************
task path: os_classify_issue/tasks/main.yml:14
redirecting (type: become) ansible.builtin.enable to ansible.netcommon.enable
fatal: [localhost]: FAILED! => {
    "msg": "Invalid become method specified, could not find matching plugin: 'enable'. Use `ansible-doc -t become -l` to list available plugins."
}

bcoca · 2020-07-21T13:43:03Z

the backport to 2.9 is #70689, which has not been merged yet.

samdoran added P3 Priority 3 - Approved, No Time Limitation verified This issue has been verified/reproduced by maintainer and removed needs_triage Needs a first human triage before being processed. labels Jul 14, 2020

s-hertel mentioned this issue Jul 15, 2020

template connection variables accessed directly before using #70657

Merged

bcoca closed this as completed in #70657 Jul 16, 2020

sivel mentioned this issue Jul 20, 2020

Setting ansible_become=no in ini-inventory for localhost leads to become being used anyway. #70759

Closed

ansible locked and limited conversation to collaborators Aug 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hash variables fail to be loaded with include_vars in a role after a group_by #70598

Hash variables fail to be loaded with include_vars in a role after a group_by #70598

jean-christophe-manciot commented Jul 13, 2020

ansibot commented Jul 13, 2020

s-hertel commented Jul 14, 2020

jean-christophe-manciot commented Jul 21, 2020

bcoca commented Jul 21, 2020

Hash variables fail to be loaded with include_vars in a role after a group_by #70598

Hash variables fail to be loaded with include_vars in a role after a group_by #70598

Comments

jean-christophe-manciot commented Jul 13, 2020

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

ACTUAL RESULTS with ansible 2.9.9: No issue

ACTUAL RESULTS with ansible 2.9.10: issue

ansibot commented Jul 13, 2020

s-hertel commented Jul 14, 2020

jean-christophe-manciot commented Jul 21, 2020

ACTUAL RESULTS with ansible 2.9.11: issue

ACTUAL RESULTS with ansible 2.11.0.dev0: issue

bcoca commented Jul 21, 2020