hostvars['name'] will not be converted to strings in the proper contexts (hostvarsvars issue)

[sasjowood]

SUMMARY

When using the templating filter to_nice_yaml (or to_yaml) and running on SLES 12sp5, an error is thrown saying that the value must be a string. On the same system, if the Ansible version used is less than 2.8.0, the error is not thrown.

I also tested with RHEL 7.2, 7.7, and 8.0 as well as SLES 12sp3, sp4, and SLES 15. For each environment I tested Ansible 2.7.0, 2.7.16, 2.8.0, 2.8.8, 2.9.0, and 2.9.4. The filter worked correctly in each of these environments.

ISSUE TYPE

• Bug Report

COMPONENT NAME

• to_yaml
• to_nice_yaml
  (may affect other templating filters)

ANSIBLE VERSION

ansible 2.9.4
  config file = None
  configured module search path = [u'/home/sles/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Jan 11 2017, 10:56:06) [GCC]

I also tested Ansible 2.8.0, 2.8.8, and 2.9.0, all of which threw the same error.

CONFIGURATION

<no output>

OS / ENVIRONMENT

SLES 12sp5

STEPS TO REPRODUCE

On a SLES 12sp5 environment with Ansible 2.8.0 - Ansible 2.9.4:

1. Create the two files specified below, parallel to each other.
2. Run the following command :

ansible-playbook test.yml

test.yml:

- name: "Test"
  hosts: localhost

  tasks:
    - name: "Create file"
      template:
        src: "./data_file.yaml.j2"
        dest: "./test_output.yaml"
        mode: 0777

data_file.yaml.j2:

{{ hostvars[inventory_hostname] | to_nice_yaml }}

EXPECTED RESULTS

From Ansible 2.7.16 on SLES 12sp5:

ansible-playbook 2.7.16
  config file = None
  configured module search path = [u'/home/sles/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.13 (default, Jan 11 2017, 10:56:06) [GCC]
No config file found; using defaults
setting up inventory plugins
/etc/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
Skipping due to inventory source not existing or not being readable by the current user
/etc/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected
Skipping due to inventory source not existing or not being readable by the current user
/etc/ansible/hosts did not meet yaml requirements, check plugin documentation if this is unexpected
Skipping due to inventory source not existing or not being readable by the current user
/etc/ansible/hosts did not meet ini requirements, check plugin documentation if this is unexpected
/etc/ansible/hosts did not meet auto requirements, check plugin documentation if this is unexpected
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: test.yml ************************************************************************************************************************************************************************************************************************************************************
1 plays in test.yml

PLAY [Test] *******************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************************************************************************
task path: /home/sles/test.yml:1
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sles
<127.0.0.1> EXEC /bin/sh -c 'echo ~sles && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/sles/.ansible/tmp/ansible-tmp-1580323027.3-205590116244428 `" && echo ansible-tmp-1580323027.3-205590116244428="` echo /home/sles/.ansible/tmp/ansible-tmp-1580323027.3-205590116244428 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<127.0.0.1> PUT /home/sles/.ansible/tmp/ansible-local-2897Ua6ysp/tmpqMNlsD TO /home/sles/.ansible/tmp/ansible-tmp-1580323027.3-205590116244428/AnsiballZ_setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/sles/.ansible/tmp/ansible-tmp-1580323027.3-205590116244428/ /home/sles/.ansible/tmp/ansible-tmp-1580323027.3-205590116244428/AnsiballZ_setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 /home/sles/.ansible/tmp/ansible-tmp-1580323027.3-205590116244428/AnsiballZ_setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/sles/.ansible/tmp/ansible-tmp-1580323027.3-205590116244428/ > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [Create file] ************************************************************************************************************************************************************************************************************************************************************
task path: /home/sles/test.yml:5
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sles
<127.0.0.1> EXEC /bin/sh -c 'echo ~sles && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703 `" && echo ansible-tmp-1580323028.21-7631150998703="` echo /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/files/stat.py
<127.0.0.1> PUT /home/sles/.ansible/tmp/ansible-local-2897Ua6ysp/tmpTkX9Xx TO /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/AnsiballZ_stat.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/ /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/AnsiballZ_stat.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/AnsiballZ_stat.py && sleep 0'
<127.0.0.1> PUT /home/sles/.ansible/tmp/ansible-local-2897Ua6ysp/tmp5d2CU8/data_file.yaml.j2 TO /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/source
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/ /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/source && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/files/copy.py
<127.0.0.1> PUT /home/sles/.ansible/tmp/ansible-local-2897Ua6ysp/tmpHhdX_e TO /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/AnsiballZ_copy.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/ /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/AnsiballZ_copy.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/AnsiballZ_copy.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "checksum": "838bbea70f3a3300b212899f8ea562459d5302ec",
    "dest": "./test_output.yaml",
    "diff": [],
    "gid": 100,
    "group": "users",
    "invocation": {
        "module_args": {
            "_original_basename": "data_file.yaml.j2",
            "attributes": null,
            "backup": false,
            "checksum": "838bbea70f3a3300b212899f8ea562459d5302ec",
            "content": null,
            "delimiter": null,
            "dest": "./test_output.yaml",
            "directory_mode": null,
            "follow": false,
            "force": true,
            "group": null,
            "local_follow": null,
            "mode": 511,
            "owner": null,
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": "/home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/source",
            "unsafe_writes": null,
            "validate": null
        }
    },
    "md5sum": "5cdc92ad2af9c22c9bea7cdbd620b038",
    "mode": "0777",
    "owner": "sles",
    "size": 22374,
    "src": "/home/sles/.ansible/tmp/ansible-tmp-1580323028.21-7631150998703/source",
    "state": "file",
    "uid": 1000
}
META: ran handlers
META: ran handlers

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

ACTUAL RESULTS

From Ansible 2.9.4 on SLES 12sp5:

ansible-playbook 2.9.4
  config file = None
  configured module search path = [u'/home/sles/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.13 (default, Jan 11 2017, 10:56:06) [GCC]
No config file found; using defaults
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
yaml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
ini declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
toml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: test.yml ************************************************************************************************************************************************************************************************************************************************************
Positional arguments: test.yml
become_method: sudo
inventory: (u'/etc/ansible/hosts',)
forks: 5
tags: (u'all',)
verbosity: 4
connection: smart
timeout: 10
1 plays in test.yml

PLAY [Test] *******************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************************************************************************
task path: /home/sles/test.yml:1
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sles
<127.0.0.1> EXEC /bin/sh -c 'echo ~sles && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/sles/.ansible/tmp/ansible-tmp-1580323198.03-103664775467738 `" && echo ansible-tmp-1580323198.03-103664775467738="` echo /home/sles/.ansible/tmp/ansible-tmp-1580323198.03-103664775467738 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<127.0.0.1> PUT /home/sles/.ansible/tmp/ansible-local-2987ukxc_h/tmphFwfYg TO /home/sles/.ansible/tmp/ansible-tmp-1580323198.03-103664775467738/AnsiballZ_setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/sles/.ansible/tmp/ansible-tmp-1580323198.03-103664775467738/ /home/sles/.ansible/tmp/ansible-tmp-1580323198.03-103664775467738/AnsiballZ_setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 /home/sles/.ansible/tmp/ansible-tmp-1580323198.03-103664775467738/AnsiballZ_setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/sles/.ansible/tmp/ansible-tmp-1580323198.03-103664775467738/ > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [Create file] ************************************************************************************************************************************************************************************************************************************************************
task path: /home/sles/test.yml:5
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sles
<127.0.0.1> EXEC /bin/sh -c 'echo ~sles && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/sles/.ansible/tmp/ansible-tmp-1580323199.2-225610635389839 `" && echo ansible-tmp-1580323199.2-225610635389839="` echo /home/sles/.ansible/tmp/ansible-tmp-1580323199.2-225610635389839 `" ) && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/sles/.ansible/tmp/ansible-tmp-1580323199.2-225610635389839/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "msg": "AnsibleError: Unexpected templating type error occurred on ({{ hostvars[inventory_hostname] | to_nice_yaml }}): value must be a string"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[ansibot]

Files identified in the description:

• lib/ansible/plugins/filter

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[JoelKle]

Same problem on my side. The to_yaml and to_nice_yaml throw a value must be a string error when used inside a template file.

If I use the build in Jinja2 filter tojson inside the template, it's working fine for JSON.
Example:
group_vars/group.yml:

variable:
  key: value

data_file.yaml.j2:

{{ variable | tojson }}

https://jinja.palletsprojects.com/en/2.11.x/templates/#tojson

[wawa19933]

This error is thrown even when used in vars section of template module.

I have found that it does not fail in virtualenv, but throws when using Ansible from system package (ArchLinux, Ansible 2.9.9). In virtualenv, however the same version works (pyenv virtualenv using system Python).

[JoelKle]

I think I found where the problem is. Looks like there is a bug in the libyaml-0-2 libary on a Ubuntu 20.04 Focal.

The to_nice_yaml Ansible filter use the pyyaml python module. As described in the off. Ansible docs.
The pyyaml python module use the underlying C lib libyaml-0-2. I think there is bug in this C lib.

When I remove this underlying libyaml from my Ansible host system, this filter works perfect.
My setup:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04 LTS
Release:	20.04
Codename:	focal

$ apt policy python3-yaml
python3-yaml:
  Installiert:           5.3.1-1
  Installationskandidat: 5.3.1-1
  Versionstabelle:
 *** 5.3.1-1 500
        500 http://de.archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt policy libyaml-0-2
libyaml-0-2:
  Installiert:           0.2.2-1
  Installationskandidat: 0.2.2-1
  Versionstabelle:
 *** 0.2.2-1 500
        500 http://de.archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo mv /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.6.bkp
$ sudo mv /usr/lib/x86_64-linux-gnu/libyaml-0.so.2 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.bkp

Checking:
$ python3
>>> import yaml
>>> yaml.__with_libyaml__
False

= pyyaml is not using the libyaml

After all these steps the filter worked again.

For me it looks like there is a bug in either the pyyaml module or the libyaml-0-2 lib. Have not enough time to troubleshoot this further.

[bcoca]

this is a HostVarsVars issue (the object you get from hostvars['host'] which tries to simulate a dictionary), it is not properly 'stingifiying' as dictionaries normally do. The fix would be to force this behaviour, which is not done as it would require templating every value and the object itself exists to avoid that for performance reasons.

[salderma]

A colleague of mine, who is running Focal has encountered this issue as well. We have a git repo w/ a Vagrantfile using Ansible as provisioner, and his machine fails to process every single hostvars magic variable use case. I am running Bionic and have no issues. The two of us have spent the day trying to figure out why things work for me and not for him when using vagrant up --provision or vagrant provision.

Psuedo removing libyaml as @JoelKle mentions is not a viable work-around due to Vagrant (ruby under the hood) relying on libyaml.

[petpetpetpet]

FWIW: I was able to workaround this issue (in my use case) by first converting the hostvars to json, then back, and then pushing that result through the to_nice_yaml filter.

"{{ hostvars[inventory_hostname] | to_json | from_json | to_nice_yaml(indent=2)}}"

[sivel]

This has been resolved in #72607 and will be included in ansible-core 2.12

If you have further questions please stop by IRC or the mailing list:

• IRC: #ansible on irc.libera.chat
• mailing list: https://groups.google.com/forum/#!forum/ansible-project