template (the lookup plugin) through to_nice_json adds double-quote

[ylmrx]

Summary

the to_nice_json filter changed behavior and now return a string instead of a (json) dict
(ie. it add double quotes around the value, and escape line returns and double quotes within)

ie. :

$ ansible-playbook play-rien.yml
[...]
TASK [deb] 
ok: [localhost] => (item="{\n    \n        \"node_name\": \"foo\",\n    \"datacenter\": \"bar\",\n}\n") => {
    "msg": "\"{\\n    \\n        \\\"node_name\\\": \\\"foo\\\",\\n    \\\"datacenter\\\": \\\"bar\\\",\\n}\\n\""
}
ok: [localhost] => (item="{\n    \n        \"node_name\": \"foo\",\n    \"datacenter\": \"bar\",\n}\n") => {
    "msg": "\"{\\n    \\n        \\\"node_name\\\": \\\"foo\\\",\\n    \\\"datacenter\\\": \\\"bar\\\",\\n}\\n\""
}
ok: [localhost] => (item={'node_name': 'foo', 'datacenter': 'bar'}) => {
    "msg": {
        "datacenter": "bar",
        "node_name": "foo"
    }
}

(prior to ansible-5.0.0 , the 3 items gave the very same output as the 3rd iteration)

Issue Type

Bug Report

Component Name

template lookup plugin

Ansible Version

$ ansible --version
ansible [core 2.12.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/fuzzy/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/fuzzy/.pyenv/versions/3.10.0/envs/deploy_env/lib/python3.10/site-packages/ansible
  ansible collection location = /home/fuzzy/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/fuzzy/.pyenv/versions/deployments/bin/ansible
  python version = 3.10.0 (default, Oct 21 2021, 12:56:50) [GCC 11.1.0]
  jinja version = 3.0.2
  libyaml = True

Configuration

$ ansible-config dump --only-changed
unrelated

OS / Environment

archlinux (but distribution seem irrelevant)

Steps to Reproduce

example playbook

- hosts: localhost
  vars:
    consul_node_name: foo
    consul_datacenter: bar
  tasks:
    - name: deb
      ansible.builtin.debug:
        msg: "{{ item }}"
      with_items:
        - "{{ lookup('template', './config.json.j2') | to_nice_json }}"
        - "{{ lookup('template', './config.json.j2') | to_json }}"
        - "{{ lookup('template', './config.json.j2') }}"

demo json template

{# This template will be passed through the 'to_nice_json' filter #}
{# The filter fixes whitespace, indentation and comma's on the last item #}
{
    {# Common Settings #}

    {## Node ##}
    "node_name": "{{ consul_node_name }}",
    "datacenter": "{{ consul_datacenter }}",
}

Expected Results

$ ansible-playbook play-rien.yml                                                                                                                                                                                                                                                                                            
[WARNING]: No inventory was parsed, only implicit localhost is available                                                                                                                                                                                                                                                                                                                       
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                                               
PLAY [localhost]
                                                                                                                                                                                                                                                                                                                                                                                               
TASK [Gathering Facts] 
ok: [localhost]                                                                                                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                                                                                               
TASK [deb] 
ok: [localhost] => (item={'datacenter': 'bar', 'node_name': 'foo'}) => {                                                                                                                                                                                                                                                                                                             
    "msg": {                                                                                                                                                                                                                                                                                                                                                                                   
        "datacenter": "bar",                                                                                                                                                                                                                                                                                                                                                             
        "node_name": "foo"                                                                                                                                                                                                                                                                                                                                                                 
    }                                                                                                                                                                                                                                                                                                                                                                                          
}                                                                                                                                                                                                                                                                                                                                                                                              
ok: [localhost] => (item={'node_name': 'foo', 'datacenter': 'bar'}) => {                                                                                                                                                                                                                                                                                                             
    "msg": {                                                                                                                                                                                                                                                                                                                                                                                   
        "datacenter": "bar",                                                                                                                                                                                                                                                                                                                                                             
        "node_name": "foo"                                                                                                                                                                                                                                                                                                                                                                 
    }                                                                                                                                                                                                                                                                                                                                                                                          
}                                                                                                                                                                                                                                                                                                                                                                                              
ok: [localhost] => (item={'node_name': 'foo', 'datacenter': 'bar'}) => {                                                                                                                                                                                                                                                                                                             
    "msg": {                                                                                                                                                                                                                                                                                                                                                                                   
        "datacenter": "bar",                                                                                                                                                                                                                                                                                                                                                             
        "node_name": "foo"                                                                                                                                                                                                                                                                                                                                                                 
    }                                                                                                                                                                                                                                                                                                                                                                                          
}                                                                                                                                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                       
PLAY RECAP 
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Actual Results

$ ansible-playbook play-rien.yml                                                                                                                                                                                                                                                                                                      [WARNING]: No inventory was parsed, only implicit localhost is available                                                                                                                                                                                                                                                                                                                       
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                                               
PLAY [localhost] 
                                                                                                                                                                                                                                                                                                                                                                                               
TASK [Gathering Facts] 
ok: [localhost]

TASK [deb] 
ok: [localhost] => (item="{\n    \n        \"node_name\": \"foo\",\n    \"datacenter\": \"bar\",\n}\n") => {
    "msg": "\"{\\n    \\n        \\\"node_name\\\": \\\"foo\\\",\\n    \\\"datacenter\\\": \\\"bar\\\",\\n}\\n\""
}
ok: [localhost] => (item="{\n    \n        \"node_name\": \"foo\",\n    \"datacenter\": \"bar\",\n}\n") => {
    "msg": "\"{\\n    \\n        \\\"node_name\\\": \\\"foo\\\",\\n    \\\"datacenter\\\": \\\"bar\\\",\\n}\\n\""
}
ok: [localhost] => (item={'node_name': 'foo', 'datacenter': 'bar'}) => {
    "msg": {
        "datacenter": "bar",
        "node_name": "foo"
    }
}

PLAY RECAP 
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• lib/ansible/plugins/lookup/template.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[sivel]

Caused by 84e473a

[bcoca]

notes:

• to_json and to_nice_json ALWAYS are supposed to return a string (of serialized JSON) otherwise they would be a Python dictionary/list. Ansible tends to automatically convert JSON into Python data structures, though we have some safeguards against it when explicitly using those filters (which would create the behavior you see).
• debug action does it's own 'jsonization' so it is probably the one adding the escapes on top of the 'JSON string' not a good way to examine the return structures, writing to a template file is preferable
• as much as this behavior might seem strange, it probably is more correct than the previous, which auto converted JSON into Python when debug action accessed it, to then convert back to JSON to display it

[ylmrx]

• I understand this might be a safer expected behavior
• regarding your item 2 and 3, debug' "jsonization" is not in cause here, as I reproduce the issue with this task too :

- name: Create configuration
  copy:
    dest: "{{ item.dest }}"
    owner: "{{ consul_user }}"
    group: "{{ consul_group }}"
    content: "{{ lookup('template', 'templates/config.json.j2') | to_nice_json }}"
    mode: "0640"
  with_items:
    - dest: "{{ consul_config_path }}/config.json"
  notify:
    - restart consul

[bcoca]

please use template instead of copy, content has many issues and the description explicitly recommends not using it in templating. In any case i would expect this to be JSON wrapped in JSON.. so |to_string would probably be the filter to use if you insist on keeping content at worse content: "{{ lookup('template', 'templates/config.json.j2') |from_json|to_nice_json

[ylmrx]

I'm actually unsure how we ended up with this construct (we are very well aware of the template module 😄 ) in our role.
I'm guessing it was failing to keep a nice indentation (the template file is way more complex than the example given above)

Thanks @bcoca for the informations (and the from_json tip/hack)

If the string output is the expected behavior, this issue can be closed

Have a great day

[nitzmahone]

Closing by request