to_json filter could not keep non-ascii characters

[tumluliu]

SUMMARY

to_json/to_nice_json filters could not properly preserve non-ascii characters in variables

ISSUE TYPE

• Bug Report

COMPONENT NAME

to_json/to_nice_json filters

ANSIBLE VERSION

ansible 2.9.2
  config file = /home/me/to/my/project/ansible.cfg
  configured module search path = ['/home/me/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/me/.pyenv/versions/3.7.4/lib/python3.7/site-packages/ansible
  executable location = /home/me/.pyenv/versions/3.7.4/bin/ansible
  python version = 3.7.4 (default, Jul 19 2019, 16:11:19) [GCC 7.4.0]

CONFIGURATION

NSIBLE_SSH_ARGS = -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null
DEFAULT_CALLBACK_WHITELIST = ['profile_tasks']
DEFAULT_FORKS = 10
DEFAULT_TIMEOUT = 30
HOST_KEY_CHECKING = False

OS / ENVIRONMENT

Ubuntu 18.04

STEPS TO REPRODUCE

test-playbook.yml:

- name: Test templating with unicodes
  hosts: localhost
  connection: local
  become: no
  tasks:
  - name: Template a json file with Chinese and German chars
    vars:
      german_var:
        a: München
        b: Öüä
    template:
      src: "test_template.txt.j2"
      dest: "test.txt"

test_template.txt.j2:

{
    "germanVar": {{ german_var | to_json }}
}
---
german_var: {{ german_var | to_yaml }}

EXPECTED RESULTS

{
    "germanVar": {"a": "München", "b": "Öüä"}
}
---
german_var: {a: München, b: Öüä}

ACTUAL RESULTS

{
    "germanVar": {"a": "M\u00fcnchen", "b": "\u00d6\u00fc\u00e4"}
}
---
german_var: {a: München, b: Öüä}

Is this actual behavior by design? If yes, is it possible to stop it from doing that converting while to keep the raw non-ascii characters, just as what to_yaml does?

[ansibot]

Files identified in the description:

• lib/ansible/plugins/filter

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[mkrizek]

ansible/lib/ansible/plugins/filter/core.py

Lines 76 to 78 in 28f9fbd

	def to_json(a, *args, **kw):
	''' Convert the value to JSON '''
	return json.dumps(a, cls=AnsibleJSONEncoder, *args, **kw)

This appears to be due to ensure_ascii arg for json.dumps [0] is True by default which we do not change in to_json filter. Changing that would fix the issue however I am not sure what would be the implications of such change.

In the meantime however you can probably work around that by doing the following:

{{ non_ascii_var | to_json(ensure_ascii=False) }}

[0] https://docs.python.org/3/library/json.html#json.dump

[tumluliu]

Thank you so much @mkrizek !! The ensure_ascii=False works like a charm. For me, this can be closed. But it would be great to have this "hidden" param documented somewhere in Ansible

[bmillemathias]

would also require some unit/integration test also, right ?

[mkrizek]

Closing as the functionality has been documented in the above PR.