Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

to_nice_yaml unable to handle unicode characters #10243

Closed
z38 opened this issue Feb 14, 2015 · 4 comments
Closed

to_nice_yaml unable to handle unicode characters #10243

z38 opened this issue Feb 14, 2015 · 4 comments

Comments

@z38
Copy link

z38 commented Feb 14, 2015

Issue Type:

Bug Report

Ansible Version:

1.8.2

Environment:

Ubuntu 14.04

Summary:

If you use {{ somevar|to_nice_yaml }} and somevar contains unicode characters (such as ä,à), the task fails.

Steps To Reproduce:

---

- name: Test to_nice_yaml
  hosts: all
  gather_facts: false
  vars:
    foo: "Téêèst"
  tasks:
    - debug: msg="{{ foo|to_nice_yaml }}"
Expected Results:

"T\\xE9\\xEA\\xE8st"\n

Actual Results:
fatal: [example.com] => Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ansible/runner/__init__.py", line 590, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/usr/lib/pymodules/python2.7/ansible/runner/__init__.py", line 792, in _executor_internal
    return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
  File "/usr/lib/pymodules/python2.7/ansible/runner/__init__.py", line 993, in _executor_internal_inner
    module_args = template.template(self.basedir, module_args, inject, fail_on_undefined=self.error_on_undefined_vars)
  File "/usr/lib/pymodules/python2.7/ansible/utils/template.py", line 116, in template
    varname = template_from_string(basedir, varname, templatevars, fail_on_undefined)
  File "/usr/lib/pymodules/python2.7/ansible/utils/template.py", line 362, in template_from_string
    res = jinja2.utils.concat(rf)
  File "<template>", line 10, in root
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)
@gtirloni
Copy link
Contributor

I was able to reproduce this issue on Fedora 21 with jinja2 2.7.3.

Something like this was supposed to have been fixed in a newer version of Jinja2 (pallets/jinja#304). I tried to update jinja2 to version 2.8.dev0 and the issue reported here isn't fixed by that.

Adding some debugging code throughout Ansible and Jinja2 modules, I can see the encoding is 'ascii' (ansible/../template.py and jinja2/../template/utils.py, for instance).

Workaround seems to edit ansible-playbook and add this to main():

reload(sys)
sys.setdefaultencoding('utf-8')

But this seems too drastic. FYI, The 'data' parameter passed to template_from_string() IS unicode.

@abadger
Copy link
Contributor

abadger commented Feb 15, 2015

Actually, python2 code should never, ever call sys.setdefaultencoding('utf-8'). (I know there's some pieces of code in ansible modules that is doing this and I'm hoping to remove them for v2). I'm trying to get some time to write up a blog post about why that is sometime in the near future. We'll see :-)

It looks like the problem is that the to_nice_yaml filter is returning values of type str but jinja2 needs values of type unicode when the value contains non-ascii characters. I'll push a commit in a little bit that should fix this.

@abadger
Copy link
Contributor

abadger commented Feb 15, 2015

That commit should fix things for you. A few weeks ago I tried to go through the filters and fix ones that could return str values with non-ascii characters but obviously I missed this one. If you run across any other ones that also throw a UnicodeDecode/UnicodeEncodeError feel free to open a bug and ping me on IRC if you think I might have missed it (abadger1999 in #ansible on freenode)

Closing This Ticket

Hi!

We believe recent commits (likely detailed above) should resolve this question or problem for you.

This will also be included in the next major release.

If you continue seeing any problems related to this issue, or if you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular
issue is resolved.

Thank you!

@z38
Copy link
Author

z38 commented Feb 15, 2015

Thank you for the very quick response, works like a charm!

abadger added a commit that referenced this issue Feb 17, 2015
@abadger @jimi-c
Fix the to_nice_yaml filter to always return unicode strings
cdd1cf5 
Fixes #10243
@ansible ansible locked and limited conversation to collaborators Apr 25, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abadger @z38 @gtirloni