Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt.utils.yamldumper.dump/safe_dump incorrectly return 'str' instead of unicode on python2.7 (centos) #49100

Closed
pruiz opened this issue Aug 13, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@pruiz
Copy link
Contributor

commented Aug 13, 2018

Description of Issue/Question

After upgrading to 2018.3.2 (from 2017.7.2), I've found that when using pillar data (with non-ascii characters) while rendering yaml sls files, it fails with: UnicodeDecodeError: 'ascii' codec can't decode byte...

AFAIK it is due to PyYAML, as invoked at salt/utils/yamldumper.py:safe_dump, returning an 'str' object, which later on is being coerced into unicode as if it where an ascii-based string.

IMHO, this can be fixed by invoking salt.utils.stringutils.to_unicode() to yaml data before returning. I'll try to submit a pull-req next.

Setup

First you will need some misc data at pillar (in my case git-based ext_pillar), like:

/tmp/k2.sls

data:
  question: Quieres Café?

Then a sample state like this one:

{%- set data = salt['pillar.get']('data') %}

/tmp/k.settings:
  file.managed:
    - encoding: UTF-8
    - contents: |
      {{ data|yaml }}

Steps to Reproduce Issue

# salt-call  -l debug state.template /tmp/k2.sls
[...]
local:
    Data failed to compile:
----------
    Rendering SLS ':/tmp/k2.sls' failed: Jinja error: 'ascii' codec can't decode byte 0xc3 in position 398: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 390, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 10, in top-level template code
  File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 805, in format_yaml
    if yaml_txt.endswith('\n...'):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 398: ordinal not in range(128)

; line 10

---
[...]
/tmp/k.settings:
  file.managed:
    - template: jinja
    - encoding: UTF-8
    - contents: |
      {{ settings|yaml }}    <======================

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 390, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
[...]
---

Versions Report

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Jul 12 2017, 17:32:34)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: centos 6.3 Final
         locale: UTF-8
        machine: x86_64
        release: 4.13.13-6-pve
         system: Linux
        version: CentOS 6.3 Final
@garethgreenaway

This comment has been minimized.

Copy link
Member

commented Aug 14, 2018

@pruiz Thanks for the report. @terminalmage Can you help get a PR for this over the finish line?

@pruiz

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

Hi,

@terminalmage & @garethgreenaway, I've just submitted a pull-req against develop with a proposed fix: #49122.

@terminalmage

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2018

The dumper functions are correct to return str types, as serialized contents must be able to be written to filehandles.

I've submitted #49125 as a correct fix for this.

@terminalmage

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2018

Closing, as #49125 was merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.