UnicodeDecodeError in highstate outputter

[Jille]

I get this crash when running state.highstate test=True:

master# salt -t 60 minion state.highstate test=True
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 99: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 240, in salt_main
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 189, in run
    self._output_ret(ret_, out)
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 243, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/pymodules/python2.7/salt/output/__init__.py", line 36, in display_output
    display_data = get_printout(out, opts)(data).rstrip()
  File "/usr/lib/pymodules/python2.7/salt/output/highstate.py", line 74, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/pymodules/python2.7/salt/output/highstate.py", line 203, in _format_host
    u'\n' + u' ' * 14)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 99: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 240, in salt_main
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 189, in run
    self._output_ret(ret_, out)
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 243, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/pymodules/python2.7/salt/output/__init__.py", line 36, in display_output
    display_data = get_printout(out, opts)(data).rstrip()
  File "/usr/lib/pymodules/python2.7/salt/output/highstate.py", line 74, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/pymodules/python2.7/salt/output/highstate.py", line 203, in _format_host
    u'\n' + u' ' * 14)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 99: ordinal not in range(128)

The state:

/some-file:
    file.managed:
        - source: 'salt://some-file'
        - show_diff: true

It's easy to reproduce: Put an lorem ipsum on /srv/salt/some-file, highstate it, log in to the minion, add some non-ascii (like \xc2\xb5) to /some-file and run another highstate and it will crash like above.

I'm running 2014.7.0rc1 at 20f2334

[basepi]

Thanks for the report. I was under the impression we had most of these unicode errors solved, but apparently not. We'll investigate this.

[perry-mitchell]

Another error dump in-case it helps:

==> web: [ERROR   ] An un-handled exception was caught by salt's global exception handler:
==> web: UnicodeEncodeError: 'ascii' codec can't encode characters in position 117297-117309: ordinal not in range(128)
==> web: Traceback (most recent call last):
==> web:   File "/usr/bin/salt-call", line 11, in <module>
==> web:     salt_call()
==> web:   File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 82, in salt_call
==> web:     client.run()
==> web:   File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 319, in run
==> web:     caller.run()
==> web:   File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 148, in run
==> web:     self.opts)
==> web:   File "/usr/lib/python2.7/dist-packages/salt/output/__init__.py", line 49, in display_output
==> web:     print(display_data)
==> web: UnicodeEncodeError: 'ascii' codec can't encode characters in position 117297-117309: ordinal not in range(128)
==> web: Traceback (most recent call last):
==> web:   File "/usr/bin/salt-call", line 11, in <module>
==> web:     salt_call()
==> web:   File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 82, in salt_call
==> web:     client.run()
==> web:   File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 319, in run
==> web:     caller.run()
==> web:   File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 148, in run
==> web:     self.opts)
==> web:   File "/usr/lib/python2.7/dist-packages/salt/output/__init__.py", line 49, in display_output
==> web:     print(display_data)
==> web: UnicodeEncodeError: 'ascii' codec can't encode characters in position 117297-117309: ordinal not in range(128)

Is there any workaround for this?

[mpaolini]

@basepi there is plenty of unsolved unicode errors: just add a non-ascii char in a state name and you'll see UnicodeEncode errors popping out from every corner. Basically I have the impression salt does not support unicode at all. Maybe we can mention it in the docs.

[perry-mitchell]

@mpaolini Why would you want to mention that in the docs? This is most certainly a bug IMHO, as it's unreasonable to expect unicode incompatibility across config files. Unicode support should be trivial.

A colleague is running the same Salt setup as I am in the same project, and his is working whilst mine is not. The only difference is our host machines, his being Ubuntu and mine being Debian. I've torn-down and up'd our Vagrant machines a ton of times to no avail. Desperately hoping for a fix or workaround...

[mpaolini]

@perry-mitchell I would mention it in the docs under known bugs. I am triying to cook a patch for fixing these issues but it won't be soon

[perry-mitchell]

@mpaolini Muchly appreciated. Do you know what particular area is causing the problem here?

[mpaolini]

@perry-mitchell mostly is 'bytestring {}'.format(some_user_input) that in python2.7 causes the infamous UnicodeEncodeError if some_user_input has a non-ascii char.

A quick fix is to use from __future__ import unicode_literals at the very top of the module, this works out well in my tests because most of the strings in the states are "application level" so they can assumed never to leave the application boundaries.

Nevertheless this small change requires very thorough analysis of were these strings flow in the program and where they touch the outside world (a pipe, a file...)

[basepi]

Yes, you would expect unicode support to be a solved problem. But that's the primary reason that Python3 exists, is because Python2 unicode support is....painful.

Thanks for the updates, @mpaolini, I look forward to seeing your fix.

[kylegato]

Adding onto this.

salt-minion 2014.1.13 (Hydrogen)
salt-master 2014.1.13 (Hydrogen)
FreeBSD 10.0-RELEASE-p12 FreeBSD 10.0-RELEASE-p12 #0: Tue Nov 4 05:07:17 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 18384: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/bin/salt-call", line 9, in <module>
    load_entry_point('salt==2014.1.13', 'console_scripts', 'salt-call')()
  File "/usr/local/lib/python2.7/site-packages/salt/scripts.py", line 82, in salt_call
    client.run()
  File "/usr/local/lib/python2.7/site-packages/salt/cli/__init__.py", line 319, in run
    caller.run()
  File "/usr/local/lib/python2.7/site-packages/salt/cli/caller.py", line 148, in run
    self.opts)
  File "/usr/local/lib/python2.7/site-packages/salt/output/__init__.py", line 35, in display_output
    display_data = get_printout(out, opts)(data).rstrip()
  File "/usr/local/lib/python2.7/site-packages/salt/output/highstate.py", line 44, in output
    return _format_host(host, hostdata)[0]
  File "/usr/local/lib/python2.7/site-packages/salt/output/highstate.py", line 200, in _format_host
    return '\n'.join(hstrs), changed
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 18384: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/bin/salt-call", line 9, in <module>
    load_entry_point('salt==2014.1.13', 'console_scripts', 'salt-call')()
  File "/usr/local/lib/python2.7/site-packages/salt/scripts.py", line 82, in salt_call
    client.run()
  File "/usr/local/lib/python2.7/site-packages/salt/cli/__init__.py", line 319, in run
    caller.run()
  File "/usr/local/lib/python2.7/site-packages/salt/cli/caller.py", line 148, in run
    self.opts)
  File "/usr/local/lib/python2.7/site-packages/salt/output/__init__.py", line 35, in display_output
    display_data = get_printout(out, opts)(data).rstrip()
  File "/usr/local/lib/python2.7/site-packages/salt/output/highstate.py", line 44, in output
    return _format_host(host, hostdata)[0]
  File "/usr/local/lib/python2.7/site-packages/salt/output/highstate.py", line 200, in _format_host
    return '\n'.join(hstrs), changed
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 18384: ordinal not in range(128)

[mpaolini]

@kylegato exactly like the issue I found previously. The whole string formatting usage is just plain broken, and has nothing to do with py3 or py2

[juangarciavargas]

I've got similar issue:

 salt -v "*" test.ping
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/usr/local/bin/salt", line 9, in 
    load_entry_point('salt==2014.7.0', 'console_scripts', 'salt')()
  File "/usr/local/lib/python2.7/site-packages/salt/scripts.py", line 240, in salt_main
    client.run()
  File "/usr/local/lib/python2.7/site-packages/salt/cli/__init__.py", line 186, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/local/lib/python2.7/site-packages/salt/client/__init__.py", line 584, in cmd_cli
    **kwargs):
  File "/usr/local/lib/python2.7/site-packages/salt/client/__init__.py", line 1212, in get_cli_event_returns
    connected_minions = salt.utils.minions.CkMinions(self.opts).connected_ids()
  File "/usr/local/lib/python2.7/site-packages/salt/utils/minions.py", line 418, in connected_ids
    addrs = salt.utils.network.local_port_tcp(int(self.opts['publish_port']))
  File "/usr/local/lib/python2.7/site-packages/salt/utils/network.py", line 882, in local_port_tcp
    ret = remotes_on_local_tcp_port(port)
  File "/usr/local/lib/python2.7/site-packages/salt/utils/network.py", line 982, in remotes_on_local_tcp_port
    data = subprocess.check_output(['lsof', '-i4TCP:{0:d}'.format(port), '-n'])
  File "/usr/local/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/usr/local/bin/salt", line 9, in 
    load_entry_point('salt==2014.7.0', 'console_scripts', 'salt')()
  File "/usr/local/lib/python2.7/site-packages/salt/scripts.py", line 240, in salt_main
    client.run()
  File "/usr/local/lib/python2.7/site-packages/salt/cli/__init__.py", line 186, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/local/lib/python2.7/site-packages/salt/client/__init__.py", line 584, in cmd_cli
    **kwargs):
  File "/usr/local/lib/python2.7/site-packages/salt/client/__init__.py", line 1212, in get_cli_event_returns
    connected_minions = salt.utils.minions.CkMinions(self.opts).connected_ids()
  File "/usr/local/lib/python2.7/site-packages/salt/utils/minions.py", line 418, in connected_ids
    addrs = salt.utils.network.local_port_tcp(int(self.opts['publish_port']))
  File "/usr/local/lib/python2.7/site-packages/salt/utils/network.py", line 882, in local_port_tcp
    ret = remotes_on_local_tcp_port(port)
  File "/usr/local/lib/python2.7/site-packages/salt/utils/network.py", line 982, in remotes_on_local_tcp_port
    data = subprocess.check_output(['lsof', '-i4TCP:{0:d}'.format(port), '-n'])
  File "/usr/local/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/local/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

[mpaolini]

@juangarciavargas that's a different issue. You don't seem to have lsof installed. How did you install salt?

(by the way: sudo apt-get install lsof should fix your issue)

[juangarciavargas]

@mpaolini Great! thanks all good now. The problem started after I upgrade the minions to Freebsd10.1
Installed lsof in master and reinstall salt in minions.

[joshmule]

So in what version is this (or will this be) fixed?

[basepi]

The fix in #18376 is in 2014.7.1.

[markuskramerIgitt]

The current implementation of sdecode loops through encodings, which cannot be a solution, and it even fails for Windows. Please use sys.stdin.encoding, as suggested in #19166.

I would like to discuss why the encoding is not part of the grain or pillar.

[eliasp]

Running into this by just using a unicode char in a comment of a YAML/SLS on 2014.7.1/2014.7.2 where it was supposed to be fixed through #18376:

# RHEL7/CentOS7 ships the quite outdated systemd version 208
# The RedHat developer Lukáš Nykrýn provides a COPR with newer versions
{% if grains['os'] in ('RedHat', 'CentOS') %}
systemd-copr:
  pkgrepo.managed:
    - humanname:           Builds of newer systemd versions for CentOS7/RHEL7
    - baseurl:             http://copr-be.cloud.fedoraproject.org/results/lnykryn/systemd/{{ repotype }}-{{ salt['grains.get']('osmajorrelease', 'unknown') }}-$basearch/
    - gpgkey:              https://copr-be.cloud.fedoraproject.org/results/lnykryn/systemd/pubkey.gpg
    - gpgcheck:            1  
    - skip_if_unavailable: True
    - enabled:             1  
    - order:               1  
{% endif %}

state.highstate output:

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 225: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 241, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 190, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 244, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/dist-packages/salt/output/__init__.py", line 44, in display_output
    display_data = get_printout(out, opts)(data).rstrip()
  File "/usr/lib/python2.7/dist-packages/salt/output/highstate.py", line 74, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/dist-packages/salt/output/highstate.py", line 102, in _format_host
    .format(hcolor, err, colors)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 225: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 241, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 190, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 244, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/dist-packages/salt/output/__init__.py", line 44, in display_output
    display_data = get_printout(out, opts)(data).rstrip()
  File "/usr/lib/python2.7/dist-packages/salt/output/highstate.py", line 74, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/dist-packages/salt/output/highstate.py", line 102, in _format_host
    .format(hcolor, err, colors)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 225: ordinal not in range(128)

               Salt: 2014.7.1
             Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
             Jinja2: 2.7.2
           M2Crypto: 0.21.1
     msgpack-python: 0.3.0
       msgpack-pure: Not Installed
           pycrypto: 2.6.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.0.1
               RAET: Not Installed
                ZMQ: 4.0.4
               Mako: 0.9.1

OS: Ubuntu 14.04.2

Minion:

               Salt: 2014.7.2
             Python: 2.7.5 (default, Jun 17 2014, 18:11:42)
             Jinja2: 2.7.2
           M2Crypto: 0.21.1
     msgpack-python: 0.4.4
       msgpack-pure: Not Installed
           pycrypto: 2.6.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.3.1
               RAET: Not Installed
                ZMQ: 3.2.5
               Mako: Not Installed

OS: CentOS 7

[rallytime]

Thanks for the update @eliasp. I've removed the "Fixed Pending Verification" label.

[ghost]

This may be a locale issue.
Salt sets the locale to LC_ALL=C but running on the minion directly had LC_ALL="en_US.UTF-8".
A cmd.run that previously failed with a "UnicodeDecodeError: 'ascii' codec can't decode byte" error succeeded after explicitly setting the locale as an env variable.

         - env:
            - LANG: 'en_US.UTF-8'
            - LC_ALL: 'en_US.UTF-8'

[knuta]

At least some of these are due to mixing the unicode strings (type 'unicode') and byte strings (type 'str'). Consider the following example:

>>> '\n'.join([u'å', u'å'])
u'\xe5\n\xe5'
>>> '\n'.join(['å', 'å'])
'\xc3\xa5\n\xc3\xa5'
>>> '\n'.join([u'å', 'å'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

In the case of highstate failing on "return '\n'.join(hstrs), changed" (one of the examples above), all input save one were byte strings, but the "Changes" part is a unicode string. This small tweak standardizing on str (the smallest change) made my run succeed (on 2014.1.13):

--- /usr/lib/python2.7/dist-packages/salt/output/highstate.py   2015-10-05 02:26:49.000000000 +0200
+++ /usr/lib/python2.7/dist-packages/salt/output/highstate.py   2015-10-05 02:27:03.000000000 +0200
@@ -138,7 +138,7 @@
             hstrs.extend([sline.format(**svars) for sline in state_lines])
             changes = '     Changes:   ' + ctext
             hstrs.append(
-                u'{0}{1}{2[ENDC]}'.format(tcolor, changes, colors)
+                '{0}{1}{2[ENDC]}'.format(tcolor, changes, colors)
             )

         # Append result counts to end of output

I haven't tried the latest version yet, I'll try to whip up a pull request if it's still an issue there.