UnicodeDecodeError: 'utf8' codec can't decode byte 0xda in position 83: invalid continuation byte

[gerardjp]

Hi All,

As suggested by @cachedout, hereby my utf8 encode error during a custom deb package install. It's a reproducible error.

The full error and stacktrace:

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xda in position 83: invalid continuation byte
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 240, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 189, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 243, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/dist-packages/salt/output/__init__.py", line 36, in display_output
    display_data = get_printout(out, opts)(data).rstrip()
  File "/usr/lib/python2.7/dist-packages/salt/output/nested.py", line 123, in output
    return nest.display(ret, __opts__.get('nested_indent', 0), '', '')
  File "/usr/lib/python2.7/dist-packages/salt/output/nested.py", line 114, in display
    out = self.display(val, indent + 4, '', out)
  File "/usr/lib/python2.7/dist-packages/salt/output/nested.py", line 114, in display
    out = self.display(val, indent + 4, '', out)
  File "/usr/lib/python2.7/dist-packages/salt/output/nested.py", line 114, in display
    out = self.display(val, indent + 4, '', out)
  File "/usr/lib/python2.7/dist-packages/salt/output/nested.py", line 114, in display
    out = self.display(val, indent + 4, '', out)
  File "/usr/lib/python2.7/dist-packages/salt/output/nested.py", line 89, in display
    prefix=prefix)
  File "/usr/lib/python2.7/dist-packages/salt/output/nested.py", line 58, in ustring
    indent, color, prefix, msg.decode(encoding), endc, suffix)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)

I'm running salt 2014.7.0rc3 (Helium)

Problem origin and solution

It seems it has todo with the output of a package being installed with dpkg. It asks about some settings in color.

• Package in question: ossec-hids-agent - 2.8.1-ubuntu3
• Taken from: https://launchpad.net/~nicolas-zin/+archive/ubuntu/ossec-ubuntu/+packages
• placed/installed straight from salt

DEBIAN_FRONTEND=noninteractive dpkg -i /tmp/ossec-hids-agent_2.8.1-ubuntu3_amd64.deb

Suppressing with the DEBIAN_FRONTEND=noninteractive doesn't help. It breaks (i think) because it needs the parameters defined upfront. This can be done with:

echo 'ossec-hids-agent  ossec-hids-agent/serverip   string  127.0.0.1' | debconf-set-selections

Workflow wise setting the presets makes it workable, however, I think de Salt guru's 😉 would like to capture the utf8 error.

The actual full error in the minion log is several screens long and contains sensitive data. It can be provided on request.

Questions welcome.

Kind regards,

Gerard.

[gerardjp]

About the logs .. This, partial, is the directly return from the command (stdout):

Configuring ossec-hids-agentESC[34m ESC[11mESC[34mESC[40m������������������ĿESC[8;6H�ESC[0;10mESC[10mESC[33mESC[40m ESC[36mESC[42mYou must specify an IP and not a hostname

And by the time it's gone through the salt logger it's encoded:

Configuring ossec-hids-agent\x1b[34m \x1b[11m\x1b[34m\x1b[40m\xc3\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xc4\xbf\x1b[8;6H\xb3\x1b[0;10m\x1b[10m\x1b[33m\x1b[40m \x1b[36m\x1b[42mYou must specify an IP and not a hostname

Hope it helps

[s0undt3ch]

@cachedout, we might need to create a console reader which should be able to strip colors on request and convert the system's output from whichever encoding it is set to Unicode at which point we can do whatever we want with that output...

[cachedout]

@gerardjp Thanks for filing this issue. We'll start investigating!

@s0undt3ch Yes, I think we're approaching the point where something like that makes a lot of sense.

[gerardjp]

Is it possible it has to do with how I formulate my service definition?

nagios3:
  service:
    - running
    - watch:
      - file: /etc/nagios3/*
      - file: /etc/nagios3/conf.d/*

When a file changes, I get the error. The file is updated though because running a second time the error goes away.

Just thought I keep posting symptoms 😄