UnicodeDecodeError when running command

[oafilipoai]

I get the following when I run a command from a dodo.py file:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/utils/python/2.7.6/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/home/utils/python/2.7.6/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/home/utils/python/2.7.6/lib/python2.7/site-packages/doit/action.py", line 142, in print_process_output
line = input.readline().decode('utf-8')
File "/home/utils/python/2.7.6/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 10: invalid start byte

It seems that doit cannot handle non utf8 encoded output. In my case, the tool I am running outputs the copyright symbol to stdout.

I attached a sample of the failing setup. Please change the extention to *.tgz and unpack it. I could not figure out how to attach a non-image file to the issue report.

[schettino72]

I did 2 changes:

1- you can explicitly specify the encoding of your process output like:

from doit.action import CmdAction

    return { ...
                'actions': [CmdAction('cat sample.txt', encoding='iso-8859-1')]}

2- doit doesnt use the output of the process (it only captures it and re-display it).
I suspect you don't care so much about it. So I changed the default behaviour of
the process output decoding from strict to replace.

So now by default you will get � instead of errors (even the cat command does the same).
If you want a strict decoding you can also specify it to CmdAction in the parameter decode_error.

Check the test cases from the commit.

[oafilipoai]

I verified and the problem is fixed in the latest checkin

[schettino72]

@oafilipoai , so are you specifying the encoding or replacing errors with question mark?

[oafilipoai]

Just replaced the errors. I do not care about the missing characters as long as the output is readable and the script does not crash.