TypeError: must be str, not bytes in Python 3.4

[Lucas-C]

Hello,

There is how to reproduce the issue:

$ echo A > dummy.txt
$ python3.4 -c 'import sh, sys; sh.cat(_in=sys.stdin, _out=sys.stdout)' < dummy.txt
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/lucas/.local/lib/python3.4/site-packages/sh.py", line 1484, in output_thread
    done = stream.read()
  File "/home/lucas/.local/lib/python3.4/site-packages/sh.py", line 1974, in read
    self.write_chunk(chunk)
  File "/home/lucas/.local/lib/python3.4/site-packages/sh.py", line 1949, in write_chunk
    self.should_quit = self.process_chunk(chunk)
  File "/home/lucas/.local/lib/python3.4/site-packages/sh.py", line 1847, in process
    handler.write(chunk)
TypeError: must be str, not bytes

However it works OK with Python 2.7.

Any idea on what is happening ?

[amoffat]

This may be a bug, I'm not sure. The code says that if the output handler is a file object, we're going to feed it raw bytes, instead of attempting to decode it to unicode first. I'm not sure I can think of the reasoning why this might be.

Unfortunately, for your case, sys.stdout has not been opened in binary mode, so it cannot accept raw bytes. However, if you change your code to this, it works because the output is decoded to a string/unicode before it is sent to sys.stdout

python3.4 -c 'import sh, sys; print(sh.cat(_in=sys.stdin))' < dummy.txt

I'll leave this issue open and see if I can remember if there is a valid reason for the current behavior

[Lucas-C]

Thanks a lot for the detailed answer, and the workaround !

[amoffat]

After some reviewing, I've determined that the behavior is desired. Python 3 changed so that sys.stdout is opened in text mode and is very strict about what it accepts, which is why it doesn't work as it does in Python 2. Another workaround, from the comments, is:

python3.4 -c 'import sh, sys; sh.cat(_in=sys.stdin, _out=sys.stdout.buffer)' < dummy.txt, where buffer is an underlying binary handle

[Lucas-C]

Perfect, thanks !

[guoew]

I also encountered the same problem, thank you to share