python3: encoding error

[timblechmann]

when using clcache/master via python 3, i encountered the following error:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python34\lib\threading.py", line 920, in _bootstrap_inner
  File "C:\Python34\lib\threading.py", line 868, in run
  File "C:\Python34\lib\subprocess.py", line 1171, in _readerthread
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 161099: character maps to <undefined>

Traceback (most recent call last):
  File "clcache.py", line 1200, in <module>
  File "clcache.py", line 1100, in main
  File "clcache.py", line 1166, in processCompileRequest
  File "clcache.py", line 1026, in processNoManifestMiss
  File "clcache.py", line 769, in invokeRealCompiler
  File "C:\Python34\lib\subprocess.py", line 960, in communicate
  File "C:\Python34\lib\subprocess.py", line 1235, in _communicate
IndexError: list index out of range

it seems to be related to the fact that the compiler is using a german locale :(

[webmaster128]

Is this one error or two different ones? I do not yet understand at which code position the problem occurs.

Can you tell whether it is because of a special character in a source file or a file path?

[timblechmann]

one error ... since it only occurs when using german language, i'd assume that its a special char in the command line output of the compiler?

[webmaster128]

Okay, here we go: http://stackoverflow.com/a/33291200

Problem is that we do not know which encoding MSVC is throwing at us in STDOUT and STDERR. It is likely to be this semi-UTF-16 encoding called MBCS/DBCS (Microsoft's definition of DBCS).

It is true that 0x81 is not Windows-1251/CP-1251 because the position is empty.

So as described in the SO answer, in invokeRealCompiler() we should not use Popen with universal_newlines=True when calling communicate(). Instead we should take the plain bytes and do not convert them into strings. This is already done ObjectCache.computeKey().

[webmaster128]

I think this might help: #117