UnicodeDecodeError while running setup.py #341
Description
While installing memory_profiler-0.59.0.tar.gz with pip install memory-profiler, I noticed the following error:
>pip install memory-profiler
Collecting memory-profiler
Downloading memory_profiler-0.59.0.tar.gz (38 kB)
Preparing metadata (setup.py) ... error
ERROR: Command errored out with exit status 1:
[...]
Complete output (7 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "[...]\setup.py", line 33, in <module>
long_description=open('README.rst').read(),
File "[...]\Python39\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 14840: character maps to <undefined>
----------------------------------------
I downloaded README.rst from 70615bc and (for me) the offending byte is now at position 15256, the last byte of the "special" closing double quotes ”, with utf-8 encoding \xe2\x80\x9d:
>>> with open("README.rst", "r") as f:
... s = f.read()
...
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 15256: character maps to <undefined>
>>> with open("README.rst", "rb") as f:
... s = f.read()
...
>>> s[15230:15270]
b'aka \xe2\x80\x9cResident Set Size\xe2\x80\x9d. \r\nIn some c'
# this byte here ^^^The issue is that the default encoding for open() is not always utf-8:
>>> with open("README.rst", "r") as f:
... print(f.encoding)
...
cp1252Replacing long_description=open('README.rst').read() with long_description=open('README.rst', encoding='utf-8').read() in setup.py would solve this issue.