Skip to content
This repository was archived by the owner on Sep 16, 2020. It is now read-only.

make combine_files() work with UTF-8 encoded files#394

Merged
AlanCoding merged 1 commit intoansible:masterfrom
evgeni:non-unicode-setup
Oct 9, 2017
Merged

make combine_files() work with UTF-8 encoded files#394
AlanCoding merged 1 commit intoansible:masterfrom
evgeni:non-unicode-setup

Conversation

@evgeni
Copy link
Contributor

@evgeni evgeni commented Oct 8, 2017

setup.py might fail on Python 3 in non-UTF8 environments while parsing
HISTORY.rst or README.rst which are not required to be ASCII-only with:

UnicodeDecodeError: 'ascii' codec can't decode …

We could also require the files to be plain ASCII, but it sounds not the
right thing to do in the 21st century.

Simple reproducer:

% LC_ALL=C python3 setup.py
Traceback (most recent call last):
  File "setup.py", line 126, in <module>
    long_description=combine_files('README.rst', 'HISTORY.rst'),
  File "setup.py", line 112, in combine_files
    file_contents.append(f.read())
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2132: ordinal not in range(128)

setup.py might fail on Python 3 in non-UTF8 environments while parsing
HISTORY.rst or README.rst which are not required to be ASCII-only with:

    UnicodeDecodeError: 'ascii' codec can't decode …

We could also require the files to be plain ASCII, but it sounds not the
right thing to do in the 21st century.

Simple reproducer:

    % LC_ALL=C python3 setup.py
    Traceback (most recent call last):
      File "setup.py", line 126, in <module>
        long_description=combine_files('README.rst', 'HISTORY.rst'),
      File "setup.py", line 112, in combine_files
        file_contents.append(f.read())
      File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2132: ordinal not in range(128)
@coveralls
Copy link

coveralls commented Oct 8, 2017

Coverage Status

Coverage remained the same at 89.986% when pulling 7566412 on evgeni:non-unicode-setup into 02c2b97 on ansible:master.

@AlanCoding
Copy link
Member

This makes sense to me. I believe the idea is that files that we are providing are implicitly utf-8, and it just needs to be marked as such.

file -I HISTORY.rst 
HISTORY.rst: text/plain; charset=utf-8

@AlanCoding AlanCoding merged commit 79d53d9 into ansible:master Oct 9, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants