Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Failing on Python 2.7 + Travis CI with unicode error #1234

Closed
LevN0 opened this issue Mar 30, 2019 · 3 comments
Closed

[BUG] Failing on Python 2.7 + Travis CI with unicode error #1234

LevN0 opened this issue Mar 30, 2019 · 3 comments

Comments

@LevN0
Copy link

@LevN0 LevN0 commented Mar 30, 2019

I ran tests a few days ago, prior to release of Tox 3.8, that were passing. Running test suite on basically identical software started failing with a Unicode Error in Tox,

https://travis-ci.org/LevN0/pds4_tools/jobs/513303961

File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tox/action.py", line 107, in popen
lines = out_path.read().split("\n")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7297: ordinal not in range(128)

Possibly connected to having unicode characters in a filename with-in the package.

Only fails on Python 2.7, succeeds on all Python 3 builds.

@sampsyo
Copy link

@sampsyo sampsyo commented Mar 30, 2019

We're running into this too: https://travis-ci.org/beetbox/beets/jobs/513379693

The problem occurs with Tox 3.8.2 and 3.8.3 but not 3.8.1. Specifically, I believe this change introduced the bug: b5105b1#diff-57c0f90ac19d5a79d6495ab22fab6539R107

The code used to just do this:

output = out_path.read()

But now it does this:

lines = out_path.read().split("\n")
output = "\n".join(lines[3:])

Because that file has unicode_literals enabled, those "\n" string literals are Unicode strings. But because we're on Python 2, the result of read()ing a file is a bytestring.

I think the right fix is to use bytestrings for those "\n" literals on Python 2 but Unicode strings on Python 3. This is sometimes called "native strings," and unicode_literals makes this hard. Future has as_native_str, but I don't (on a cursory glance) see an equivalent utility in six.

sampsyo added a commit to beetbox/beets that referenced this issue Mar 30, 2019
To avoid a Python 2 bug introduced in tox 3.8.2 but not fixed yet:
tox-dev/tox#1234
@asottile
Copy link
Member

@asottile asottile commented Mar 31, 2019

@sampsyo no need for a special library, to get a native-literal just use str("\n"). however I took a different approach in #1237 (use the 3.x behaviour in python2 as well)

@sampsyo
Copy link

@sampsyo sampsyo commented Mar 31, 2019

Excellent point, and yes, your fix is better. 👍

@helpr helpr bot added the pr-merged label Apr 1, 2019
@tox-dev tox-dev locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants