[Fix] s3 ls command fails when UTF-8 is piped
#1844
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| uni_print(u'SomeChars\u2713\u2714OtherChars', out) | ||
| # Unicode characters get replaced with their | ||
| # UTF-8 byte value. | ||
| self.assertEqual(buf.getvalue(), b'SomeChars\xe2\x9c\x93\xe2\x9c\x94OtherChars') |
There was a problem hiding this comment.
I'm probably missing something, but how does this get utf-8 encoded if your code change above sets the default to ascii?
There was a problem hiding this comment.
It was just an observation – I actually don't know :S – maybe it's a Python default when encoding unicode strings to ascii.
There was a problem hiding this comment.
Okay. I found out (I think). TextIOWrapper sets encoding to UTF-8 when it's passed encoding=None, so this test never hits the UnicodeEncodeError branch of uni_print.
I'm gonna update this branch to fix it.
Piping the output of the `aws s3 ls` command when used on a bucket that has keys with UTF-8 characters in it, will raise an exception, exit the program and not list all files. E.g. `aws s3 ls s3://<BUCKET-WITH-UTF8> | wc -l` will print the error `encode() argument 1 must be string, not None` and exit, upon reaching the UTF-8 key. It fails because when stdout is being piped `sys.stdout.encoding` is `None`.
malthejorgensen
force-pushed
the
fix-s3-pipe-encoding-none
branch
from
35d60d1
to
e1c35c2
Compare
|
Thanks for updating. Looks good to me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Piping the output of the
aws s3 lscommand when used on a bucket that has keys with UTF-8 characters in it, will raise an exception, exit the program and not list all files.E.g.
aws s3 ls s3://<BUCKET-WITH-UTF8> | wc -lwill print the errorencode() argument 1 must be string, not Noneand exit, upon reaching the UTF-8 key.It fails because when stdout is being piped
sys.stdout.encodingisNone.I am aware that the
PYTHONIOENCODINGenvironment variable can be set in order to change that, but it seems that the current code is trying to default toasciiand simply failing to default: awscli/customizations/s3/utils.py, line 398 (latest commit on develop)