Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeDecodeError obfuscates authentication error #708

Closed
onyxfish opened this issue Mar 19, 2014 · 5 comments · Fixed by boto/botocore#711
Closed

UnicodeDecodeError obfuscates authentication error #708

onyxfish opened this issue Mar 19, 2014 · 5 comments · Fixed by boto/botocore#711
Labels

Comments

@onyxfish
Copy link

@onyxfish onyxfish commented Mar 19, 2014

While helping debug issues a coworker was having I discovered this UnicodeDecodeError was masking an auth error.

2014-03-19 12:07:55,125 - awscli.customizations.s3.s3handler - DEBUG - Exception caught during task execution: 'ascii' codec can't decode byte 0xe2 in position 24: ordinal not in range(128)
Traceback (most recent call last):
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/awscli/customizations/s3/http://s3handler.py", line 76, in call
    total_files, total_parts = self._enqueue_tasks(files)
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/awscli/customizations/s3/http://s3handler.py", line 154, in _enqueue_tasks
    for filename in files:
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/awscli/customizations/s3/http://comparator.py", line 90, in call
    dest_file = advance_iterator(dest_files)
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/awscli/customizations/s3/http://filters.py", line 87, in call
    for file_info in file_infos:
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/awscli/customizations/s3/http://filegenerator.py", line 76, in call
    for src_path, size, last_update in file_list:
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/awscli/customizations/s3/http://filegenerator.py", line 167, in list_objects
    for html_response, response_data in iterator:
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/botocore/http://paginate.py", line 141, in __iter__
    **current_kwargs)
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/botocore/http://operation.py", line 68, in call
    response = endpoint.make_request(self, params)
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/botocore/http://endpoint.py", line 75, in make_request
    prepared_request = self.prepare_request(request, do_auth)
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/botocore/http://endpoint.py", line 91, in prepare_request
    self.auth.add_auth(request=request)
  File "/Users/quoctrungbui/.virtualenvs/dailygraphics/lib/python2.7/site-packages/botocore/http://auth.py", line 451, in add_auth
    signature))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 24: ordinal not in range(128)
2014-03-19 12:07:55,128 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2014-03-19 12:07:55,128 - awscli.customizations.s3.executor - DEBUG - Received print task: {'message': "'ascii' codec can't decode byte 0xe2 in position 24: ordinal not in range(128)", 'error': True}

The real issue was that when he pasted his AWS credentials into his .bash_profile he had inadvertently pasted them onto the same line.

@jamesls

This comment has been minimized.

Copy link
Member

@jamesls jamesls commented Mar 19, 2014

Can you share a set of steps to repro this error? Do you mean that the bash profile looked something like:

export AWS_ACCESS_KEY_ID=foo export AWS_SECRET_ACCESS_KEY=bar
@onyxfish

This comment has been minimized.

Copy link
Author

@onyxfish onyxfish commented Mar 19, 2014

Yep, that's exactly what it looked like. Sorry I forgot to include the original command, here it is:

aws s3 sync .gzip/graphics/pizza s3://stage-apps.npr.org/dailygraphics/graphics/pizza --acl "public-read" --exclude "*.html" --exclude "*.js" --exclude "*.json" --exclude "*.css" --exclude "*.xml"  --cache-control "max-age=5" --region "us-east-1
@danielgtaylor

This comment has been minimized.

Copy link
Contributor

@danielgtaylor danielgtaylor commented Mar 19, 2014

I can reproduce this, but only by introducing a unicode character into the exported variable. For example:

$ export AWS_ACCESS_KEY_ID=foo export AWS_SECRET_ACCESS_KEY=bar
$ aws s3 ls

A client error (InvalidAccessKeyId) occurred when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

$ export AWS_ACCESS_KEY_ID=fooüexport AWS_SECRET_ACCESS_KEY=bar
$ aws s3 ls

'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

Running with --debug:

$ aws --version
aws-cli/1.3.1 Python/2.7.5 Darwin/13.1.0

$ aws s3 ls --debug
2014-03-19 11:40:49,478 - awscli.clidriver - DEBUG - CLI version: aws-cli/1.3.1 Python/2.7.5 Darwin/13.1.0, botocore version: 0.35.0
2014-03-19 11:40:49,479 - botocore.service - DEBUG - Creating service object for: s3
2014-03-19 11:40:49,480 - botocore.base - DEBUG - Attempting to load: aws/s3
2014-03-19 11:40:49,508 - botocore.base - DEBUG - Found data file: /Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/data/aws/s3.json
2014-03-19 11:40:49,508 - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function signature_overrides at 0x1035f8398>
2014-03-19 11:40:49,508 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x1035f81b8>
2014-03-19 11:40:49,508 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2014-03-19 11:40:49,508 - botocore.hooks - DEBUG - Event creating-endpoint.s3: calling handler <function maybe_switch_to_s3sigv4 at 0x1035f82a8>
2014-03-19 11:40:49,509 - botocore.credentials - INFO - Found credentials in Environment variables.
2014-03-19 11:40:49,571 - botocore.service - DEBUG - Creating operation objects for: Service(s3)
2014-03-19 11:40:49,579 - botocore.operation - DEBUG - Operation:ListBuckets called with kwargs: {}
2014-03-19 11:40:49,579 - botocore.operation - DEBUG - Creating parameter objects for: Operation:ListBuckets
2014-03-19 11:40:49,579 - botocore.endpoint - DEBUG - Making request for Operation:ListBuckets (verify_ssl=True) with params: {'headers': {}, 'uri_params': {}, 'payload': <botocore.payload.XMLPayload object at 0x103a13bd0>}
2014-03-19 11:40:49,579 - botocore.endpoint - DEBUG - Building URI for rest endpoint.
2014-03-19 11:40:49,580 - botocore.endpoint - DEBUG - Templated URI path: /
2014-03-19 11:40:49,580 - botocore.endpoint - DEBUG - Templated URI query_params: 
2014-03-19 11:40:49,580 - botocore.endpoint - DEBUG - Rendered path: /
2014-03-19 11:40:49,580 - botocore.endpoint - DEBUG - Rendered query_params: 
2014-03-19 11:40:49,580 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler <function fix_s3_host at 0x1035f80c8>
2014-03-19 11:40:49,581 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/
2014-03-19 11:40:49,581 - botocore.handlers - DEBUG - Not changing URI, bucket is not DNS compatible: 
2014-03-19 11:40:49,581 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2014-03-19 11:40:49,581 - botocore.auth - DEBUG - HTTP request method: GET
2014-03-19 11:40:49,581 - botocore.auth - DEBUG - StringToSign:
GET


Wed, 19 Mar 2014 18:40:49 GMT
/
2014-03-19 11:40:49,582 - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/clidriver.py", line 187, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/customizations/s3/s3.py", line 151, in __call__
    remaining, parsed_globals)
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/customizations/s3/s3.py", line 262, in __call__
    return self._do_command(parsed_args, parsed_globals)
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/customizations/s3/s3.py", line 354, in _do_command
    self._list_all_buckets()
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/customizations/s3/s3.py", line 394, in _list_all_buckets
    response_data = operation.call(self.endpoint)[1]
  File "/Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/operation.py", line 68, in call
    response = endpoint.make_request(self, params)
  File "/Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/endpoint.py", line 75, in make_request
    prepared_request = self.prepare_request(request, do_auth)
  File "/Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/endpoint.py", line 91, in prepare_request
    self.auth.add_auth(request=request)
  File "/Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/auth.py", line 451, in add_auth
    signature))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)
2014-03-19 11:40:49,634 - awscli.clidriver - DEBUG - Exiting with rc 255

'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

The problem appears to be that self.credentials.access_key isn't being decoded before being used, though typically this key contains no unicode characters.

@danielgtaylor danielgtaylor added accepted and removed s3sync labels Mar 19, 2014
@onyxfish

This comment has been minimized.

Copy link
Author

@onyxfish onyxfish commented Mar 19, 2014

I don't have his configuration in front of me, but I didn't notice that it had any unicode characters in it. If it did, they must have been a valid part of the auth key, because it worked correctly once I moved the config onto it's own line...

@danielgtaylor

This comment has been minimized.

Copy link
Contributor

@danielgtaylor danielgtaylor commented Mar 19, 2014

If I had to guess, it may have been some copy/paste error where an existing newline was transformed into some unicode character (possibly one that isn't visible), hence the two exports on the same line. Either way we should be able to fix this so that the correct error is returned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.