Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore paramfile for s3 --website-redirect #1137

Merged
merged 2 commits into from
Feb 10, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ CHANGELOG
Next Release (TBD)
==================

* bugfix:``aws s3``: Fix issue where literal value for ``--website-redirect``
was not being used.
(`issue 1137 <https://github.com/aws/aws-cli/pull/1137>`__)
* bugfix:``aws sqs purge-queue``: Fix issue with the processing
of the ``--queue-url`` parameter
(`issue 1126 <https://github.com/aws/aws-cli/issues/1126>`__)
Expand Down
6 changes: 6 additions & 0 deletions awscli/paramfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,12 @@
'cloudformation.update-stack.stack-policy-url',
'cloudformation.set-stack-policy.stack-policy-url',

# We will want to change the event name to ``s3`` as opposed to
# custom in the near future along with ``s3`` to ``s3api``.
'custom.cp.website-redirect',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a side note, I do not like that all custom commands have custom as its service in the event that is emitted. It is not good if there are ever custom commands that overlap in the cli namespace even though they may have a different command lineage.

However, I do not think any change can be made till after the command lineage PR is merged.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let's get the command lineage PR merged so we can update this PR to avoid custom.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. This change may be a little larger than expected (a change in the clidriver and custom commands). If you look further down in the list of black listed arguments, the s3api commands that use --website-redirect have the event listed as s3...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: the change for the data driven commands need to come in arguments.py

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for all the spamming. Actually a change will need to take place at unpack_argument in argprocess.py: https://github.com/aws/aws-cli/blob/develop/awscli/argprocess.py#L63

The reason being that the service_name/operation_name input limits all commands to the strict two command length structure.

For example, for the custom top layer s3 command, a custom had to be added to fit the input because both service_name and operation_name are required. This is an issue because if I wanted to apply a handler to the event custom.s3.*.some-argument I would not be able to register to all of s3's operation commands because s3's operation commands must emit s3.operation.some-argument or custom.operation.some-argument to fit the inputs to that function.

This is also an issue for commands that have more than two commands like waiters.

I propose to leave the PR as is and make this change in a subsequent PR because it is outside the scope of the current bug fix as I will have to change the clidriver and some handlers as well.

@jamesls Let me know what you think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference is to just put in the long term fix rather than a temporary fix that will be removed once the changes are plumbed through clidriver/handlers.

However, I'm ok with putting this fix in for the time being. Code and test themselves look good. Feel free to merge if you want.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a reminder to fix it in my latest commit.

'custom.mv.website-redirect',
'custom.sync.website-redirect',

'sqs.add-permission.queue-url',
'sqs.change-message-visibility.queue-url',
'sqs.change-message-visibility-batch.queue-url',
Expand Down
19 changes: 18 additions & 1 deletion tests/integration/customizations/s3/test_plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,12 +162,16 @@ def list_buckets(self):
return parsed['Buckets']

def content_type_for_key(self, bucket_name, key_name):
parsed = self.head_object(bucket_name, key_name)
return parsed['ContentType']

def head_object(self, bucket_name, key_name):
operation = self.service.get_operation('HeadObject')
endpoint = self.service.get_endpoint(self.regions[bucket_name])
http, parsed = operation.call(
endpoint, bucket=bucket_name, key=key_name)
self.assertEqual(http.status_code, 200)
return parsed['ContentType']
return parsed

def assert_no_errors(self, p):
self.assertEqual(
Expand Down Expand Up @@ -497,6 +501,19 @@ def test_download_encrypted_kms_object(self):
with open(local_filename, 'r') as f:
self.assertEqual(f.read(), contents)

def test_website_redirect_ignore_paramfile(self):
bucket_name = self.create_bucket()
foo_txt = self.files.create_file('foo.txt', 'bar')
website_redirect = 'http://someserver'
p = aws('s3 cp %s s3://%s/foo.txt --website-redirect %s' %
(foo_txt, bucket_name, website_redirect))
self.assert_no_errors(p)

# Ensure that the web address is used as opposed to the contents
# of the web address. We can check via a head object.
response = self.head_object(bucket_name, 'foo.txt')
self.assertEqual(response['WebsiteRedirectLocation'], website_redirect)


class TestSync(BaseS3CLICommand):
def test_sync_with_plus_chars_paginate(self):
Expand Down
13 changes: 13 additions & 0 deletions tests/unit/customizations/s3/test_cp_command.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,19 @@ def test_operations_used_in_recursive_download(self):
self.assertEqual(len(self.operations_called), 1, self.operations_called)
self.assertEqual(self.operations_called[0][0].name, 'ListObjects')

def test_website_redirect_ignore_paramfile(self):
full_path = self.files.create_file('foo.txt', 'mycontent')
cmdline = '%s %s s3://bucket/key.txt --website-redirect %s' % \
(self.prefix, full_path, 'http://someserver')
self.parsed_responses = [{'ETag': '"c8afdb36c52cf4727836669019e69222"'}]
self.run_cmd(cmdline, expected_rc=0)
# Make sure that the specified web address is used as opposed to the
# contents of the web address.
self.assertEqual(
self.operations_called[0][1]['website_redirect_location'],
'http://someserver'
)


if __name__ == "__main__":
unittest.main()
14 changes: 14 additions & 0 deletions tests/unit/customizations/s3/test_mv_command.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@ def test_cant_mv_object_with_implied_name(self):
stderr = self.run_cmd(cmdline, expected_rc=255)[1]
self.assertIn('Cannot mv a file onto itself', stderr)

def test_website_redirect_ignore_paramfile(self):
full_path = self.files.create_file('foo.txt', 'mycontent')
cmdline = '%s %s s3://bucket/key.txt --website-redirect %s' % \
(self.prefix, full_path, 'http://someserver')
self.parsed_responses = [{'ETag': '"c8afdb36c52cf4727836669019e69222"'}]
self.run_cmd(cmdline, expected_rc=0)
self.assertEqual(self.operations_called[0][0].name, 'PutObject')
# Make sure that the specified web address is used as opposed to the
# contents of the web address.
self.assertEqual(
self.operations_called[0][1]['website_redirect_location'],
'http://someserver'
)


if __name__ == "__main__":
unittest.main()
51 changes: 51 additions & 0 deletions tests/unit/customizations/s3/test_sync_command.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Copyright 2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
from awscli.testutils import BaseAWSCommandParamsTest, FileCreator
import re

import mock
from awscli.compat import six


class TestSyncCommand(BaseAWSCommandParamsTest):

prefix = 's3 sync '

def setUp(self):
super(TestSyncCommand, self).setUp()
self.files = FileCreator()

def tearDown(self):
super(TestSyncCommand, self).tearDown()
self.files.remove_all()

def test_website_redirect_ignore_paramfile(self):
full_path = self.files.create_file('foo.txt', 'mycontent')
cmdline = '%s %s s3://bucket/key.txt --website-redirect %s' % \
(self.prefix, self.files.rootdir, 'http://someserver')
self.parsed_responses = [
{"CommonPrefixes": [], "Contents": []},
{'ETag': '"c8afdb36c52cf4727836669019e69222"'}
]
self.run_cmd(cmdline, expected_rc=0)

# The only operations we should have called are ListObjects/PutObject.
self.assertEqual(len(self.operations_called), 2, self.operations_called)
self.assertEqual(self.operations_called[0][0].name, 'ListObjects')
self.assertEqual(self.operations_called[1][0].name, 'PutObject')
# Make sure that the specified web address is used as opposed to the
# contents of the web address when uploading the object
self.assertEqual(
self.operations_called[1][1]['website_redirect_location'],
'http://someserver'
)