Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
HDDS-657. Remove {volume} path segments from all the remaining rest e…
…ndpoints. Contributed by Elek, Marton.
  • Loading branch information
elek committed Oct 16, 2018
1 parent 0bf8a11 commit 0c2914e
Show file tree
Hide file tree
Showing 38 changed files with 899 additions and 826 deletions.
27 changes: 27 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/s3/README.md
@@ -0,0 +1,27 @@
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

## Ozone S3 Gatway Acceptance Tests

Note: the aws cli based acceptance tests can be cross-checked with the original AWS s3 endpoint.

You need to

1. Create a bucket
2. Configure your local aws cli
3. Set bucket/endpointurl during the robot test execution

```
robot -v bucket:ozonetest -v OZONE_S3_SET_CREDENTIALS:false -v ENDPOINT_URL:https://s3.us-east-2.amazonaws.com smoketest/s3
```
21 changes: 21 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/s3/__init__.robot
@@ -0,0 +1,21 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

*** Settings ***
Library OperatingSystem
Library String
Resource ../commonlib.robot
Resource ./commonawslib.robot
Test Setup Setup s3 tests
Expand Up @@ -18,46 +18,30 @@ Documentation S3 gateway test with aws cli
Library OperatingSystem Library OperatingSystem
Library String Library String
Resource ../commonlib.robot Resource ../commonlib.robot
Resource ./commonawslib.robot
Suite Setup Setup s3 tests


*** Variables *** *** Variables ***
${ENDPOINT_URL} http://s3g:9878 ${ENDPOINT_URL} http://s3g:9878

${BUCKET} generated
*** Keywords ***
Execute AWSCli
[Arguments] ${command}
${output} = Execute aws s3 --endpoint-url ${ENDPOINT_URL}/${VOLUME} ${command}
[return] ${output}


*** Test Cases *** *** Test Cases ***


Create volume and bucket for the tests
${postfix} = Generate Random String 5 [NUMBERS]
Set Suite Variable ${BUCKET} bucket-${postfix}
Set Suite Variable ${VOLUME} vol-${postfix}
Log Testing s3 commands in /${VOLUME}/${BUCKET}
${result} = Execute ozone sh volume create /${VOLUME} --user hadoop
${result} = Execute ozone sh bucket create /${VOLUME}/${BUCKET}

Install aws s3 cli
Execute sudo apt-get install -y awscli
Set Environment Variable AWS_ACCESS_KEY_ID ANYID
Set Environment Variable AWS_SECRET_ACCESS_KEY ANYKEY

File upload and directory list File upload and directory list
Execute date > /tmp/testfile Execute date > /tmp/testfile
${result} = Execute AWSCli cp /tmp/testfile s3://${BUCKET} ${result} = Execute AWSS3Cli cp /tmp/testfile s3://${BUCKET}
Should contain ${result} upload Should contain ${result} upload
${result} = Execute AWSCli cp /tmp/testfile s3://${BUCKET}/dir1/dir2/file ${result} = Execute AWSS3Cli cp /tmp/testfile s3://${BUCKET}/dir1/dir2/file
Should contain ${result} upload Should contain ${result} upload
${result} = Execute AWSCli ls s3://${BUCKET} ${result} = Execute AWSS3Cli ls s3://${BUCKET}
Should contain ${result} testfile Should contain ${result} testfile
Should contain ${result} dir1 Should contain ${result} dir1
Should not contain ${result} dir2 Should not contain ${result} dir2
${result} = Execute AWSCli ls s3://${BUCKET}/dir1/ ${result} = Execute AWSS3Cli ls s3://${BUCKET}/dir1/
Should not contain ${result} testfile Should not contain ${result} testfile
Should not contain ${result} dir1 Should not contain ${result} dir1
Should contain ${result} dir2 Should contain ${result} dir2
${result} = Execute AWSCli ls s3://${BUCKET}/dir1/dir2/ ${result} = Execute AWSS3Cli ls s3://${BUCKET}/dir1/dir2/
Should not contain ${result} testfile Should not contain ${result} testfile
Should not contain ${result} dir1 Should not contain ${result} dir1
Should contain ${result} file Should contain ${result} file
34 changes: 34 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/s3/buckethead.robot
@@ -0,0 +1,34 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

*** Settings ***
Documentation S3 gateway test with aws cli
Library OperatingSystem
Library String
Resource ../commonlib.robot
Resource commonawslib.robot
Test Setup Setup s3 tests

*** Variables ***
${ENDPOINT_URL} http://s3g:9878
${BUCKET} generated

*** Test Cases ***

Head Bucket not existent
${result} = Execute AWSS3APICli head-bucket --bucket ${BUCKET}
${result} = Execute AWSS3APICli and checkrc head-bucket --bucket ozonenosuchbucketqqweqwe 255
Should contain ${result} Bad Request
Should contain ${result} 400
32 changes: 32 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/s3/bucketlist.robot
@@ -0,0 +1,32 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

*** Settings ***
Documentation S3 gateway test with aws cli
Library OperatingSystem
Library String
Resource ../commonlib.robot
Resource commonawslib.robot
Test Setup Setup s3 tests

*** Variables ***
${ENDPOINT_URL} http://s3g:9878
${BUCKET} generated

*** Test Cases ***

List buckets
${result} = Execute AWSS3APICli list-buckets | jq -r '.Buckets[].Name'
Should contain ${result} ${BUCKET}
66 changes: 0 additions & 66 deletions hadoop-ozone/dist/src/main/smoketest/s3/bucketv2.robot

This file was deleted.

71 changes: 0 additions & 71 deletions hadoop-ozone/dist/src/main/smoketest/s3/bucketv4.robot

This file was deleted.

43 changes: 41 additions & 2 deletions hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot
Expand Up @@ -16,13 +16,52 @@
*** Settings *** *** Settings ***
Resource ../commonlib.robot Resource ../commonlib.robot


*** Variables ***
${OZONE_S3_HEADER_VERSION} v2
${OZONE_S3_SET_CREDENTIALS} true

*** Keywords *** *** Keywords ***
Execute AWSS3APICli Execute AWSS3APICli
[Arguments] ${command} [Arguments] ${command}
${output} = Execute aws s3api --endpoint-url ${ENDPOINT_URL} ${command} ${output} = Execute aws s3api --endpoint-url ${ENDPOINT_URL} ${command}
[return] ${output} [return] ${output}


Execute AWSS3APICli and checkrc Execute AWSS3APICli and checkrc
[Arguments] ${command} ${expected_error_code} [Arguments] ${command} ${expected_error_code}
${output} = Execute and checkrc aws s3api --endpoint-url ${ENDPOINT_URL} ${command} ${expected_error_code} ${output} = Execute and checkrc aws s3api --endpoint-url ${ENDPOINT_URL} ${command} ${expected_error_code}
[return] ${output} [return] ${output}

Execute AWSS3Cli
[Arguments] ${command}
${output} = Execute aws s3 --endpoint-url ${ENDPOINT_URL} ${command}
[return] ${output}

Install aws cli
${rc} ${output} = Run And Return Rc And Output which apt-get
Run Keyword if '${rc}' == '0' Install aws cli s3 debian

Install aws cli s3 debian
Execute sudo apt-get install -y awscli

Setup v2 headers
Set Environment Variable AWS_ACCESS_KEY_ID ANYID
Set Environment Variable AWS_SECRET_ACCESS_KEY ANYKEY

Setup v4 headers
Execute aws configure set default.s3.signature_version s3v4
Execute aws configure set aws_access_key_id default1
Execute aws configure set aws_secret_access_key defaultsecret
Execute aws configure set region us-west-1
Create bucket
${postfix} = Generate Random String 5 [NUMBERS]
Set Suite Variable ${BUCKET} bucket-${postfix}
Execute AWSS3APICli create-bucket --bucket ${BUCKET}

Setup credentials
Run Keyword if '${OZONE_S3_HEADER_VERSION}' == 'v4' Setup v4 headers
Run Keyword if '${OZONE_S3_HEADER_VERSION}' != 'v4' Setup v2 headers

Setup s3 tests
Run Keyword Install aws cli
Run Keyword if '${OZONE_S3_SET_CREDENTIALS}' == 'true' Setup credentials
Run Keyword if '${BUCKET}' == 'generated' Create bucket

0 comments on commit 0c2914e

Please sign in to comment.