Skip to content

Commit

Permalink
HDDS-4238. Test AWS S3 client compatibility with fs incompatible keys
Browse files Browse the repository at this point in the history
  • Loading branch information
@elek
elek committed Sep 13, 2020
1 parent 72e3215 commit d21890b
Show file tree
Hide file tree
Showing 2 changed files with 104 additions and 0 deletions.
12 changes: 12 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ Execute AWSS3APICli and checkrc
${output} = Execute and checkrc aws s3api --endpoint-url ${ENDPOINT_URL} ${command} ${expected_error_code}
[return] ${output}


Execute AWSS3Cli and checkrc
[Arguments] ${command} ${expected_error_code}
${output} = Execute and checkrc aws s3 --endpoint-url ${ENDPOINT_URL} ${command} ${expected_error_code}
[return] ${output}


Execute AWSS3Cli
[Arguments] ${command}
${output} = Execute aws s3 --endpoint-url ${ENDPOINT_URL} ${command}
Expand Down Expand Up @@ -91,6 +98,7 @@ Create bucket with name
Should contain ${result} ${ENDPOINT_URL}/${bucket}

Setup s3 tests
Run Keyword Generate random prefix
Run Keyword Install aws cli
Run Keyword if '${OZONE_S3_SET_CREDENTIALS}' == 'true' Setup v4 headers
${BUCKET} = Run Keyword if '${BUCKET}' == 'generated' Create bucket
Expand All @@ -109,3 +117,7 @@ Create link
[arguments] ${bucket}
Execute ozone sh bucket link o3://${OM_SERVICE_ID}/legacy/source-bucket o3://${OM_SERVICE_ID}/s3v/${bucket}
[return] ${bucket}

Generate random prefix
${random} = Generate Random String 5 [NUMBERS]
Set Suite Variable ${PREFIX} ${random}
92 changes: 92 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/s3/s3-vs-filepath.robot
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

*** Settings ***
Documentation S3a test with Ozone
Library OperatingSystem
Library String
Resource ../commonlib.robot
Resource commonawslib.robot
Test Timeout 5 minutes
Suite Setup Setup s3 tests

*** Variables ***
${ENDPOINT_URL} http://s3g:9878
${BUCKET} generated

*** Keywords ***
Create Dest Bucket

${postfix} = Generate Random String 5 [NUMBERS]
Set Suite Variable ${DESTBUCKET} destbucket-${postfix}
Execute AWSS3APICli create-bucket --bucket ${DESTBUCKET}


*** Test Cases ***
Key with double slash
Execute echo "Testcontent" > /tmp/conflict
Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/comp//file1 --body /tmp/conflict
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/comp
Should contain ${result} ${PREFIX}/comp//file1
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/comp
Should contain ${result} PRE comp/
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/comp/
Should contain ${result} PRE /

Key with dot
Execute echo "Testcontent" > /tmp/conflict
Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/kwd/./file1 --body /tmp/conflict
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/kwd
Should contain ${result} ${PREFIX}/kwd/./file1
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/kwd
Should contain ${result} PRE kwd/
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/kwd/
Should contain ${result} PRE .

Key with dotdot
Execute echo "Testcontent" > /tmp/conflict
Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/kwdd/../file1 --body /tmp/conflict
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/kwdd
Should contain ${result} ${PREFIX}/kwdd/../file1
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/kwdd
Should contain ${result} PRE kwdd/
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/kwdd/
Should contain ${result} PRE ..

Key with closing slash
Execute echo "Testcontent" > /tmp/testfile
Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/ks/prefix/file1/ --body /tmp/testfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/ks/ | jq -r '.Contents[] | .Key'
Should contain ${result} ${PREFIX}/ks/prefix/file1/
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/ks | jq -r '.Contents[] | .Key' | wc -l
Should Be Equal ${result} 1


Empty key without closing slash
Execute : > /tmp/testfile
Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/ekwos/prefix/file1 --body /tmp/testfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/ekwos | jq -r '.Contents[] | .Key'
Should contain ${result} ${PREFIX}/ekwos/prefix/file1
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/ekwos | jq -r '.Contents[] | .Key' | wc -l
Should Be Equal ${result} 1


Empty key with closing slash
Execute : > /tmp/testfile
Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/ekws/prefix/file1/ --body /tmp/testfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/ekws | jq -r '.Contents[] | .Key'
Should contain ${result} ${PREFIX}/ekws/prefix/file1/
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/ekws | jq -r '.Contents[] | .Key' | wc -l
Should Be Equal ${result} 1

0 comments on commit d21890b

Please sign in to comment.