Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-16117 Update AWS SDK to 1.11.563 #818

Closed

Conversation

@steveloughran
Copy link
Contributor

steveloughran commented May 14, 2019

Contributed by Steve Loughran.

Change-Id: Iff125bf3e2f409ae7fd0d05440f4f7ba38c24331

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented May 14, 2019

Testing

  • full s3a tests run in auth+nonauth, scale against s3 ireland
  • test output reviewed

TODO:

  • switch to v4 auth only, rerun all,
  • do those FS commands to run on the CLI
@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented May 31, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 33 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 68 Maven dependency ordering for branch
+1 mvninstall 1041 trunk passed
+1 compile 1028 trunk passed
+1 mvnsite 837 trunk passed
+1 shadedclient 3649 branch has no errors when building and testing our client artifacts.
+1 javadoc 334 trunk passed
_ Patch Compile Tests _
0 mvndep 34 Maven dependency ordering for patch
+1 mvninstall 1025 the patch passed
+1 compile 978 the patch passed
+1 javac 978 the patch passed
+1 mvnsite 817 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 633 patch has no errors when building and testing our client artifacts.
+1 javadoc 340 the patch passed
_ Other Tests _
-1 unit 8824 root in the patch failed.
+1 asflicense 45 The patch does not generate ASF License warnings.
16844
Reason Tests
Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/artifact/out/Dockerfile
GITHUB PR #818
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname Linux e0d54062f2f2 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 52128e3
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/artifact/out/patch-unit-root.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/testReport/
Max. process+thread count 4541 (vs. ulimit of 5500)
modules C: hadoop-project . U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented May 31, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 35 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 66 Maven dependency ordering for branch
+1 mvninstall 1041 trunk passed
+1 compile 1014 trunk passed
+1 mvnsite 850 trunk passed
+1 shadedclient 3619 branch has no errors when building and testing our client artifacts.
+1 javadoc 329 trunk passed
_ Patch Compile Tests _
0 mvndep 30 Maven dependency ordering for patch
+1 mvninstall 1026 the patch passed
+1 compile 986 the patch passed
+1 javac 986 the patch passed
+1 mvnsite 803 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 615 patch has no errors when building and testing our client artifacts.
+1 javadoc 323 the patch passed
_ Other Tests _
-1 unit 954 root in the patch failed.
+1 asflicense 39 The patch does not generate ASF License warnings.
8889
Reason Tests
Failed junit tests hadoop.util.TestDiskChecker
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/artifact/out/Dockerfile
GITHUB PR #818
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname Linux b39b6c1814a6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1ae062c
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/artifact/out/patch-unit-root.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/testReport/
Max. process+thread count 1347 (vs. ulimit of 5500)
modules C: hadoop-project . U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Contributed by Steve Loughran.

Change-Id: Iff125bf3e2f409ae7fd0d05440f4f7ba38c24331
+review and update the AWS SDK notice to be consistent with the artifacts we bundle and the current NOTICE file in the AWS SDK

Change-Id: I43057100de9c9e8486099e5326fca3fce7ffd55a
Change-Id: Ib6305366bfc11fe58996ffd9ee942e5867429818
@steveloughran steveloughran force-pushed the steveloughran:s3/HADOOP-16117-SDK-update branch from f7c76d6 to 01e36bb Jun 5, 2019
@steveloughran steveloughran changed the title HADOOP-116117 Update AWS SDK to 1.11.549. HADOOP-16117 Update AWS SDK to 1.11.563 Jun 5, 2019
@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 5, 2019

Latest patch: tested s3 ireland DDB+ scale. Followed SDK update protocol in docs and added some more commands for the runbook: expunge and hdfs fetchdt. Couldn't use hadoop dtutil on the CLI as I couldn't use a -D option to set the delegation tokens in that command.

I've been using this with the HADOOP-15663 PR to create on-demand DDB tables (#879) and I have not encountered problems. IMO, this update is ready to go with trunk for broader testing

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jun 6, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 32 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 65 Maven dependency ordering for branch
+1 mvninstall 1042 trunk passed
+1 compile 1014 trunk passed
+1 mvnsite 844 trunk passed
+1 shadedclient 3636 branch has no errors when building and testing our client artifacts.
+1 javadoc 342 trunk passed
_ Patch Compile Tests _
0 mvndep 23 Maven dependency ordering for patch
+1 mvninstall 1073 the patch passed
+1 compile 965 the patch passed
+1 javac 965 the patch passed
+1 mvnsite 828 the patch passed
-1 whitespace 0 The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 668 patch has no errors when building and testing our client artifacts.
+1 javadoc 379 the patch passed
_ Other Tests _
-1 unit 8185 root in the patch failed.
+1 asflicense 46 The patch does not generate ASF License warnings.
16342
Reason Tests
Failed junit tests hadoop.hdfs.TestMultipleNNPortQOP
hadoop.hdfs.web.TestWebHdfsTimeouts
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/artifact/out/Dockerfile
GITHUB PR #818
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname Linux 69a492540875 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3b1c257
Default Java 1.8.0_212
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/artifact/out/whitespace-eol.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/artifact/out/patch-unit-root.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/testReport/
Max. process+thread count 4427 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-tools/hadoop-aws . U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

ajfabbri left a comment

+1 commit to trunk and continue integration testing.

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 6, 2019

merged into trunk

@steveloughran steveloughran deleted the steveloughran:s3/HADOOP-16117-SDK-update branch Jun 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.