Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-17735. Upgrade AWS SDK to 1.11.1026 #3057

Merged
merged 1 commit into from May 31, 2021
Merged

HADOOP-17735. Upgrade AWS SDK to 1.11.1026 #3057

merged 1 commit into from May 31, 2021

Conversation

steveloughran
Copy link
Contributor

Upgrade the AWS SDK to the latest version.

Tested: S3 london

-Dparallel-tests -DtestsThreadCount=8 -Dmarkers=keep -Ds3guard -Ddynamo
 -Dparallel-tests -DtestsThreadCount=8 -Dmarkers=delete -Dscale -Ds3guard -Ddynamo

some intermittent underful buffers on read(); the usual

Also

  • dependency review: no new spurious artifacts
  • built: release; playing with that right now

@steveloughran
HADOOP-17735. Upgrade AWS SDK to 1.11.1026
a98c409 
Change-Id: I2cea7e1971076b7af91ed569ab5929f457d176bf
@steveloughran steveloughran added build fs/s3 changes related to hadoop-aws; submitter must declare test endpoint labels May 26, 2021
@steveloughran
Copy link
Contributor Author

OK, full testing.md runbook done (with iostats and audit logging enabled). All good. For the curious, here are the iostats in the ITestS3AInputStreamPerformance output

2021-05-26 19:53:20,019 [JUnit] INFO  s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:dumpFileSystemIOStatistics(123)) - Aggregate FileSystem Statistics counters=((action_executor_acquired=1)
(action_http_get_request=15)
(action_http_head_request=24)
(audit_request_execution=66)
(audit_span_creation=50)
(directories_created=8)
(directories_deleted=7)
(fake_directories_deleted=1)
(files_created=1)
(files_deleted=1)
(object_bulk_delete_request=2)
(object_delete_objects=9)
(object_delete_request=7)
(object_list_request=9)
(object_metadata_request=24)
(object_put_bytes=32768)
(object_put_request=9)
(object_put_request_completed=9)
(op_create=1)
(op_delete=8)
(op_get_file_status=9)
(op_mkdirs=8)
(op_open=8)
(s3guard_metadatastore_initialization=8)
(s3guard_metadatastore_put_path_request=17)
(s3guard_metadatastore_record_reads=40)
(s3guard_metadatastore_record_writes=26)
(store_io_request=67)
(stream_aborted=2)
(stream_read_bytes=93473433)
(stream_read_bytes_backwards_on_seek=12713984)
(stream_read_bytes_discarded_in_abort=43889622)
(stream_read_bytes_discarded_in_close=252395)
(stream_read_close_operations=8)
(stream_read_closed=13)
(stream_read_fully_operations=8)
(stream_read_opened=15)
(stream_read_operations=14445)
(stream_read_operations_incomplete=3295)
(stream_read_seek_backward_operations=4)
(stream_read_seek_bytes_discarded=45092691)
(stream_read_seek_bytes_skipped=55054163)
(stream_read_seek_forward_operations=175)
(stream_read_seek_operations=179)
(stream_read_seek_policy_changed=9)
(stream_read_total_bytes=138818519)
(stream_write_block_uploads=1)
(stream_write_bytes=32768)
(stream_write_total_data=65536));

gauges=((stream_write_block_uploads_pending=1));

minimums=((action_executor_acquired.min=0)
(action_http_get_request.min=24)
(action_http_head_request.min=22)
(object_bulk_delete_request.min=49)
(object_delete_request.min=28)
(object_list_request.min=35)
(object_put_request.min=95)
(op_create.min=66)
(op_delete.min=48)
(op_get_file_status.min=157)
(op_mkdirs.min=152));

maximums=((action_executor_acquired.max=0)
(action_http_get_request.max=664)
(action_http_head_request.max=1688)
(object_bulk_delete_request.max=65)
(object_delete_request.max=37)
(object_list_request.max=75)
(object_put_request.max=360)
(op_create.max=66)
(op_delete.max=291)
(op_get_file_status.max=1690)
(op_mkdirs.max=446));

means=((action_executor_acquired.mean=(samples=1, sum=0, mean=0.0000))
(action_http_get_request.mean=(samples=15, sum=2583, mean=172.2000))
(action_http_head_request.mean=(samples=24, sum=8556, mean=356.5000))
(object_bulk_delete_request.mean=(samples=2, sum=114, mean=57.0000))
(object_delete_request.mean=(samples=7, sum=228, mean=32.5714))
(object_list_request.mean=(samples=9, sum=392, mean=43.5556))
(object_put_request.mean=(samples=9, sum=1390, mean=154.4444))
(op_create.mean=(samples=1, sum=66, mean=66.0000))
(op_delete.mean=(samples=8, sum=672, mean=84.0000))
(op_get_file_status.mean=(samples=9, sum=6857, mean=761.8889))
(op_mkdirs.mean=(samples=8, sum=1676, mean=209.5000)));

@steveloughran
Copy link
Contributor Author

And hadoop-cloud-storage POM trace.

not over-happy that hadoop-huaweicloud is adding log4j 2 to the classpath, but that's unrelated to this JIRA. And it won't be ending up in the hadoop common classpath,

[INFO] org.apache.hadoop:hadoop-cloud-storage:jar:3.4.0-SNAPSHOT
[INFO] +- org.apache.hadoop:hadoop-annotations:jar:3.4.0-SNAPSHOT:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:provided
[INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.0:provided
[INFO] |  +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.0:compile
[INFO] |  +- com.google.guava:guava:jar:27.0-jre:compile
[INFO] |  |  +- com.google.guava:failureaccess:jar:1.0:compile
[INFO] |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  +- org.checkerframework:checker-qual:jar:2.5.2:compile
[INFO] |  |  +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  +- commons-cli:commons-cli:jar:1.2:provided
[INFO] |  +- org.apache.commons:commons-math3:jar:3.1.1:provided
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.15:compile
[INFO] |  +- commons-io:commons-io:jar:2.8.0:provided
[INFO] |  +- commons-net:commons-net:jar:3.6:provided
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:provided
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
[INFO] |  +- jakarta.activation:jakarta.activation-api:jar:1.2.1:provided
[INFO] |  +- org.eclipse.jetty:jetty-server:jar:9.4.40.v20210413:provided
[INFO] |  |  +- org.eclipse.jetty:jetty-http:jar:9.4.40.v20210413:provided
[INFO] |  |  \- org.eclipse.jetty:jetty-io:jar:9.4.40.v20210413:provided
[INFO] |  +- org.eclipse.jetty:jetty-util:jar:9.4.40.v20210413:compile
[INFO] |  +- org.eclipse.jetty:jetty-servlet:jar:9.4.40.v20210413:provided
[INFO] |  |  \- org.eclipse.jetty:jetty-security:jar:9.4.40.v20210413:provided
[INFO] |  +- org.eclipse.jetty:jetty-webapp:jar:9.4.40.v20210413:provided
[INFO] |  |  \- org.eclipse.jetty:jetty-xml:jar:9.4.40.v20210413:provided
[INFO] |  +- javax.servlet.jsp:jsp-api:jar:2.1:provided
[INFO] |  +- com.sun.jersey:jersey-servlet:jar:1.19:provided
[INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] |  +- log4j:log4j:jar:1.2.17:provided
[INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.4:provided
[INFO] |  +- org.apache.commons:commons-configuration2:jar:2.1.1:provided
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.7:provided
[INFO] |  +- org.apache.commons:commons-text:jar:1.4:provided
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.30:provided
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:provided
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:provided
[INFO] |  +- com.google.re2j:re2j:jar:1.1:provided
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:provided
[INFO] |  +- com.google.code.gson:gson:jar:2.2.4:provided
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:3.4.0-SNAPSHOT:provided
[INFO] |  |  +- com.nimbusds:nimbus-jose-jwt:jar:9.8.1:compile
[INFO] |  |  |  \- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] |  |  +- net.minidev:json-smart:jar:2.4.2:provided
[INFO] |  |  |  \- net.minidev:accessors-smart:jar:2.4.2:provided
[INFO] |  |  |     \- org.ow2.asm:asm:jar:5.0.4:provided
[INFO] |  |  +- org.apache.curator:curator-framework:jar:4.2.0:provided
[INFO] |  |  \- org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
[INFO] |  |     +- org.apache.kerby:kerb-client:jar:1.0.1:provided
[INFO] |  |     |  +- org.apache.kerby:kerby-config:jar:1.0.1:provided
[INFO] |  |     |  +- org.apache.kerby:kerb-common:jar:1.0.1:provided
[INFO] |  |     |  |  \- org.apache.kerby:kerb-crypto:jar:1.0.1:provided
[INFO] |  |     |  +- org.apache.kerby:kerb-util:jar:1.0.1:provided
[INFO] |  |     |  \- org.apache.kerby:token-provider:jar:1.0.1:provided
[INFO] |  |     \- org.apache.kerby:kerb-admin:jar:1.0.1:provided
[INFO] |  |        +- org.apache.kerby:kerb-server:jar:1.0.1:provided
[INFO] |  |        |  \- org.apache.kerby:kerb-identity:jar:1.0.1:provided
[INFO] |  |        \- org.apache.kerby:kerby-xdr:jar:1.0.1:provided
[INFO] |  +- org.apache.curator:curator-client:jar:4.2.0:provided
[INFO] |  +- org.apache.curator:curator-recipes:jar:4.2.0:provided
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.19:provided
[INFO] |  +- org.bouncycastle:bcprov-jdk15on:jar:1.68:provided
[INFO] |  +- org.apache.kerby:kerb-core:jar:1.0.1:provided
[INFO] |  |  \- org.apache.kerby:kerby-pkix:jar:1.0.1:provided
[INFO] |  |     +- org.apache.kerby:kerby-asn1:jar:1.0.1:provided
[INFO] |  |     \- org.apache.kerby:kerby-util:jar:1.0.1:provided
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.10.5.1:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.10.5:compile
[INFO] |  +- org.codehaus.woodstox:stax2-api:jar:4.2.1:provided
[INFO] |  +- com.fasterxml.woodstox:woodstox-core:jar:5.3.0:provided
[INFO] |  +- dnsjava:dnsjava:jar:2.1.7:provided
[INFO] |  \- org.xerial.snappy:snappy-java:jar:1.1.8.2:provided
[INFO] +- org.apache.hadoop:hadoop-aliyun:jar:3.4.0-SNAPSHOT:compile
[INFO] |  \- com.aliyun.oss:aliyun-sdk-oss:jar:3.4.1:compile
[INFO] |     +- org.jdom:jdom:jar:1.1:compile
[INFO] |     +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |     +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
[INFO] |     +- com.aliyun:aliyun-java-sdk-ram:jar:3.0.0:compile
[INFO] |     +- com.aliyun:aliyun-java-sdk-sts:jar:3.0.0:compile
[INFO] |     \- com.aliyun:aliyun-java-sdk-ecs:jar:4.2.0:compile
[INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0-SNAPSHOT:compile
[INFO] |  +- com.amazonaws:aws-java-sdk-bundle:jar:1.11.1026:compile
[INFO] |  \- org.wildfly.openssl:wildfly-openssl:jar:1.0.7.Final:compile
[INFO] +- org.apache.hadoop:hadoop-azure:jar:3.4.0-SNAPSHOT:compile
[INFO] |  +- com.microsoft.azure:azure-storage:jar:7.0.1:compile
[INFO] |  |  \- com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile
[INFO] |  +- org.eclipse.jetty:jetty-util-ajax:jar:9.4.40.v20210413:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.0-SNAPSHOT:compile
[INFO] |  \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile
[INFO] +- org.apache.hadoop:hadoop-openstack:jar:3.4.0-SNAPSHOT:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.13:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.10.5:compile
[INFO] +- org.apache.hadoop:hadoop-cos:jar:3.4.0-SNAPSHOT:compile
[INFO] |  \- com.qcloud:cos_api-bundle:jar:5.6.19:compile
[INFO] \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.0-SNAPSHOT:compile
[INFO]    \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile
[INFO]       +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile
[INFO]       +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile
[INFO]       +- org.apache.logging.log4j:log4j-core:jar:2.12.0:compile
[INFO]       \- org.apache.logging.log4j:log4j-api:jar:2.12.0:compile
[INFO] ------------------------------------------------------------------------

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 17s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 37s trunk passed
+1 💚 compile 0m 21s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 compile 0m 19s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 mvnsite 0m 25s trunk passed
+1 💚 javadoc 0m 23s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 0m 21s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 shadedclient 60m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 15s the patch passed
+1 💚 compile 0m 14s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 javac 0m 14s the patch passed
+1 💚 compile 0m 13s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 javac 0m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 17s the patch passed
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 15s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 0m 14s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 shadedclient 14m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 18s hadoop-project in the patch passed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
80m 57s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3057/1/artifact/out/Dockerfile
GITHUB PR #3057
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml
uname Linux deba9d2983fa 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / a98c409
Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3057/1/testReport/
Max. process+thread count 677 (vs. ulimit of 5500)
modules C: hadoop-project U: hadoop-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3057/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

mukund-thakur
mukund-thakur approved these changes May 31, 2021
View reviewed changes
Copy link
Contributor

@mukund-thakur mukund-thakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@steveloughran steveloughran merged commit bd7b97b into apache:trunk May 31, 2021
asfgit pushed a commit that referenced this pull request May 31, 2021
@steveloughran
HADOOP-17735. Upgrade AWS SDK to 1.11.1026 (#3057)
bb04b60 
Change-Id: Ia9b32ab35454ada5f5e3d47b72cd9f2a2446a309
@steveloughran
Copy link
Contributor Author

Merged to trunk & 3.3.2

  1. This was my first attempt at using the ipad github client to do the merge, wouldn't again. Doesn't seem to let me edit the message. I had to run to the mac, boot it after an OS update and verify that the commit message was OK.
  2. Should we get this into branch-3.3.1 if it's not too late? Makes sense to get the latest version out ASAP

@steveloughran steveloughran deleted the s3/HADOOP-17735-sdk branch October 15, 2021 19:42
kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021
@steveloughran
HADOOP-17735. Upgrade AWS SDK to 1.11.1026 (apache#3057)
a2144e3
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
@steveloughran
CDPD-26398. HADOOP-17735. Upgrade AWS SDK to 1.11.1026 (apache#3057)
efbc172 
Change-Id: Ia9b32ab35454ada5f5e3d47b72cd9f2a2446a309
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@risdenk risdenk risdenk approved these changes

@mukund-thakur mukund-thakur mukund-thakur approved these changes

@bogthe bogthe bogthe approved these changes

@aajisaka aajisaka Awaiting requested review from aajisaka

@jojochuang jojochuang Awaiting requested review from jojochuang

Assignees
No one assigned
Labels
build fs/s3 changes related to hadoop-aws; submitter must declare test endpoint
Projects
None yet
Milestone
No milestone
5 participants
@steveloughran @hadoop-yetus @risdenk @mukund-thakur @bogthe