HADOOP-18995. Upgrades SDK version to 2.21.33 for S3 Express One Zone support.

[ahmarsuhail]

Description of PR

Upgrades SDK version to 2.21.33, which adds in S3 Express One Zone support.

With this upgrade, it is possible to connect to an One Zone bucket. Some tests from the S3A test suite will currently fail against a one zone bucket, as one zone buckets do not support some S3 standard features (eg: SSE-KMS), and certain operations behave slightly differently (eg: listMPU will return a directory that has incomplete MPUs). These will be addressed in HADOOP-18996.

How was this patch tested?

Tested in eu-west-1 with mvn -Dparallel-tests -DtestsThreadCount=16 clean verify and scale config, against S3 Standard. All good.

[steveloughran]

Did you try out the CLI commands as the usual qualification docs cover? it's the way some regressions have surfaced in the past, generally through new things being logged

[steveloughran]

testing locally. the shaded pom (still) complains about logging and doesn't let us turn on httpclient logging through log4j files. Otherwise all good.

cloudstore programmatically turns on shaded httpclient logs if you set -debug on the cli. How about we add a virtual log org.apache.hadoop.fs.s3a.http which, if set to debug will tell s3afs to switch logging to debug behind the scenes?

(note; test executed from uk against uswest2 over VPN; latency is due to the size of the planet)

 bin/hadoop fs -ls -R s3a://stevel--usw2-az1--x-s3/
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user/stevel
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user/stevel/target
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user/stevel/target/test
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data/Z5ozmhmq7a
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data/9r43oqPJPq
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/user/stevel/target/test/data/Q3pU1HXsFY
drwxrwxrwx   - stevel stevel          0 2023-11-29 13:07 s3a://stevel--usw2-az1--x-s3/test
2023-11-29 13:07:47,117 [shutdown-hook-0] INFO  statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((action_http_head_request=4)
(audit_request_execution=19)
(audit_span_creation=13)
(object_list_request=14)
(object_metadata_request=4)
(op_get_file_status=1)
(op_glob_status=1)
(op_list_status=10)
(store_io_request=19));

gauges=();

minimums=((action_http_head_request.min=152)
(object_list_request.min=147)
(op_get_file_status.min=0)
(op_glob_status.min=4)
(op_list_status.min=152));

maximums=((action_http_head_request.max=184)
(object_list_request.max=1038)
(op_get_file_status.max=0)
(op_glob_status.max=4)
(op_list_status.max=1048));

means=((action_http_head_request.mean=(samples=4, sum=679, mean=169.7500))
(object_list_request.mean=(samples=14, sum=3080, mean=220.0000))
(op_get_file_status.mean=(samples=1, sum=0, mean=0.0000))
(op_glob_status.mean=(samples=1, sum=4, mean=4.0000))
(op_list_status.mean=(samples=10, sum=3787, mean=378.7000)));

[steveloughran]

+1

merge at your leisure

[steveloughran]

(merged myself so I can start with the next bit)

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 1s		Maven dependency ordering for branch
-1 ❌	mvninstall	32m 23s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	16m 20s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	14m 38s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	18m 10s		trunk passed
+1 💚	javadoc	8m 31s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 39s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	47m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 56s		Maven dependency ordering for patch
+1 💚	mvninstall	28m 45s		the patch passed
+1 💚	compile	15m 41s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	15m 41s		the patch passed
+1 💚	compile	17m 10s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	17m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	13m 42s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 20s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 42s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	48m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	734m 31s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 28s		The patch does not generate ASF License warnings.
		1020m 49s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6306/1/artifact/out/Dockerfile
GITHUB PR	#6306
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux aef8b3f4ce7c 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 3660425
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6306/1/testReport/
Max. process+thread count	3907 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6306/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.