HADOOP-18999. Debug level logging within AWS V2 SDK httpclient #6322

steveloughran · 2023-12-04T15:48:28Z

Aims to set debug logging of http messages if the log org.apache.hadoop.fs.s3a.logging.sdk is set to debug.

# AWS v2 low level logging
log4j.logger.org.apache.hadoop.fs.s3a.logging.sdk=DEBUG

This doesn't work in production, even though it's based on the cloudstore StoreEntryPoint code which does. No idea why not

How was this patch tested?

new test to show logs can be changed
running some itests doesn't show new value
nor does CLI

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

This doesn't work, even though it's based on the cloudstore StoreEntryPoint code which does. No idea why not Change-Id: Ie8390a59243c454f79bf5f5027d1d1cdde9e8b9c

and trace level switches sdk classes to tracing Change-Id: I192a186c4597f1fad7ae4ca8741795049b795ea2

hadoop-yetus · 2023-12-04T17:12:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 59s		trunk passed
+1 💚	compile	0m 22s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 19s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 17s		trunk passed
+1 💚	mvnsite	0m 22s		trunk passed
+1 💚	javadoc	0m 16s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 19s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	0m 40s		trunk passed
+1 💚	shadedclient	20m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 18s		the patch passed
+1 💚	compile	0m 15s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 15s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 6 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 11s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 5 new + 2 unchanged - 0 fixed = 7 total (was 2)
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 9s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
-1 ❌	javadoc	0m 17s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~~20.04-b08 with JDK Private Build-1.8.0_392-8u392-ga-1~~20.04-b08 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	spotbugs	0m 42s		the patch passed
+1 💚	shadedclient	20m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 4s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 23s		The patch does not generate ASF License warnings.
		83m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6322/1/artifact/out/Dockerfile
GITHUB PR	#6322
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux c99896eeac9c 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2f2bc53`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6322/1/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6322/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-12-04T17:17:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 13s		trunk passed
+1 💚	compile	0m 22s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 19s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 19s		trunk passed
+1 💚	mvnsite	0m 25s		trunk passed
+1 💚	javadoc	0m 16s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 20s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	0m 40s		trunk passed
+1 💚	shadedclient	20m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 16s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 16s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 6 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 11s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 5 new + 2 unchanged - 0 fixed = 7 total (was 2)
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	javadoc	0m 9s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
-1 ❌	javadoc	0m 16s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt	hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_392-8u392-ga-1~~20.04-b08 with JDK Private Build-1.8.0_392-8u392-ga-1~~20.04-b08 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	spotbugs	0m 40s		the patch passed
+1 💚	shadedclient	20m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 8s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		84m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6322/2/artifact/out/Dockerfile
GITHUB PR	#6322
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux ca7583804523 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `815009e`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6322/2/testReport/
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6322/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2023-12-08T14:38:09Z

no longer needed as 2.21.41 logs properly!

steveloughran added 2 commits December 4, 2023 15:38

HADOOP-18999. Debug level logging within AWS V2 SDK httpclient

2f2bc53

This doesn't work, even though it's based on the cloudstore StoreEntryPoint code which does. No idea why not Change-Id: Ie8390a59243c454f79bf5f5027d1d1cdde9e8b9c

HADOOP-18999. Debug level logging within AWS V2 SDK httpclient

815009e

and trace level switches sdk classes to tracing Change-Id: I192a186c4597f1fad7ae4ca8741795049b795ea2

github-actions bot added trunk TOOLS AWS labels Dec 4, 2023

steveloughran closed this Dec 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18999. Debug level logging within AWS V2 SDK httpclient #6322

HADOOP-18999. Debug level logging within AWS V2 SDK httpclient #6322

steveloughran commented Dec 4, 2023 •

edited

hadoop-yetus commented Dec 4, 2023

hadoop-yetus commented Dec 4, 2023

steveloughran commented Dec 8, 2023

HADOOP-18999. Debug level logging within AWS V2 SDK httpclient #6322

HADOOP-18999. Debug level logging within AWS V2 SDK httpclient #6322

Conversation

steveloughran commented Dec 4, 2023 • edited

How was this patch tested?

For code changes:

hadoop-yetus commented Dec 4, 2023

hadoop-yetus commented Dec 4, 2023

steveloughran commented Dec 8, 2023

steveloughran commented Dec 4, 2023 •

edited