HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used #6165

virajjasani · 2023-10-10T05:01:04Z

No description provided.

virajjasani · 2023-10-10T05:03:02Z

Endpoint: us-west-2

mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch (configured with encryptions: SSE, DSSE and NONE)

hadoop-yetus · 2023-10-10T07:08:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 21s		trunk passed
+1 💚	compile	0m 45s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 38s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 36s		trunk passed
+1 💚	mvnsite	0m 46s		trunk passed
+1 💚	javadoc	0m 32s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 38s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 12s		trunk passed
+1 💚	shadedclient	33m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 34s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 33s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 5s		the patch passed
+1 💚	shadedclient	33m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 48s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		125m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/1/artifact/out/Dockerfile
GITHUB PR	#6165
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b413114cafde 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `766c422`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/1/testReport/
Max. process+thread count	560 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2023-10-18T00:37:54Z

@ahmarsuhail @mehakmeet @mukund-thakur could you please review this PR?

ahmarsuhail

think we should remove the encryption setting from conf instead of skipping.

ahmarsuhail · 2023-10-18T12:16:29Z

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardTool.java

@@ -71,6 +72,7 @@ public void testLandsatBucketRequireGuarded() throws Throwable {
  @Test
  public void testLandsatBucketRequireUnencrypted() throws Throwable {
    skipIfClientSideEncryption();
+    skipIfAnyEncryptionSet(getConfiguration());


instead of skipping, how about removeBaseAndBucketOverrides(getConfiguration(), Constants.S3_ENCRYPTION_ALGORITHM); and you can remove the skipIfClientSideEncryption() too?

Gets better coverage even if encryption settings are used.

Nice, it would be better coverage regardless of encryption configured. Tested the changes with and without encryption.
Thanks @ahmarsuhail

hadoop-yetus · 2023-10-19T02:12:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 7s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 8s		trunk passed
+1 💚	shadedclient	32m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 33s		the patch passed
+1 💚	compile	0m 24s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 6s		the patch passed
+1 💚	shadedclient	32m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	2m 44s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		122m 45s

Reason	Tests
Failed junit tests	hadoop.fs.s3a.impl.TestErrorTranslation

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/2/artifact/out/Dockerfile
GITHUB PR	#6165
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 4e8c05fd5174 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1d3be91`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/2/testReport/
Max. process+thread count	558 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ahmarsuhail

Remove code in S3ATestUtils and Yetus is complaining about a unit test failure.

ahmarsuhail · 2023-10-19T07:23:10Z

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java

+   *
+   * @param configuration configuration to probe.
+   */
+  public static void skipIfAnyEncryptionSet(Configuration configuration) throws IOException {


think we can remove this now?

virajjasani · 2023-10-19T07:27:47Z

TestErrorTranslation seems recent regression, the base branch of this PR does not even have the test.

hadoop-yetus · 2023-10-19T09:18:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 12s		trunk passed
+1 💚	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 33s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 24s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 12s		trunk passed
-1 ❌	shadedclient	49m 14s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 22s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	compile	0m 22s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.
-1 ❌	javac	0m 22s	/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.
-1 ❌	compile	0m 23s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.
-1 ❌	javac	0m 23s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 55 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 624 line(s) with tabs.
-0 ⚠️	checkstyle	0m 20s	/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 ❌	mvnsite	0m 23s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 23s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.
-1 ❌	spotbugs	0m 22s	/patch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	shadedclient	4m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 23s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+0 🆗	asflicense	0m 22s		ASF License check generated no output?
		109m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/3/artifact/out/Dockerfile
GITHUB PR	#6165
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1266bcef325d 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b0e8436`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/3/testReport/
Max. process+thread count	160 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ahmarsuhail

+1, LGTM. Merging.

…pache#6165) HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used. Contributed by Viraj Jasani.

HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used

766c422

github-actions bot added trunk TOOLS AWS labels Oct 10, 2023

ahmarsuhail requested changes Oct 18, 2023

View reviewed changes

addendum

1d3be91

ahmarsuhail requested changes Oct 19, 2023

View reviewed changes

addendum

b0e8436

ahmarsuhail approved these changes Oct 20, 2023

View reviewed changes

ahmarsuhail merged commit acaf8ef into apache:trunk Oct 20, 2023
2 of 13 checks passed

jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024

HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used (a…

611c01d

…pache#6165) HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used. Contributed by Viraj Jasani.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used #6165

HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used #6165

virajjasani commented Oct 10, 2023

virajjasani commented Oct 10, 2023

hadoop-yetus commented Oct 10, 2023

virajjasani commented Oct 18, 2023

ahmarsuhail left a comment

ahmarsuhail Oct 18, 2023

virajjasani Oct 19, 2023

hadoop-yetus commented Oct 19, 2023

ahmarsuhail left a comment

ahmarsuhail Oct 19, 2023

virajjasani Oct 19, 2023

virajjasani commented Oct 19, 2023

hadoop-yetus commented Oct 19, 2023

ahmarsuhail left a comment

HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used #6165

HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used #6165

Conversation

virajjasani commented Oct 10, 2023

virajjasani commented Oct 10, 2023

hadoop-yetus commented Oct 10, 2023

virajjasani commented Oct 18, 2023

ahmarsuhail left a comment

Choose a reason for hiding this comment

ahmarsuhail Oct 18, 2023

Choose a reason for hiding this comment

virajjasani Oct 19, 2023

Choose a reason for hiding this comment

hadoop-yetus commented Oct 19, 2023

ahmarsuhail left a comment

Choose a reason for hiding this comment

ahmarsuhail Oct 19, 2023

Choose a reason for hiding this comment

virajjasani Oct 19, 2023

Choose a reason for hiding this comment

virajjasani commented Oct 19, 2023

hadoop-yetus commented Oct 19, 2023

ahmarsuhail left a comment

Choose a reason for hiding this comment