HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider #4737

virajjasani · 2022-08-12T03:55:37Z

No description provided.

virajjasani · 2022-08-12T03:56:40Z

Test results look good, tested against endpoint us-west-2.

hadoop-yetus · 2022-08-12T05:37:53Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 37s		trunk passed
+1 💚	compile	1m 1s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 52s		trunk passed
+1 💚	mvnsite	1m 2s		trunk passed
+1 💚	javadoc	0m 49s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 50s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 32s		trunk passed
+1 💚	shadedclient	21m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 41s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 41s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 27s		the patch passed
+1 💚	mvnsite	0m 41s		the patch passed
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 31s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 14s		the patch passed
+1 💚	shadedclient	20m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 44s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		100m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/1/artifact/out/Dockerfile
GITHUB PR	#4737
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 24882f69015d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `984d82b`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/1/testReport/
Max. process+thread count	681 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2022-08-15T23:11:50Z

@mukund-thakur could you please review this PR?

virajjasani · 2022-08-18T00:48:32Z

@steveloughran could you please review as per convenience? nothing urgent.

mukund-thakur

looks good.

mukund-thakur · 2022-08-18T17:28:07Z

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java

-    assertNotNull(stat);
-    assertEquals(testFile, stat.getPath());
+    try (FileSystem fs = FileSystem.newInstance(testFile.toUri(), conf)) {
+      assertNotNull(fs);


Now as you are fixing this already, why don't we add error messages in the assert statements for better error reporting. thx.

Done, thanks @mukund-thakur

mukund-thakur

Sorry for adding extra work.

mukund-thakur · 2022-08-18T18:37:13Z

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java

-    assertEquals(testFile, stat.getPath());
+    try (FileSystem fs = FileSystem.newInstance(testFile.toUri(), conf)) {
+      assertNotNull("S3AFileSystem instance must not be null", fs);
+      assertTrue(fs instanceof S3AFileSystem);


here as well please.

Ah, missed it. Done, thanks

mukund-thakur · 2022-08-18T18:37:37Z

...-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java

+      assertTrue(fs instanceof S3AFileSystem);
+      FileStatus stat = fs.getFileStatus(testFile);
+      assertNotNull("FileStatus with qualified path must not be null", stat);
+      assertEquals(testFile, stat.getPath());


mukund-thakur

LGTM +1,
let's wait for yetus to finish.

hadoop-yetus · 2022-08-18T19:54:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 22s		trunk passed
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 45s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 43s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 40s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 25s		trunk passed
+1 💚	shadedclient	23m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	0m 40s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 40s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 24s		the patch passed
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 28s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 12s		the patch passed
+1 💚	shadedclient	23m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 46s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		105m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/2/artifact/out/Dockerfile
GITHUB PR	#4737
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7e6ca432ae51 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `762d33d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/2/testReport/
Max. process+thread count	604 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-08-18T21:09:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 32s		trunk passed
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 46s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 43s		trunk passed
+1 💚	mvnsite	0m 52s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 42s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 30s		trunk passed
+1 💚	shadedclient	24m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 42s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 23s		the patch passed
+1 💚	mvnsite	0m 38s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 27s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 14s		the patch passed
+1 💚	shadedclient	23m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 49s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		105m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/3/artifact/out/Dockerfile
GITHUB PR	#4737
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e7079b49869c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `be4360a`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/3/testReport/
Max. process+thread count	607 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4737/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

…4737) Contributed By: Viraj Jasani

mukund-thakur · 2022-08-18T23:00:11Z

merged to branch-3.3 as well.

…pache#4737) Contributed By: Viraj Jasani

HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider

984d82b

mukund-thakur reviewed Aug 18, 2022

View reviewed changes

add message for better debugging

762d33d

mukund-thakur reviewed Aug 18, 2022

View reviewed changes

addressing review

be4360a

mukund-thakur approved these changes Aug 18, 2022

View reviewed changes

mukund-thakur merged commit 7f03025 into apache:trunk Aug 18, 2022

mukund-thakur pushed a commit that referenced this pull request Aug 18, 2022

HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider (#…

51ddd02

…4737) Contributed By: Viraj Jasani

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider (a…

9187ae9

…pache#4737) Contributed By: Viraj Jasani

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider #4737

HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider #4737

virajjasani commented Aug 12, 2022

virajjasani commented Aug 12, 2022

hadoop-yetus commented Aug 12, 2022

virajjasani commented Aug 15, 2022

virajjasani commented Aug 18, 2022 •

edited

Loading

mukund-thakur left a comment

mukund-thakur Aug 18, 2022

virajjasani Aug 18, 2022

mukund-thakur left a comment

mukund-thakur Aug 18, 2022

virajjasani Aug 18, 2022

mukund-thakur Aug 18, 2022

mukund-thakur left a comment

hadoop-yetus commented Aug 18, 2022

hadoop-yetus commented Aug 18, 2022

mukund-thakur commented Aug 18, 2022

HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider #4737

HADOOP-18403. Fix FileSystem leak in ITestS3AAWSCredentialsProvider #4737

Conversation

virajjasani commented Aug 12, 2022

virajjasani commented Aug 12, 2022

hadoop-yetus commented Aug 12, 2022

virajjasani commented Aug 15, 2022

virajjasani commented Aug 18, 2022 • edited Loading

mukund-thakur left a comment

Choose a reason for hiding this comment

mukund-thakur Aug 18, 2022

Choose a reason for hiding this comment

virajjasani Aug 18, 2022

Choose a reason for hiding this comment

mukund-thakur left a comment

Choose a reason for hiding this comment

mukund-thakur Aug 18, 2022

Choose a reason for hiding this comment

virajjasani Aug 18, 2022

Choose a reason for hiding this comment

mukund-thakur Aug 18, 2022

Choose a reason for hiding this comment

mukund-thakur left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Aug 18, 2022

hadoop-yetus commented Aug 18, 2022

mukund-thakur commented Aug 18, 2022

virajjasani commented Aug 18, 2022 •

edited

Loading