HADOOP-18340. deleteOnExit does not work with S3AFileSystem #4608

huaxiangsun · 2022-07-21T23:27:21Z

Description of PR

processDeleteOnExit() is overiden in S3AFilesystem, it skips exist() check and delete objects without checking if FileSystem is closed.

How was this patch tested?

A new unitest case is added. And all unittest cases under hadoop-tools/hadoop-aws passed.
mvn -Dparallel-tests clean test

Did S3A Integration tests against us-west-2 region and there are a few failures/errors. Run the trunk code without the patch, there are same errors/failures. The errors/failures are not caused by the patch, probably due to misconfiguration ( I could not figure out)
mvn -Dparallel-tests clean verify

The result is
`

Tests	Errors	Failures	Skipped	Success Rate	Time
1252	6	1	270	77.875%	3,627.473

The errors are
org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob#testCommonCrawlLookup[1] + [ Detail ] | 0.324

| s3a://hbase-test-data/fork-0001/test: getFileStatus on s3a://hbase-test-data/fork-0001/test: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: XJ3DCCR6Q7SXTJDW; S3 Extended Request ID: fRmP3m1lThWxhj3s9VkSNEtuBz1JeBWYw65aRajrSg/H7IN+muB7d8PavSeqJ2urvLZtguTbnlc=; Proxy: null), S3 Extended Request ID: fRmP3m1lThWxhj3s9VkSNEtuBz1JeBWYw65aRajrSg/H7IN+muB7d8PavSeqJ2urvLZtguTbnlc=:InvalidAccessKeyId |
| |
| testJobSubmissionCollectsTokens[1] + [ Detail ] | 0.329
| s3a://hbase-test-data/fork-0001/test: getFileStatus on s3a://hbase-test-data/fork-0001/test: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: XJ35WHK7X6EMP9B6; S3 Extended Request ID: rtWEsDYcGqNiaoKy2D5EQQqN+O7MbYe1bYbiSmkF+FOz9/wb6+t+dQooqj7ppCSCZMBgC3PeEw4=; Proxy: null), S3 Extended Request ID: rtWEsDYcGqNiaoKy2D5EQQqN+O7MbYe1bYbiSmkF+FOz9/wb6+t+dQooqj7ppCSCZMBgC3PeEw4=:InvalidAccessKeyId
`

`
org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion#testBlankRegionTriggersSDKResolution + [ Detail ] | 2.817

| [Client region name] expected:<"[mars-north]-2"> but was:<"[us-west]-2">
and
org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials#estSTS
| : request session credentials: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Cannot call GetSessionToken with session credentials (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: d935696d-7d98-4a1c-825f-22bf3d28ed9d; Proxy: null):AccessDenied
`

For code changes:

[x ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[x ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

huaxiangsun · 2022-07-22T15:22:33Z

Will address the checkstyle issues.

huaxiangsun · 2022-07-27T17:21:53Z

Hi @steveloughran, can you take a look at the patch? Thanks.

steveloughran

code in s3afs and the tests all look good.

But i think the changes should be restricted to s3a fs, even if duplicates a bit of the superclass.

That whole section in the javadocs at the top of the Filesystem. class explains why; you've already seen some of those test values but it's and external implementations which we don't have control of. Adding anything is making a commitment to preserve a new public API forever.

I will be happier if you were just do it all in S3AFileSystem.

override deleteOnExit(Path f) and

skip the exists check because it doesn't do the right thing if you call the method while writing a file, because the file isn't visible until close. saves HEAD/List probes too
save the list to a set local to s3a fs. you could make its getter protected so that mock test can access it.

now, what about an integration test too?

add a test case which creates a new fs in the test case (keeping the normal getFileSystem() fs for assertions)

adds a file which doesn't exist
adds a file which doesn't exist, then create it
adds a file which does exist
add a directory path
close the fs, use ContractTestUtils methods to assert the files and dirs are gone.

seem good?

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ADeleteOnExit.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

huaxiangsun · 2022-07-28T23:26:51Z

code in s3afs and the tests all look good.

But i think the changes should be restricted to s3a fs, even if duplicates a bit of the superclass.

That whole section in the javadocs at the top of the Filesystem. class explains why; you've already seen some of those test values but it's and external implementations which we don't have control of. Adding anything is making a commitment to preserve a new public API forever.

Thanks! Points well taken.

I will be happier if you were just do it all in S3AFileSystem.

override deleteOnExit(Path f) and

skip the exists check because it doesn't do the right thing if you call the method while writing a file, because the file isn't visible until close. saves HEAD/List probes too

save the list to a set local to s3a fs. you could make its getter protected so that mock test can access it.

Got it, for s3a fs, the deleteOnExit set is going to be in s3a.

now, what about an integration test too?

Will try to add an IT.

add a test case which creates a new fs in the test case (keeping the normal getFileSystem() fs for assertions)

adds a file which doesn't exist

adds a file which doesn't exist, then create it

adds a file which does exist

add a directory path

close the fs, use ContractTestUtils methods to assert the files and dirs are gone.

seem good?

Sounds good to me, thanks for the feedbacks. May come back to you for test case, going to check it out first by myself.

steveloughran · 2022-07-29T10:25:20Z

yes; look at some of the other ITest cases in the s3a code.

if you aren't set up to run the hadoop-aws test suite, get set up to do that, as we require anyone submitting a pr to run the tests themselves first. getting them to work in your IDE is best for debugging.

huaxiangsun · 2022-08-01T17:20:11Z

Thanks for info for ITtest for s3a. I did run IT for s3a before submitting the patch, but did not try to run in IDE yet. Let me try.

huaxiangsun · 2022-08-02T22:10:40Z

Thanks @steveloughran for the helps. I pushed a new patch, which I hope addresses your comments.

Undo the changes in FileSystem.java, code is duplicated in S3AFileSystem.
Incorporated some of your changes in HADOOP-13611
Added a new s3a integration test case for deleteOnExit.

I have run all integration test under hadoop-aws. Here are the results.

Last Published: 2022-08-02 | Version: 3.4.0-SNAPSHOT

Failsafe Report

Summary

[Summary] [Package List] [Test Cases]

Tests	Errors	Failures	Skipped	Success Rate	Time
1253	7	0	270	77.893%	3,301.515

For errored ones, they are same when I run with trunk branch. I think there is some misconfiguration in my environment. They are listed below.

testSTS + [ Detail ]	7.277
	Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

	testSessionTokenExpiry + [ Detail ]
	Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.


	testSessionRequestExceptionTranslation + [ Detail ]
	Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.


	testSessionTokenPropagation + [ Detail ]
	Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.


	testInvalidSTSBinding + [ Detail ]
	Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

testCommonCrawlLookup[1] + [ Detail ] 0.234
s3a://hbase-test-data/fork-0002/test: getFileStatus on s3a://hbase-test-data/fork-0002/test: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: ZANXYYBWFXQN5C9D; S3 Extended Request ID: Se9ShKMS7T2H4oZb6kFEl+gUJFOIV9fgmnrzISp5NZi5QlbSLQQZdt6iRVbpKpRewK32iPominY=; Proxy: null), S3 Extended Request ID: Se9ShKMS7T2H4oZb6kFEl+gUJFOIV9fgmnrzISp5NZi5QlbSLQQZdt6iRVbpKpRewK32iPominY=:InvalidAccessKeyId

testJobSubmissionCollectsTokens[1] + [ Detail ] 0.218
s3a://hbase-test-data/fork-0002/test: getFileStatus on s3a://hbase-test-data/fork-0002/test: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: ZANK83F5X125CJB4; S3 Extended Request ID: EaB07gY+S8pH7WObAt7xPaqdX83XDtbX61qmDnoEsVyMLbC9IiDuyx5yKZpRS4q/Tkgjtcx6qzs=; Proxy: null), S3 Extended Request ID: EaB07gY+S8pH7WObAt7xPaqdX83XDtbX61qmDnoEsVyMLbC9IiDuyx5yKZpRS4q/Tkgjtcx6qzs=:InvalidAccessKeyId

huaxiangsun · 2022-08-02T22:18:44Z

I also run all unitest cases under hadoop-aws. They passed.
INFO] Results:
[INFO]
[WARNING] Tests run: 407, Failures: 0, Errors: 0, Skipped: 4
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:37 min
[INFO] Finished at: 2022-08-02T10:29:52-07:00
[INFO] ------------------------------------------------------------------------

huaxiangsun · 2022-08-05T17:08:08Z

@steveloughran Sorry to bother you while you are busy, just a gentle reminder for the patch I posted couple days ago. Appreciate that you can review the patch when you get chance, thanks.

steveloughran · 2022-08-08T19:05:43Z

yes, sorry, been neglecting this

steveloughran

all the production code is good; some minor changes to the tests and its ready for the next hadoop release

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADeleteOnExit.java

huaxiangsun · 2022-08-09T19:33:52Z

I also run the new IT test without change in S3AFileSystem, it failed as expected.

hadoop-yetus · 2022-08-09T21:11:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 21s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 39s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 35s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 19s		trunk passed
+1 💚	shadedclient	21m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 36s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 5s		the patch passed
+1 💚	shadedclient	19m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		98m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/7/artifact/out/Dockerfile
GITHUB PR	#4608
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 98c795ea550b 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fb2ae77`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/7/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-08-09T21:17:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 57s		trunk passed
+1 💚	compile	0m 58s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 49s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 41s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	0m 37s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 39s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 39s		trunk passed
+1 💚	shadedclient	24m 33s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 45s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 31s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 23s		the patch passed
+1 💚	shadedclient	24m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 47s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		107m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/6/artifact/out/Dockerfile
GITHUB PR	#4608
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 107e0a86a207 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fb2ae77`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/6/testReport/
Max. process+thread count	597 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-08-09T21:34:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 2s		trunk passed
+1 💚	compile	1m 4s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 49s		trunk passed
+1 💚	mvnsite	1m 3s		trunk passed
+1 💚	javadoc	0m 44s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 49s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 33s		trunk passed
+1 💚	shadedclient	27m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 42s		the patch passed
+1 💚	compile	0m 43s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 43s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 36s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 27s		the patch passed
+1 💚	mvnsite	0m 44s		the patch passed
+1 💚	javadoc	0m 22s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 22s		the patch passed
+1 💚	shadedclient	26m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 59s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		116m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/8/artifact/out/Dockerfile
GITHUB PR	#4608
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 02e1a93d5d2e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fb2ae77`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/8/testReport/
Max. process+thread count	607 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4608/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

+1. merging to trunk. will backport with a retest. patches which only touch hadoop-aws are always less risky than anything which changes hadoop-common, but i should test the changes before pushing to branch-3.3

Contributed by Huaxiang Sun

huaxiangsun · 2022-08-11T21:09:17Z

Thanks @steveloughran for the review, really nice comments!

) Contributed by Huaxiang Sun

steveloughran requested changes Jul 28, 2022

View reviewed changes

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ADeleteOnExit.java Show resolved Hide resolved

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java Outdated Show resolved Hide resolved

huaxiangsun force-pushed the trunk.HADOOP-18340 branch from db475ed to 74f042d Compare August 2, 2022 22:02

HADOOP-18340 deleteOnExit does not work with S3AFileSystem

38cd6ad

huaxiangsun force-pushed the trunk.HADOOP-18340 branch from 74f042d to 38cd6ad Compare August 2, 2022 22:34

steveloughran requested changes Aug 8, 2022

View reviewed changes

huaxiangsun closed this Aug 9, 2022

huaxiangsun reopened this Aug 9, 2022

address review comments in ITestS3ADeleteOnExit

fb2ae77

apache deleted a comment from hadoop-yetus Aug 11, 2022

steveloughran approved these changes Aug 11, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Aug 11, 2022

steveloughran changed the title ~~HADOOP-18340 deleteOnExit does not work with S3AFileSystem~~ HADOOP-18340. deleteOnExit does not work with S3AFileSystem Aug 11, 2022

steveloughran merged commit e9509ac into apache:trunk Aug 11, 2022

asfgit pushed a commit that referenced this pull request Aug 11, 2022

HADOOP-18340. deleteOnExit does not work with S3AFileSystem (#4608)

1b9135e

Contributed by Huaxiang Sun

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18340. deleteOnExit does not work with S3AFileSystem (apache#4608

e05e063

) Contributed by Huaxiang Sun

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18340. deleteOnExit does not work with S3AFileSystem #4608

HADOOP-18340. deleteOnExit does not work with S3AFileSystem #4608

huaxiangsun commented Jul 21, 2022 •

edited

Loading

huaxiangsun commented Jul 22, 2022

huaxiangsun commented Jul 27, 2022

steveloughran left a comment

huaxiangsun commented Jul 28, 2022

steveloughran commented Jul 29, 2022

huaxiangsun commented Aug 1, 2022

huaxiangsun commented Aug 2, 2022 •

edited

Loading

Failsafe Report

Summary

huaxiangsun commented Aug 2, 2022

huaxiangsun commented Aug 5, 2022

steveloughran commented Aug 8, 2022

steveloughran left a comment

huaxiangsun commented Aug 9, 2022

hadoop-yetus commented Aug 9, 2022

hadoop-yetus commented Aug 9, 2022

hadoop-yetus commented Aug 9, 2022

steveloughran left a comment

huaxiangsun commented Aug 11, 2022

HADOOP-18340. deleteOnExit does not work with S3AFileSystem #4608

HADOOP-18340. deleteOnExit does not work with S3AFileSystem #4608

Conversation

huaxiangsun commented Jul 21, 2022 • edited Loading

Description of PR

How was this patch tested?

For code changes:

huaxiangsun commented Jul 22, 2022

huaxiangsun commented Jul 27, 2022

steveloughran left a comment

Choose a reason for hiding this comment

huaxiangsun commented Jul 28, 2022

steveloughran commented Jul 29, 2022

huaxiangsun commented Aug 1, 2022

huaxiangsun commented Aug 2, 2022 • edited Loading

Failsafe Report

Summary

huaxiangsun commented Aug 2, 2022

huaxiangsun commented Aug 5, 2022

steveloughran commented Aug 8, 2022

steveloughran left a comment

Choose a reason for hiding this comment

huaxiangsun commented Aug 9, 2022

hadoop-yetus commented Aug 9, 2022

hadoop-yetus commented Aug 9, 2022

hadoop-yetus commented Aug 9, 2022

steveloughran left a comment

Choose a reason for hiding this comment

huaxiangsun commented Aug 11, 2022

huaxiangsun commented Jul 21, 2022 •

edited

Loading

huaxiangsun commented Aug 2, 2022 •

edited

Loading