Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists #992

Closed

Conversation

@steveloughran
Copy link
Contributor

steveloughran commented Jun 19, 2019

This patch

  • changes the default for the staging committer to append, as we get for the classic FOF committer
  • adds a check for the dest path being a file not a dir
  • adds tests for this
  • Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.

Testing: S3A Ireland w/ S3Guard. the Staging test failed, which makes me think that the settings weren't getting through

@steveloughran steveloughran force-pushed the steveloughran:s3/HADOOP-16357-commit-append branch from 55d0ee9 to af06a4a Jun 19, 2019
@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 19, 2019

Test results (rebased patch): MR tests fail, but not ITestDirectoryCommitProtocol, which is complaining that the dest dir exists in job setup

org.apache.hadoop.fs.PathExistsException: `s3a://hwdev-steve-ireland-new/test/DELAY_LISTING_ME/ITestDirectoryCommitProtocol-testRecoveryAndCleanup': Setting job as Task committer attempt_200707120058_0001_m_000000_0: Destination path exists and committer conflict resolution mode is "fail"

	at org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter.failDestinationExists(StagingCommitter.java:878)
	at org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter.setupJob(DirectoryStagingCommitter.java:85)
	at org.apache.hadoop.fs.s3a.commit.AbstractITCommitProtocol.testRecoveryAndCleanup(AbstractITCommitProtocol.java:587)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:745)
@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 19, 2019

ITestDirectoryCommitProtocol now working. unsetting per-bucket options made this go away, though I couldn't find out exactly where it was being set (core-site -> auth-keys.xml -> out of hadoop source tree XML)

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jun 19, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 23 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
0 mvndep 74 Maven dependency ordering for branch
+1 mvninstall 1337 trunk passed
+1 compile 1178 trunk passed
+1 checkstyle 158 trunk passed
+1 mvnsite 138 trunk passed
+1 shadedclient 1101 branch has no errors when building and testing our client artifacts.
+1 javadoc 95 trunk passed
0 spotbugs 67 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 195 trunk passed
_ Patch Compile Tests _
0 mvndep 22 Maven dependency ordering for patch
+1 mvninstall 79 the patch passed
+1 compile 1191 the patch passed
+1 javac 1191 the patch passed
-0 checkstyle 150 root: The patch generated 5 new + 14 unchanged - 0 fixed = 19 total (was 14)
+1 mvnsite 121 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 726 patch has no errors when building and testing our client artifacts.
+1 javadoc 94 the patch passed
+1 findbugs 204 the patch passed
_ Other Tests _
+1 unit 536 hadoop-common in the patch passed.
-1 unit 283 hadoop-aws in the patch failed.
+1 asflicense 47 The patch does not generate ASF License warnings.
7728
Reason Tests
Failed junit tests hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux 6e2b3815149e 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 71ecd2e
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/testReport/
Max. process+thread count 1322 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jun 20, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 598 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
0 mvndep 69 Maven dependency ordering for branch
+1 mvninstall 1050 trunk passed
+1 compile 1053 trunk passed
+1 checkstyle 142 trunk passed
+1 mvnsite 131 trunk passed
+1 shadedclient 1014 branch has no errors when building and testing our client artifacts.
+1 javadoc 106 trunk passed
0 spotbugs 68 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 184 trunk passed
_ Patch Compile Tests _
0 mvndep 24 Maven dependency ordering for patch
+1 mvninstall 80 the patch passed
+1 compile 999 the patch passed
+1 javac 999 the patch passed
-0 checkstyle 185 root: The patch generated 5 new + 14 unchanged - 0 fixed = 19 total (was 14)
+1 mvnsite 132 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 687 patch has no errors when building and testing our client artifacts.
+1 javadoc 106 the patch passed
+1 findbugs 207 the patch passed
_ Other Tests _
+1 unit 531 hadoop-common in the patch passed.
-1 unit 293 hadoop-aws in the patch failed.
+1 asflicense 53 The patch does not generate ASF License warnings.
7667
Reason Tests
Failed junit tests hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux b11b4b62f323 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 71ecd2e
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/testReport/
Max. process+thread count 1376 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jun 20, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 42 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 52 Maven dependency ordering for branch
+1 mvninstall 1236 trunk passed
+1 compile 1024 trunk passed
+1 checkstyle 186 trunk passed
+1 mvnsite 130 trunk passed
+1 shadedclient 1103 branch has no errors when building and testing our client artifacts.
+1 javadoc 96 trunk passed
0 spotbugs 64 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 185 trunk passed
_ Patch Compile Tests _
0 mvndep 21 Maven dependency ordering for patch
+1 mvninstall 76 the patch passed
+1 compile 986 the patch passed
+1 javac 986 the patch passed
-0 checkstyle 148 root: The patch generated 6 new + 14 unchanged - 0 fixed = 20 total (was 14)
+1 mvnsite 122 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 1 The patch has no ill-formed XML file.
+1 shadedclient 849 patch has no errors when building and testing our client artifacts.
+1 javadoc 94 the patch passed
+1 findbugs 204 the patch passed
_ Other Tests _
+1 unit 522 hadoop-common in the patch passed.
-1 unit 292 hadoop-aws in the patch failed.
+1 asflicense 46 The patch does not generate ASF License warnings.
7376
Reason Tests
Failed junit tests hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux d8193d94ac33 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 5bfdf62
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/testReport/
Max. process+thread count 1504 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jun 20, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 40 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 21 Maven dependency ordering for branch
+1 mvninstall 1131 trunk passed
+1 compile 1069 trunk passed
+1 checkstyle 146 trunk passed
+1 mvnsite 123 trunk passed
+1 shadedclient 1064 branch has no errors when building and testing our client artifacts.
+1 javadoc 93 trunk passed
0 spotbugs 67 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 196 trunk passed
_ Patch Compile Tests _
0 mvndep 25 Maven dependency ordering for patch
+1 mvninstall 79 the patch passed
+1 compile 1026 the patch passed
+1 javac 1026 the patch passed
-0 checkstyle 151 root: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14)
+1 mvnsite 121 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 1 The patch has no ill-formed XML file.
+1 shadedclient 743 patch has no errors when building and testing our client artifacts.
+1 javadoc 119 the patch passed
+1 findbugs 259 the patch passed
_ Other Tests _
+1 unit 631 hadoop-common in the patch passed.
-1 unit 321 hadoop-aws in the patch failed.
+1 asflicense 60 The patch does not generate ASF License warnings.
7436
Reason Tests
Failed junit tests hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux 70ad53e83e67 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 5bfdf62
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/testReport/
Max. process+thread count 1490 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/4/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 24, 2019

checkstyle is confused. This is an enum. There are no values. If I added a default I'd just get told off "all options are already covered"

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 24, 2019

reviewing my own patch (and looking @ failure reports of unrelated runs), I plan to change the probes for _SUCCESS of the previous stage to be different: fail with a message "previous stage did not generate output"; always work out and report which stage in the list has the first missing entry, therefore is the first failure to worry about

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 24, 2019

  • also got a failure in the first test case where the source dir still existed. Need to Make sure it is deleted in setup
@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 24, 2019

[ERROR] test_110_teragen(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter)  Time elapsed: 0.981 s  <<< ERROR!org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory s3a://hwdev-steve-ireland-new/terasort-ITestTerasortMagicCommitter/sortin already exists
	at org.apache.hadoop.examples.terasort.TeraOutputFormat.checkOutputSpecs(TeraOutputFormat.java:130)
	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:277)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1576)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1573)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1573)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594)
	at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:304)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.fs.s3a.commit.terasort.AbstractCommitTerasortIT.executeStage(AbstractCommitTerasortIT.java:139)
	at org.apache.hadoop.fs.s3a.commit.terasort.AbstractCommitTerasortIT.test_110_teragen(AbstractCommitTerasortIT.java:167)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
``
@bgaborg

This comment has been minimized.

Copy link
Contributor

bgaborg commented Jun 24, 2019

Thanks for working on this @steveloughran; this will stabilize the integration tests.
When running the tests I get the following error:

[ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.781 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
[ERROR] testDefault(org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit)  Time elapsed: 0.22 s  <<< FAILURE!
java.lang.AssertionError: Conflict resolution mode in PartitionedStagingCommitter{StagingCommitter{AbstractS3ACommitter{role=Task committer attempt_job_0001_r_000002_3, name=partitioned, outputPath=s3a://bucket-name/output/path, workPath=file:/var/folders/n_/cnl5_2s56511p7f5842_p8yc0000gp/T/buffer/736a1d32-6626-45bf-ab48-ad50650e9459/_temporary/0/_temporary/attempt_job_0001_r_000002_3}, conflictResolution=APPEND, wrappedCommitter=FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_job_0001}; taskId=attempt_job_0001_r_000002_3, status=''}; org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter@31d48fe3}; outputPath=file:/var/folders/n_/cnl5_2s56511p7f5842_p8yc0000gp/T/cluster/gaborbota/736a1d32-6626-45bf-ab48-ad50650e9459/staging-uploads, workPath=null, algorithmVersion=1, skipCleanup=false, ignoreCleanupFailures=false}}} expected:<FAIL> but was:<APPEND>

Maybe I do something wrong, forgot to set something?

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 24, 2019

@bgaborg yes, seeing that too.

@steveloughran steveloughran force-pushed the steveloughran:s3/HADOOP-16357-commit-append branch 2 times, most recently from 63c1c24 to e27aa70 Jun 24, 2019
@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 24, 2019

Gabor, fixed that; also rebased to trunk.

  • I have not done a full itest on this yet*
@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jun 24, 2019

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 91 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 9 new or modified test files.
_ trunk Compile Tests _
0 mvndep 27 Maven dependency ordering for branch
+1 mvninstall 1294 trunk passed
+1 compile 964 trunk passed
+1 checkstyle 143 trunk passed
+1 mvnsite 123 trunk passed
+1 shadedclient 1016 branch has no errors when building and testing our client artifacts.
+1 javadoc 94 trunk passed
0 spotbugs 65 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 180 trunk passed
_ Patch Compile Tests _
0 mvndep 20 Maven dependency ordering for patch
+1 mvninstall 77 the patch passed
+1 compile 924 the patch passed
+1 javac 924 the patch passed
-0 checkstyle 146 root: The patch generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15)
+1 mvnsite 118 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 1 The patch has no ill-formed XML file.
+1 shadedclient 706 patch has no errors when building and testing our client artifacts.
+1 javadoc 92 the patch passed
+1 findbugs 196 the patch passed
_ Other Tests _
+1 unit 535 hadoop-common in the patch passed.
+1 unit 289 hadoop-aws in the patch passed.
+1 asflicense 43 The patch does not generate ASF License warnings.
7088
Subsystem Report/Notes
Docker Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux 93bda9c4257e 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 129576f
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/artifact/out/diff-checkstyle-root.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/testReport/
Max. process+thread count 1347 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/7/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran changed the title HADOOP-16537 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists Jun 25, 2019
@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jun 25, 2019

Tested: S3A ireland

All good except the root dir tests, which Gabor and I are worrying about on the basis that it may be a sign of a regression

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jun 25, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 35 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 9 new or modified test files.
_ trunk Compile Tests _
0 mvndep 66 Maven dependency ordering for branch
+1 mvninstall 1085 trunk passed
+1 compile 1046 trunk passed
+1 checkstyle 135 trunk passed
+1 mvnsite 130 trunk passed
+1 shadedclient 992 branch has no errors when building and testing our client artifacts.
+1 javadoc 92 trunk passed
0 spotbugs 71 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 189 trunk passed
_ Patch Compile Tests _
0 mvndep 20 Maven dependency ordering for patch
+1 mvninstall 79 the patch passed
+1 compile 997 the patch passed
+1 javac 997 the patch passed
-0 checkstyle 142 root: The patch generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15)
+1 mvnsite 126 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 1 The patch has no ill-formed XML file.
+1 shadedclient 669 patch has no errors when building and testing our client artifacts.
+1 javadoc 104 the patch passed
+1 findbugs 200 the patch passed
_ Other Tests _
-1 unit 519 hadoop-common in the patch failed.
+1 unit 295 hadoop-aws in the patch passed.
+1 asflicense 51 The patch does not generate ASF License warnings.
6995
Reason Tests
Failed junit tests hadoop.util.TestReadWriteDiskValidator
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux 4a6eada0bd8c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 48d7f00
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/testReport/
Max. process+thread count 1427 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/8/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

This changes the default conflict resolution mode (in code, core-default and docs) to append, to be consistent with FileOutputCommitter. Also checks for all conflict modes that the destination doesn't exist; if not calls mkdirs. If it does exist, verify that it is a directory.

Committer test  doesn't set partition policy or conflict mode, so verifies the defaults work (the test was changed before the source tests, to verify this was true)

Most the change is actually to the mocking tests, because of the switch from FS.exists(path) to getFileStatus() + mkdirs. this is why I hate mocking so much

Change-Id: I6fe9a12eea702cf75b35b8c7fb0db3f185d3f40b
…citly go back to append; new defaults aren't being picked up

Change-Id: I7837cd100c77eeaafafbb422074b0944282fde82
… clearing bucket options.

Somehow locally the commit confict mode fs.s3a.committer.staging.conflict-mode was retaining as fail, even with the core-default.xml set. Extra tests to log the origin of the option on an assert failure implied somehow this was coming via test/resources/core-site.xml and any transitive XMLIncludes from there.

resetting the per-bucket options was enough to make this "go away"; the test is retained in case someone else's setting is similar.

Change-Id: I3a882919a78438b22fa90e9bceb516a4de640082
Note: if the core-site actually sets it for all buckets, rather than per-bucket, then the new test will fail. Why not clear it? Because we want to verify that the default is now append.
Change-Id: I8da28bce5c5715501fb539e5bafdc247a9c81b62
fix up a test failure

Change-Id: I3af7235173b4e35ac51ba1258c4693ad4064b2b0
+ log the terasort CSV file to $TMP rather than the remote FS, where it was being deleted in teardown

Change-Id: I5e2e68f7e3d356d8f3dbde784a5fb0796cc888e6
Change-Id: I6224eb5b11de937e6f21d834e5a28a101f1eef89
@steveloughran steveloughran force-pushed the steveloughran:s3/HADOOP-16357-commit-append branch from e4afee1 to 899c36b Jul 1, 2019
@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jul 1, 2019

rebase to trunk and push up again to kick off a retest

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jul 1, 2019

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 89 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 9 new or modified test files.
_ trunk Compile Tests _
0 mvndep 69 Maven dependency ordering for branch
+1 mvninstall 1213 trunk passed
+1 compile 1066 trunk passed
+1 checkstyle 152 trunk passed
+1 mvnsite 134 trunk passed
+1 shadedclient 1111 branch has no errors when building and testing our client artifacts.
+1 javadoc 105 trunk passed
0 spotbugs 73 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 208 trunk passed
_ Patch Compile Tests _
0 mvndep 23 Maven dependency ordering for patch
+1 mvninstall 83 the patch passed
+1 compile 993 the patch passed
+1 javac 993 the patch passed
-0 checkstyle 151 root: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)
+1 mvnsite 126 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 745 patch has no errors when building and testing our client artifacts.
+1 javadoc 97 the patch passed
+1 findbugs 206 the patch passed
_ Other Tests _
+1 unit 559 hadoop-common in the patch passed.
+1 unit 284 hadoop-aws in the patch passed.
+1 asflicense 46 The patch does not generate ASF License warnings.
7463
Subsystem Report/Notes
Docker Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux 559b90f4a4a0 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1e727cf
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/artifact/out/diff-checkstyle-root.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/testReport/
Max. process+thread count 1633 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/9/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Jul 1, 2019

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 31 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 9 new or modified test files.
_ trunk Compile Tests _
0 mvndep 62 Maven dependency ordering for branch
+1 mvninstall 1079 trunk passed
+1 compile 1068 trunk passed
+1 checkstyle 140 trunk passed
+1 mvnsite 111 trunk passed
+1 shadedclient 918 branch has no errors when building and testing our client artifacts.
+1 javadoc 86 trunk passed
0 spotbugs 61 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 169 trunk passed
_ Patch Compile Tests _
0 mvndep 21 Maven dependency ordering for patch
+1 mvninstall 74 the patch passed
+1 compile 996 the patch passed
+1 javac 996 the patch passed
-0 checkstyle 138 root: The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)
+1 mvnsite 115 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 1 The patch has no ill-formed XML file.
+1 shadedclient 636 patch has no errors when building and testing our client artifacts.
+1 javadoc 108 the patch passed
+1 findbugs 206 the patch passed
_ Other Tests _
+1 unit 538 hadoop-common in the patch passed.
+1 unit 297 hadoop-aws in the patch passed.
+1 asflicense 53 The patch does not generate ASF License warnings.
6888
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/artifact/out/Dockerfile
GITHUB PR #992
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname Linux 0e01d3bec442 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1e727cf
Default Java 1.8.0_212
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/artifact/out/diff-checkstyle-root.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/testReport/
Max. process+thread count 1361 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-992/10/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jul 9, 2019

checkstyle is mistaken; this patch is ready to go in.

@mackrorysd

This comment has been minimized.

Copy link
Contributor

mackrorysd commented Jul 9, 2019

I'm gonna give a +0.5 for now. I reviewed the code and it looks good to me, but I don't think I fully understand the practical consequences of changing the default resolution mode just yet.

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jul 10, 2019

we're changing the default mode to == that of FileOutputCommitter, namely "it's ok if the dest exists"

Spark will refuse to execute if the dest exists, so to you use any of the conflict options you actually need to permit overwrite -so you won't change behaviour without noticing

What this does do is make the committer more consistent with the existing one

@steveloughran steveloughran added the fs/s3 label Jul 10, 2019
@mackrorysd

This comment has been minimized.

Copy link
Contributor

mackrorysd commented Jul 10, 2019

Ok, sounds reasonable enough. +1 from me.

@steveloughran

This comment has been minimized.

Copy link
Contributor Author

steveloughran commented Jul 11, 2019

thanks, committed to trunk.

@steveloughran steveloughran deleted the steveloughran:s3/HADOOP-16357-commit-append branch Jul 30, 2019
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
…erFactory (apache#992)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.