Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-16186. NPE in ITestS3AFileSystemContract teardown in DynamoDBMetadataStore.lambda$listChildren #628

Closed
wants to merge 3 commits into from

Conversation

@bgaborg
Copy link
Contributor

bgaborg commented Mar 20, 2019

No description provided.

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Mar 20, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 506 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 1054 trunk passed
+1 compile 30 trunk passed
+1 checkstyle 19 trunk passed
+1 mvnsite 35 trunk passed
+1 shadedclient 703 branch has no errors when building and testing our client artifacts.
+1 findbugs 40 trunk passed
+1 javadoc 21 trunk passed
_ Patch Compile Tests _
+1 mvninstall 29 the patch passed
+1 compile 27 the patch passed
+1 javac 27 the patch passed
+1 checkstyle 17 the patch passed
+1 mvnsite 31 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 718 patch has no errors when building and testing our client artifacts.
+1 findbugs 48 the patch passed
+1 javadoc 19 the patch passed
_ Other Tests _
+1 unit 276 hadoop-aws in the patch passed.
+1 asflicense 25 The patch does not generate ASF License warnings.
3672
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-628/1/artifact/out/Dockerfile
GITHUB PR #628
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f393073c2885 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 310ebf5
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-628/1/testReport/
Max. process+thread count 447 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-628/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran

This comment has been minimized.

Copy link
Contributor

steveloughran commented Mar 21, 2019

as usual: which S3 endpoint did you test with; which s3guard build options, etc...

: new DirListingMetadata(path, metas, isAuthoritative,
if (!metas.isEmpty() && dirPathMeta == null) {
// We handle this case as the directory is deleted.
LOG.warn("Directory metadata is null, but the list of the "

This comment has been minimized.

Copy link
@steveloughran

steveloughran Mar 21, 2019

Contributor

we need to review a message which makes sense for someone who doesn't know the db structures. And match it with a troubleshooting doc stack trace.

This comment has been minimized.

Copy link
@bgaborg

bgaborg Mar 22, 2019

Author Contributor

Something like:

Directory marker is deleted, but the list of the directory elements is not empty: {}. This case is handled as the directory is deleted.

It would be better?

@bgaborg bgaborg force-pushed the bgaborg:HADOOP-16186 branch from 0bfe474 to fc2026c Mar 26, 2019
@bgaborg

This comment has been minimized.

Copy link
Contributor Author

bgaborg commented Mar 26, 2019

refactored a little bit, and added a unit test

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Mar 26, 2019

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 27 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 1012 trunk passed
+1 compile 31 trunk passed
+1 checkstyle 22 trunk passed
+1 mvnsite 33 trunk passed
+1 shadedclient 691 branch has no errors when building and testing our client artifacts.
+1 findbugs 43 trunk passed
+1 javadoc 21 trunk passed
_ Patch Compile Tests _
+1 mvninstall 31 the patch passed
+1 compile 30 the patch passed
+1 javac 30 the patch passed
-0 checkstyle 16 hadoop-tools/hadoop-aws: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 mvnsite 34 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 752 patch has no errors when building and testing our client artifacts.
+1 findbugs 54 the patch passed
+1 javadoc 22 the patch passed
_ Other Tests _
+1 unit 275 hadoop-aws in the patch passed.
+1 asflicense 24 The patch does not generate ASF License warnings.
3180
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-628/2/artifact/out/Dockerfile
GITHUB PR #628
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 8340d03bc4e8 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 5257f50
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-628/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-628/2/testReport/
Max. process+thread count 446 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-628/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran self-assigned this Mar 26, 2019
@steveloughran

This comment has been minimized.

Copy link
Contributor

steveloughran commented Mar 26, 2019

Code LGTM, only a couple of comments & it'll be good to go.
+1 pending those changes. thanks

@bgaborg

This comment has been minimized.

Copy link
Contributor Author

bgaborg commented Mar 27, 2019

Tested against ireland.
I still have one known issue: https://issues.apache.org/jira/browse/HADOOP-16207.
Maybe we can pull this, and fix that issue later?

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 66.011 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitMRJob
[ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitMRJob) Time elapsed: 37.181 s <<< ERROR!
java.io.FileNotFoundException: Path s3a://cloudera-dev-gabor-ireland/fork-0006/test/DELAY_LISTING_ME/testMRJob is recorded as deleted by S3Guard
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2369)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2343)
at org.apache.hadoop.fs.contract.ContractTestUtils.assertIsDirectory(ContractTestUtils.java:559)
at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertIsDirectory(AbstractFSContractTestBase.java:327)
at org.apache.hadoop.fs.s3a.commit.AbstractITCommitMRJob.testMRJob(AbstractITCommitMRJob.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)

@hadoop-yetus

This comment has been minimized.

Copy link

hadoop-yetus commented Mar 27, 2019

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 75 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 1390 trunk passed
+1 compile 39 trunk passed
+1 checkstyle 27 trunk passed
+1 mvnsite 43 trunk passed
+1 shadedclient 885 branch has no errors when building and testing our client artifacts.
+1 findbugs 60 trunk passed
+1 javadoc 30 trunk passed
_ Patch Compile Tests _
+1 mvninstall 36 the patch passed
+1 compile 36 the patch passed
+1 javac 36 the patch passed
+1 checkstyle 22 the patch passed
+1 mvnsite 39 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 953 patch has no errors when building and testing our client artifacts.
+1 findbugs 63 the patch passed
+1 javadoc 27 the patch passed
_ Other Tests _
+1 unit 283 hadoop-aws in the patch passed.
+1 asflicense 33 The patch does not generate ASF License warnings.
4125
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-628/3/artifact/out/Dockerfile
GITHUB PR #628
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 1f31ddf4858c 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / a4cd75e
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-628/3/testReport/
Max. process+thread count 320 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-628/3/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran added the fs/s3 label Mar 28, 2019
@steveloughran

This comment has been minimized.

Copy link
Contributor

steveloughran commented Mar 28, 2019

committed to trunk; closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.