Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. #2088

Merged
merged 3 commits into from Jun 23, 2020
Merged

HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. #2088

merged 3 commits into from Jun 23, 2020

Conversation

umamaheswararao
Copy link
Contributor

@umamaheswararao
Copy link
Contributor Author

Updated a PR with the draft and test which explains the idea.
Fallback can be used for better purposed like using it as default target fs for whole cluster correctly.
We still need to fix things in mkdirs etc to make mkdir work on internal dirs if fallback is set. I will file other JIRAs later for them.
Please review this PR and let me know thoughts.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 26s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 1m 3s Maven dependency ordering for branch
+1 💚 mvninstall 21m 38s trunk passed
+1 💚 compile 20m 36s trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚 compile 17m 16s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 checkstyle 2m 48s trunk passed
+1 💚 mvnsite 2m 45s trunk passed
+1 💚 shadedclient 21m 16s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 37s hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
-1 ❌ javadoc 0m 40s hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚 javadoc 1m 41s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗 spotbugs 3m 10s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 16s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for patch
+1 💚 mvninstall 1m 59s the patch passed
+1 💚 compile 19m 58s the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚 javac 19m 58s the patch passed
+1 💚 compile 17m 23s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 javac 17m 23s the patch passed
-0 ⚠️ checkstyle 2m 50s root: The patch generated 5 new + 58 unchanged - 1 fixed = 63 total (was 59)
+1 💚 mvnsite 2m 44s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 6s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 35s hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
-1 ❌ javadoc 0m 40s hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚 javadoc 1m 40s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 findbugs 5m 34s the patch passed
_ Other Tests _
+1 💚 unit 9m 18s hadoop-common in the patch passed.
-1 ❌ unit 91m 51s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 53s The patch does not generate ASF License warnings.
266m 0s
Reason Tests
Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/artifact/out/Dockerfile
GITHUB PR #2088
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 7e662fd8235b 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / ce1008f
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/artifact/out/diff-checkstyle-root.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/testReport/
Max. process+thread count 3422 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/1/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn approved these changes Jun 23, 2020
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanx @umamaheswararao for the work here. the code changes LGTM

...p-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java Outdated Show resolved Hide resolved
...ct/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkFallback.java Outdated Show resolved Hide resolved
@umamaheswararao umamaheswararao merged commit 7c02d18 into apache:trunk Jun 23, 2020
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for branch
+1 💚 mvninstall 26m 3s trunk passed
+1 💚 compile 27m 7s trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚 compile 21m 40s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 checkstyle 3m 26s trunk passed
+1 💚 mvnsite 3m 2s trunk passed
+1 💚 shadedclient 26m 36s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 42s hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
-1 ❌ javadoc 0m 42s hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚 javadoc 1m 44s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗 spotbugs 4m 16s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 7m 20s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 2m 3s the patch passed
+1 💚 compile 19m 59s the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚 javac 19m 59s the patch passed
+1 💚 compile 17m 23s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 javac 17m 23s the patch passed
-0 ⚠️ checkstyle 2m 47s root: The patch generated 1 new + 104 unchanged - 1 fixed = 105 total (was 105)
+1 💚 mvnsite 2m 44s the patch passed
+1 💚 whitespace 0m 1s The patch has no whitespace issues.
+1 💚 shadedclient 15m 49s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 40s hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
-1 ❌ javadoc 0m 46s hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚 javadoc 1m 46s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 findbugs 5m 52s the patch passed
_ Other Tests _
+1 💚 unit 9m 25s hadoop-common in the patch passed.
-1 ❌ unit 91m 14s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 51s The patch does not generate ASF License warnings.
289m 27s
Reason Tests
Failed junit tests hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
hadoop.hdfs.TestStripedFileAppend
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/artifact/out/Dockerfile
GITHUB PR #2088
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 4c23838b0bc2 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 201d734
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/artifact/out/diff-checkstyle-root.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/testReport/
Max. process+thread count 3368 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2088/3/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ayushtkn ayushtkn ayushtkn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@umamaheswararao @hadoop-yetus @ayushtkn