Skip to content

HADOOP-19085 Add hadoop-compat-bench #6602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

HADOOP-19085 Add hadoop-compat-bench #6602

wants to merge 1 commit into from
+6,127 −0

Conversation

HanFreedom
Copy link

@HanFreedom HanFreedom commented Mar 1, 2024 •
edited
Loading

Description of PR

A new hadoop-compat-bench module introducing a quick HCFS compatibility assessment tool to Hadoop for FileSystem implementations, as described and discussed in HADOOP-19085.

How was this patch tested?

This is a new and standalone module module tested by its own unit tests.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 56s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 13 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 44m 15s trunk passed
+1 💚 compile 1m 27s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 compile 1m 12s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 50s trunk passed
+1 💚 mvnsite 1m 21s trunk passed
+1 💚 javadoc 2m 2s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 1m 46s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 4m 44s trunk passed
+1 💚 shadedclient 36m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 35s Maven dependency ordering for patch
+1 💚 mvninstall 1m 49s the patch passed
+1 💚 compile 1m 19s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javac 1m 19s the patch passed
+1 💚 compile 1m 7s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 1m 7s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 41s the patch passed
+1 💚 mvnsite 1m 36s the patch passed
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 javadoc 2m 15s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 2m 3s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 5m 39s the patch passed
+1 💚 shadedclient 36m 43s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 4m 37s hadoop-compat-bench in the patch passed.
+1 💚 unit 66m 38s hadoop-tools in the patch passed.
+1 💚 asflicense 0m 43s The patch does not generate ASF License warnings.
218m 6s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/1/artifact/out/Dockerfile
GITHUB PR #6602
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient xmllint shellcheck shelldocs spotbugs checkstyle
uname Linux 9ea78207b427 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 149cc23
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/1/testReport/
Max. process+thread count 1041 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-compat-bench hadoop-tools U: hadoop-tools
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@HanFreedom HanFreedom force-pushed the hadoop-compatibility-benchmark branch 2 times, most recently from a4ee2ad to e6cb258 Compare March 6, 2024 13:58
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 18m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 1s The patch appears to include 13 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 43m 43s trunk passed
+1 💚 compile 1m 29s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 1m 13s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 49s trunk passed
+1 💚 mvnsite 1m 16s trunk passed
+1 💚 javadoc 2m 4s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 47s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 4m 45s trunk passed
+1 💚 shadedclient 36m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 36s Maven dependency ordering for patch
+1 💚 mvninstall 1m 46s the patch passed
+1 💚 compile 1m 20s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 1m 20s the patch passed
+1 💚 compile 1m 7s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 1m 7s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 41s the patch passed
+1 💚 mvnsite 1m 34s the patch passed
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 javadoc 2m 15s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 2m 3s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 5m 33s the patch passed
+1 💚 shadedclient 36m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 4m 38s hadoop-compat-bench in the patch passed.
+1 💚 unit 67m 19s hadoop-tools in the patch passed.
+1 💚 asflicense 0m 44s The patch does not generate ASF License warnings.
235m 58s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/2/artifact/out/Dockerfile
GITHUB PR #6602
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs spotbugs checkstyle markdownlint
uname Linux 43aa7709eb3e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e6cb258
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/2/testReport/
Max. process+thread count 1042 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-compat-bench hadoop-tools U: hadoop-tools
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 19m 3s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 13 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 49m 24s trunk passed
+1 💚 compile 1m 34s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 1m 10s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 49s trunk passed
+1 💚 mvnsite 1m 17s trunk passed
+1 💚 javadoc 1m 58s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 40s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 4m 47s trunk passed
+1 💚 shadedclient 41m 34s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 36s Maven dependency ordering for patch
+1 💚 mvninstall 1m 44s the patch passed
+1 💚 compile 1m 24s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 1m 24s the patch passed
+1 💚 compile 1m 3s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 1m 3s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 0m 43s the patch passed
+1 💚 mvnsite 1m 31s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 javadoc 2m 11s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 54s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 5m 33s the patch passed
+1 💚 shadedclient 41m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 4m 17s hadoop-compat-bench in the patch passed.
+1 💚 unit 65m 13s hadoop-tools in the patch passed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
249m 28s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/3/artifact/out/Dockerfile
GITHUB PR #6602
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs spotbugs checkstyle markdownlint
uname Linux 5cd5f9c2412e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e6cb258
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/3/testReport/
Max. process+thread count 972 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-compat-bench hadoop-tools U: hadoop-tools
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@HanFreedom HanFreedom force-pushed the hadoop-compatibility-benchmark branch from e6cb258 to 65ed74c Compare March 10, 2024 07:36
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 17m 37s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 13 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 44m 0s trunk passed
+1 💚 compile 1m 26s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 1m 14s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 52s trunk passed
+1 💚 mvnsite 1m 17s trunk passed
+1 💚 javadoc 2m 3s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 1m 47s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 4m 42s trunk passed
+1 💚 shadedclient 36m 28s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 37s Maven dependency ordering for patch
+1 💚 mvninstall 1m 46s the patch passed
+1 💚 compile 1m 21s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 1m 21s the patch passed
+1 💚 compile 1m 7s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 1m 7s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 40s the patch passed
+1 💚 mvnsite 1m 33s the patch passed
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 javadoc 2m 14s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 2m 4s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 5m 36s the patch passed
+1 💚 shadedclient 36m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 4m 49s hadoop-compat-bench in the patch passed.
+1 💚 unit 67m 19s hadoop-tools in the patch passed.
+1 💚 asflicense 0m 44s The patch does not generate ASF License warnings.
235m 10s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/4/artifact/out/Dockerfile
GITHUB PR #6602
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs spotbugs checkstyle markdownlint
uname Linux cccc4ceab242 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 65ed74c
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/4/testReport/
Max. process+thread count 1021 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-compat-bench hadoop-tools U: hadoop-tools
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6602/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
-1 ❌ patch 1m 04s #6602 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
Subsystem Report/Notes
GITHUB PR #6602
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6602/1/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@github-actions GitHub Actions
Copy link
Contributor

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

@github-actions github-actions bot added the Stale label Sep 30, 2025
@github-actions github-actions bot closed this Oct 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
build Stale TOOLS trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@HanFreedom @hadoop-yetus @ColdL