HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB #5129

haiyang1987 · 2022-11-13T05:11:48Z

Description of PR

HDFS-16840
Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB

the usage description support print out storage and erasure coding policy id, -maxSize, -step, -format and -addr
maybe can add relevant information in HDFSCommands.md to describe how to use it.

…Commands.md

haiyang1987 · 2022-11-13T05:15:49Z

Hi @ZanderXu @tomscut can you help me review this small change, Thanks！

hadoop-yetus · 2022-11-13T12:49:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 20s		trunk passed
+1 💚	compile	1m 39s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 30s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 17s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 16s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 46s		trunk passed
+1 💚	shadedclient	26m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 22s		the patch passed
+1 💚	compile	1m 26s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 26s		the patch passed
+1 💚	compile	1m 20s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 31s		the patch passed
+1 💚	shadedclient	26m 8s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	337m 41s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		456m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.mover.TestMover

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/1/artifact/out/Dockerfile
GITHUB PR	#5129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 7e9555d0c07b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `738c6ed`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/1/testReport/
Max. process+thread count	2041 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

haiyang1987 · 2022-11-15T11:34:31Z

The failed unit test seems unrelated to the change.

tasanuma

LGTM.

ZanderXu

Thanks @haiyang1987 for your report. Leave some minor comments.

...hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java

haiyang1987 · 2022-11-19T10:12:23Z

update PR and adding more information the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB

@ZanderXu @tasanuma please help me to reivew it again, Thanks.

hadoop-yetus · 2022-11-19T18:29:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 33s		trunk passed
+1 💚	compile	1m 38s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 27s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 17s		trunk passed
+1 💚	mvnsite	1m 40s		trunk passed
+1 💚	javadoc	1m 16s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 39s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 42s		trunk passed
+1 💚	shadedclient	26m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	1m 30s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 30s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	mvnsite	1m 25s		the patch passed
+1 💚	javadoc	0m 59s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 31s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 32s		the patch passed
+1 💚	shadedclient	26m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	387m 43s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		506m 11s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/2/artifact/out/Dockerfile
GITHUB PR	#5129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 6930f64dc6e4 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3cfa327`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/2/testReport/
Max. process+thread count	2074 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-11-21T01:08:23Z

...hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java

@@ -81,6 +81,8 @@ public class OfflineImageViewerPB {
      + "    changed via the -delimiter argument.\n"
      + "    -sp print storage policy, used by delimiter only.\n"
      + "    -ec print erasure coding policy, used by delimiter only.\n"
+      + "    -m,--multiThread defines multiThread to process sub-sections, \n"


Are there some duplicated options here?

ok, i will remove --multiThread

...hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java

haiyang1987 · 2022-11-22T02:11:54Z

Update PR
@ZanderXu @tomscut @tasanuma please help me to reivew it again, Thanks.

hadoop-yetus · 2022-11-22T10:31:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 29s		trunk passed
+1 💚	compile	1m 40s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 29s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 37s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 44s		trunk passed
+1 💚	shadedclient	26m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	1m 29s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 29s		the patch passed
+1 💚	compile	1m 20s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	mvnsite	1m 26s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 29s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 32s		the patch passed
+1 💚	shadedclient	26m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	375m 54s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 56s		The patch does not generate ASF License warnings.
		499m 46s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/3/artifact/out/Dockerfile
GITHUB PR	#5129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 658e6c93199c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ef9b78e`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/3/testReport/
Max. process+thread count	1976 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5129/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

tomscut

LGTM.

tomscut · 2022-11-24T01:01:26Z

Thanks @haiyang1987 for your contribution! Thanks @tasanuma and @ZanderXu for your review!

…d and OfflineImageViewerPB (apache#5129) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org> Reviewed-by: ZanderXu <zanderxu@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>

HDFS-16840. Add sp and ec policy opiton description about oiv in HDFS…

738c6ed

…Commands.md

tasanuma approved these changes Nov 17, 2022

View reviewed changes

ZanderXu reviewed Nov 18, 2022

View reviewed changes

...hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java Show resolved Hide resolved

HDFS-16840. Modify patch based on comments

3cfa327

haiyang1987 changed the title ~~HDFS-16840. Add sp and ec policy opiton description about oiv in HDFSCommands.md~~ HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB Nov 19, 2022

tomscut reviewed Nov 21, 2022

View reviewed changes

...hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java Outdated Show resolved Hide resolved

HDFS-16840. Modify patch based on comments

ef9b78e

ZanderXu approved these changes Nov 23, 2022

View reviewed changes

tomscut approved these changes Nov 24, 2022

View reviewed changes

tomscut merged commit dfa9eda into apache:trunk Nov 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB #5129

HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB #5129

haiyang1987 commented Nov 13, 2022 •

edited

Loading

haiyang1987 commented Nov 13, 2022

hadoop-yetus commented Nov 13, 2022

haiyang1987 commented Nov 15, 2022

tasanuma left a comment

ZanderXu left a comment

haiyang1987 commented Nov 19, 2022

hadoop-yetus commented Nov 19, 2022

tomscut Nov 21, 2022 •

edited

Loading

haiyang1987 Nov 22, 2022

haiyang1987 commented Nov 22, 2022

hadoop-yetus commented Nov 22, 2022

tomscut left a comment

tomscut commented Nov 24, 2022

HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB #5129

HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB #5129

Conversation

haiyang1987 commented Nov 13, 2022 • edited Loading

Description of PR

haiyang1987 commented Nov 13, 2022

hadoop-yetus commented Nov 13, 2022

haiyang1987 commented Nov 15, 2022

tasanuma left a comment

Choose a reason for hiding this comment

ZanderXu left a comment

Choose a reason for hiding this comment

haiyang1987 commented Nov 19, 2022

hadoop-yetus commented Nov 19, 2022

tomscut Nov 21, 2022 • edited Loading

Choose a reason for hiding this comment

haiyang1987 Nov 22, 2022

Choose a reason for hiding this comment

haiyang1987 commented Nov 22, 2022

hadoop-yetus commented Nov 22, 2022

tomscut left a comment

Choose a reason for hiding this comment

tomscut commented Nov 24, 2022

haiyang1987 commented Nov 13, 2022 •

edited

Loading

tomscut Nov 21, 2022 •

edited

Loading