HBASE-23106 [HBCK2/hbase-operator-tools] Add a WAL verifier

[saintstack]

Adds simple WALReader.

Here is example of what it looks like when it runs:

kalashnikov:hbase.apache.git stack$ ./bin/hbase org.apache.hadoop.hbase.wal.WALReader /tmp/0000000000000932379
2019-10-03 12:28:46,375 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
/tmp/0000000000000932379
3448 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932373 3271
11309 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932374 7767
18565 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932375 7162
34217 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932376 15558
44758 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932377 10447
57357 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932378 12505
60722 GENIE2_modality_syncdata/3c2fc74c7d67cd20883799b3f7789b3b/932379 3271
kalashnikov:hbase.apache.git stack$  ./bin/hbase org.apache.hadoop.hbase.wal.WALReader /tmp/0000000000000932380
2019-10-03 12:28:52,110 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
/tmp/0000000000000932380
2019-10-03 12:28:52,568 WARN  [main] hbase.KeyValueUtil: 8\x8E\xF8o\xF4\x03"\xD1\x89\xB8\x86\x5C\xECo\x16\x96\x5C,\xC4EH\xACO\x1E\xA4VF\xA7\xFB\xF9\x7F\xAF](\xFC\xE2\x07QQ\xE6\xA8-\xD1l\x0C\xE7\x13YX_[\x89+n&JS \xE5\x81\xF4\xFAH/\xBF\xF4\xBD\x95\x1F\x8F*\x96\x09\xC8Q\xEAoJt2\xA3E'\xA2\x01\xAB\xF2y%B\x18\xBAI)\x02\x0B\x0F\xFBv^\xF4\x85$^>\x14\x84\xA7\xBEb\xFE{N\xEB7W~~\xFB\x5C\x0E=\xD6FG\xB1ms"\x04\x81\xC69\x97i\x16m\xD0\x1F\xBC0iR.\xBB6\xEE\xEF\xDE\x05OF>\x029\x01\xF6\xFBJ\x8D\xA7\xDA\xE8x\xB2\x5C\xED20\x85Z\x09y\x82x\x98\xBAQ~,\x80@\x87r!\xAFa\x8F$\xDB\xC7\x06\x84\xE6\xEAe\xD0\xE2xF\x8E\x9A\x17mtbpb\x0F\xF2\xA4\x13\xBAn\x06j\x86\xFA\x92p\x1F\xBB\x97\xD7WZ\x09\xED(\x15\x8B\xAA\xDE\x09\xB8\xFE\x91\x8C\x0F\x17\x19s\x09\xF6\x95\x8DT\xAC\x0A\x12\xDALq@\xAALB\x8C\xB3\x0A\xCE\xA4k\xA5\x9F\xA25\xDBh.\xA5\xED\xD1(\x17s\x01\xCA\x94Gm\xDB%\x19\x9CI\x9FZw\xB2\xB8\xE9:\x8B\x9B\xD2\xF6\xE0N\x07j\xEDE)\x9C\xBA5"\xA5n\x9D\x088qh/\x85\xE6\xC4\x85\x10\xD9\x0A)=1+\x98\x05\x09\xBE\xCA\x99C\xDD!\x9F\x08\x9D\x94\x0F\xD2\xFA\xD3\xFB\xA6\x91, offset=0, length=1080
2019-10-03 12:28:52,569 WARN  [main] wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good position in file, from 1261 to 83
java.io.EOFException: EOF  while reading 12 WAL KVs; started reading at 177 and read up to 1261
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:397)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:98)
        at org.apache.hadoop.hbase.wal.WALReader.read(WALReader.java:41)
        at org.apache.hadoop.hbase.wal.WALReader.main(WALReader.java:59)
Caused by: java.lang.IllegalArgumentException: Invalid value length in KeyValue, valueLength=1986, KeyValueBytesHex=\x00\x00\x00T\x00\x00\x07\xC2\x00$efbb2495-8230-4262-a09d-e3261e54eb51\x04blobpm|com.apple.contact.people|pmap\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04\x1F\x8B\x08\x00\x00\x00\x00\x00\x00\x00\xEDW_l\x1CG\x19\x1F\x9F\xFF\xC4\xB1\x1D\xC7I\x13D+H\xF3`J\x03\xD9\xFB\xBB\xB7w\x974\x98\xB3\x0F\x07'\xD7\xC4\xF8\xE2@bJ5\xBB;\xEB\x1Bgwg33k\xDF%P\xB5O\xF0\x82T\x09\x1E\xA8hy\x00!\xD1\xD2\xAAOH \x90\xFA\xD8\x0A\x04\x0FH}\xA2\xA2\x0F \xF1\x80\xE0\x85J\x88"\x01\xDF\xEC\x9F\xBB\xB5\xE3\x18\xF3\xCEJ\xB7\xDA\xF9f\xBE\xDF\xF7\xFF\xFB\xE6^\xFF\x0B\x1A\x17\x1C\x9D\xDF\xC2\xDB8\x1FJ\xEA\xE6\x97\x98\xEB\x12KR\xE6\x8B\xF9u\xDFc6u(6]2\xA4?\xBA\x88\x9E\xFF\xD5W\xFE~&\x87F\xDAh\xC4\x92\xE8#m\xC5^P\xEC\x85\xE1\xB1\x8B\xBD\x00\x90KCd\x8B\xF9V\xC89\xF1%\x08I??\x8FE\xF7i\x1C\xCC\xDF\xC4nH\xC4MJv\x1E7\xDA/~\xE9[\xE7.\xE5\x10\xEAqT=$\xFFP\xAE\xC2\xB0_\xFA\xFD\xC9W\xBE\xAE0@\xC5Q\x0F\x07\x12=\x99Qr\x88Tx\x00)V\xFB\x89\xC3\x89M\xE5\x8C\xA2\xD1\x154-\xC8\xA6\x07{Ocqg\x05\xCD$\xABN\x97:r\x03M&K!Qi\xE3\x90\xAA\xCCwb\x1EP\x09!t\x1C~\x1F\x0By\xCA~\x08\x9F\xA4\xECk;\x0B\xCD\xF2w\x1BRyT!\xCD\x81\x81\xF9\xFF\x0Dc\x18\x94\x91e4\xE52l/cK2\x0E\x11zr_$\x97YwD~\x8D\xC07\xC7\xBEl\xC3\xD2Y\xFF\xD1\xF9\xF3\xEFl\xFD9\x0A\xCA\x98\xE8\xFB\x90:\x85\xFD]\x11q\x17vq\xCFw\x80!\x0E\x8E~X\x89\xF3\xD7\x98\xEF`\xCA\x15+\xF9F\xF9O\x9D\xFBoM\xC7i\xA5\x1D\x1AB\xF1\xFE\xF4\xCC\xF7\xBF\xFDZ\xEB\xF6W\x0FJ\xC9\x98\xB7i\x0A`\xB5\xE4\x17B\x12\x12[\xF1v9\xF3\xE9=\xC2\xC1\xFA\xA5pa\xED\x0F`\xFD\x0A\xD4\x9C\xC4\x92\x00\x96q\x08\xAC\xEB;\xBE*\xC0,X\xE5\xFD7\x7F\xF6\x86\xC7^M#\x8A\x16>\x8B\x90\xB8\x8B\x9E\x83\xB4S\xEF\xB9\xFF\xD3\x0E\xA6I\xD4\xB2\x98\x97\xC7A\xE0\x12\xE5y\x09~\xCE\x07\x84\xC1\xF2B\xB9Z\xB4\xEAV\xA3\xA1\xE9Ub\xC2\xAB^\xD3\xCC\x92ch\x95F\xADQ\xAD\x98\xF5Z\xB1b.\xB0K%\xCB\xB1 \x19?\xA5p\x04\xE5\x14`8\xC44\x8F-\x02H\x5CP!\xF3M\x8B\xAC\xC6\x9F*\x827N\xFD\xEB\xB1\x17\xFF\x96\xFBg\x0E\x8Dn\xA0#p\xAE\x85!\x0DPnc\xB1\x8D\x8E%\xCB\xEB\xE6\x16\xB41\x89\xCE\xB4\x01\xB7\xA0p\x0B\xB0S\x00\xA0\x0E\xE1\x14\xBB\xF4\x9EB\xBA\xD8\x8E\xF8o\xF4\x03"\xD1\x89\xB8\x86\x5C\xECo\x16\x96\x5C,\xC4EH\xACO\x1E\xA4VF\xA7\xFB\xF9\x7F\xAF](\xFC\xE2\x07QQ\xE6\xA8-\xD1l\x0C\xE7\x13YX_[\x89+n&JS \xE5\x81\xF4\xFAH/\xBF\xF4\xBD\x95\x1F\x8F*\x96\x09\xC8Q\xEAoJt2\xA3E'\xA2\x01\xAB\xF2y%B\x18\xBAI)\x02\x0B\x0F\xFBv^\xF4\x85$^>\x14\x84\xA7\xBEb\xFE{N\xEB7W~~\xFB\x5C\x0E=\xD6FG\xB1ms"\x04\x81\xC69\x97i\x16m\xD0\x1F\xBC0iR.\xBB6\xEE\xEF\xDE\x05OF>\x029\x01\xF6\xFBJ\x8D\xA7\xDA\xE8x\xB2\x5C\xED20\x85Z\x09y\x82x\x98\xBAQ~,\x80@\x87r!\xAFa\x8F$\xDB\xC7\x06\x84\xE6\xEAe\xD0\xE2xF\x8E\x9A\x17mtbpb\x0F\xF2\xA4\x13\xBAn\x06j\x86\xFA\x92p\x1F\xBB\x97\xD7WZ\x09\xED(\x15\x8B\xAA\xDE\x09\xB8\xFE\x91\x8C\x0F\x17\x19s\x09\xF6\x95\x8DT\xAC\x0A\x12\xDALq@\xAALB\x8C\xB3\x0A\xCE\xA4k\xA5\x9F\xA25\xDBh.\xA5\xED\xD1(\x17s\x01\xCA\x94Gm\xDB%\x19\x9CI\x9FZw\xB2\xB8\xE9:\x8B\x9B\xD2\xF6\xE0N\x07j\xEDE)\x9C\xBA5"\xA5n\x9D\x088qh/\x85\xE6\xC4\x85\x10\xD9\x0A)=1+\x98\x05\x09\xBE\xCA\x99C\xDD!\x9F\x08\x9D\x94\x0F\xD2\xFA\xD3\xFB\xA6\x91, offset=0, length=1080
        at org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:556)
        at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:344)
        at org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
        at org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
        at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
        at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:214)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:382)
        ... 3 more

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	1m 14s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💛	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
💙	mvndep	0m 35s	Maven dependency ordering for branch
💚	mvninstall	5m 56s	master passed
💚	compile	3m 20s	master passed
💚	checkstyle	2m 55s	master passed
💙	refguide	6m 20s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
💚	shadedjars	5m 1s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	3m 25s	master passed
💙	spotbugs	3m 59s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	19m 16s	master passed
		_ Patch Compile Tests _
💙	mvndep	0m 14s	Maven dependency ordering for patch
💚	mvninstall	5m 39s	the patch passed
💚	compile	3m 21s	the patch passed
💚	javac	3m 21s	the patch passed
💔	checkstyle	2m 59s	root: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
💚	whitespace	0m 0s	The patch has no whitespace issues.
💙	refguide	6m 37s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
💚	shadedjars	5m 46s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	17m 40s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	3m 25s	the patch passed
💚	findbugs	20m 6s	the patch passed
		_ Other Tests _
💔	unit	276m 16s	root in the patch failed.
💚	asflicense	0m 58s	The patch does not generate ASF License warnings.
		398m 21s

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/1/artifact/out/Dockerfile
GITHUB PR	#685
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide
uname	Linux 7b32b648d862 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-685/out/precommit/personality/provided.sh
git revision	master / 7ee6d59
Default Java	1.8.0_181
refguide	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/1/artifact/out/branch-site/book.html
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/1/artifact/out/diff-checkstyle-root.txt
refguide	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/1/artifact/out/patch-site/book.html
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/1/artifact/out/patch-unit-root.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/1/testReport/
Max. process+thread count	4685 (vs. ulimit of 10000)
modules	C: hbase-server . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/1/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
💙	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
💚	dupname	0m 0s	No case conflicting files found.
💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
💚	@author	0m 0s	The patch does not contain any @author tags.
💛	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
💙	mvndep	0m 35s	Maven dependency ordering for branch
💚	mvninstall	5m 50s	master passed
💚	compile	3m 18s	master passed
💚	checkstyle	2m 50s	master passed
💙	refguide	6m 20s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
💚	shadedjars	5m 1s	branch has no errors when building our shaded downstream artifacts.
💚	javadoc	3m 24s	master passed
💙	spotbugs	3m 57s	Used deprecated FindBugs config; considering switching to SpotBugs.
💚	findbugs	20m 3s	master passed
		_ Patch Compile Tests _
💙	mvndep	0m 13s	Maven dependency ordering for patch
💚	mvninstall	5m 26s	the patch passed
💚	compile	3m 16s	the patch passed
💚	javac	3m 16s	the patch passed
💔	checkstyle	2m 57s	root: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
💚	whitespace	0m 0s	The patch has no whitespace issues.
💙	refguide	6m 12s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
💚	shadedjars	5m 1s	patch has no errors when building our shaded downstream artifacts.
💚	hadoopcheck	17m 14s	Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚	javadoc	3m 25s	the patch passed
💚	findbugs	19m 26s	the patch passed
		_ Other Tests _
💔	unit	276m 23s	root in the patch failed.
💚	asflicense	0m 57s	The patch does not generate ASF License warnings.
		396m 15s

Reason	Tests
Failed junit tests	hadoop.hbase.client.TestMobSnapshotCloneIndependence
	hadoop.hbase.util.TestFromClientSide3WoUnsafe

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/2/artifact/out/Dockerfile
GITHUB PR	#685
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide
uname	Linux 18871b027ade 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-685/out/precommit/personality/provided.sh
git revision	master / 5aa8d3a
Default Java	1.8.0_181
refguide	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/2/artifact/out/branch-site/book.html
checkstyle	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/2/artifact/out/diff-checkstyle-root.txt
refguide	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/2/artifact/out/patch-site/book.html
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/2/artifact/out/patch-unit-root.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/2/testReport/
Max. process+thread count	5048 (vs. ulimit of 10000)
modules	C: hbase-server . U: .
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-685/2/console
versions	git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by	Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

[saintstack]

Closing. @wchevreuil reminded me of WALPrettyPrinter which is more than what I needed. Let me make new PR that tries to talk about file formats and tools to use in the refguide.