ZOOKEEPER-3427: Introduce SnapshotComparer that assists debugging with snapshots.

[mayawang]

SnapshotComparer is a tool that loads and compares two snapshots, with configurable threshold and various filters. It's useful in use cases that involve snapshot analysis, such as offline data consistency checking, and data trending analysis (e.g. what's growing under which zNode path during when).

See detailed usage doc in zookeeperTools.md.

This PR replaces and addresses comments and feedbacks in #984

Change Notes:

1. Made this tool support gz/snappy compressed snapshot formats in addition to uncompressed snapshot format.
2. Fixed a minor bug in CompareLine(), improved error handling and docs.
3. Added shell script to make SnapshotComparer tool more user friendly.
4. Added detailed usage docs in zookeeperTools.md.

Testing:

Style check

mvn -DskipTests checkstyle:check

Local Testing by running shell script and compare different file formats:

Example results:

1. snappy vs gz:

# $ bin/zkSnapshotComparer.sh -l /zookeeper-data/backup/snapshot.0.snappy -r /zookeeper-data/backup/snapshot.8.gz -b 100000 -n 100
...
Deserialized snapshot in snapshot.0.snappy in 0.030391 seconds
Processed data tree in 0.098400 seconds
2020-01-13 00:37:09,089 [myid:] - INFO  [main:WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2020-01-13 00:37:09,089 [myid:] - INFO  [main:WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
Deserialized snapshot in snapshot.8.gz in 0.000734 seconds
Processed data tree in 0.000140 seconds
Node count: 4
Total size: 0
Max depth: 3
Count of nodes at depth 1: 1
Count of nodes at depth 2: 1
Count of nodes at depth 3: 2

Node count: 8
Total size: 0
Max depth: 4
Count of nodes at depth 1: 1
Count of nodes at depth 2: 2
Count of nodes at depth 3: 4
Count of nodes at depth 4: 1

Analysis for depth 0
Analysis for depth 1
Analysis for depth 2
Analysis for depth 3
All layers compared.

2. snappy vs snappy

$ bin/zkSnapshotComparer.sh -l /zookeeper-data/backup/snapshot.0.snappy -r /zookeeper-data/backup/snapshot.d.snappy -b 100000 -n 100
...
Deserialized snapshot in snapshot.d.snappy in 0.000593 seconds
Processed data tree in 0.000211 seconds
Node count: 4
Total size: 0
Max depth: 3
Count of nodes at depth 1: 1
Count of nodes at depth 2: 1
Count of nodes at depth 3: 2

Node count: 10
Total size: 0
Max depth: 4
Count of nodes at depth 1: 1
Count of nodes at depth 2: 2
Count of nodes at depth 3: 4
Count of nodes at depth 4: 3

Analysis for depth 0
Analysis for depth 1
Analysis for depth 2
Analysis for depth 3
All layers compared.

3. uncompressed vs snappy

$ bin/zkSnapshotComparer.sh -l /zookeeper-data/backup/snapshot.0 -r /zookeeper-data/backup/snapshot.d.snappy -b 100000 -n 100
...
Deserialized snapshot in snapshot.0 in 0.003973 seconds
Processed data tree in 0.077579 seconds
2020-01-13 00:41:19,566 [myid:] - INFO  [main:WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2020-01-13 00:41:19,566 [myid:] - INFO  [main:WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
Deserialized snapshot in snapshot.d.snappy in 0.032078 seconds
Processed data tree in 0.000609 seconds
Node count: 4
Total size: 0
Max depth: 3
Count of nodes at depth 1: 1
Count of nodes at depth 2: 1
Count of nodes at depth 3: 2

Node count: 10
Total size: 0
Max depth: 4
Count of nodes at depth 1: 1
Count of nodes at depth 2: 2
Count of nodes at depth 3: 4
Count of nodes at depth 4: 3

Analysis for depth 0
Analysis for depth 1
Analysis for depth 2
Analysis for depth 3
All layers compared.

[maoling]

we also need a zkSnapshotComparer.cmd to support windows OS

[maoling]

and data trending analysis (e.g. what's growing under which zNode path during when).

For my understanding, this tool also needs to tell users what the specific znode paths added, updated, deleted comparing one snapshot to another

[mayawang]

@maoling Thank you very much for the review! Those are all very helpful comments. I just updated #984 to preserve previous discussion. I created this PR because I was blocked by a mvn compilation issue which I was able to workaround for now.

I will close this PR and address your comments in #984.