scripts: add support for parsing ZooKeeper transaction logs

Motivation:

dCache is now strongly dependent on ZooKeeper functioning correctly.  We
have started receiving reports that indicate internal failures within
ZK. Therefore, the ability to diagnose ZK problems is now important.

Modification:

Add a 'dcache' command that parses the ZooKeeper transaction logs.  By
default, it targets the directory that the embedded ZK server uses, but
the command can target other directories.

Result:

The 'dcache' script can now parse ZooKeeper transaction logs as an aid
to diagnosing potential problems.

Require-notes: yes
Require-book: no
Target: master
Request: 4.1
Request: 4.0
Request: 3.2
Request: 3.1
Request: 3.0
Request: 2.16
Ticket: http://rt.dcache.org/Ticket/Display.html?id=9332
Patch: https://rb.dcache.org/r/10913/
Acked-by: Tigran Mkrtchyan

skel/bin/dcache

@@ -45,6 +45,7 @@ usage()
     echo "   database doc <cell>@<domain> <out-dir>"
     echo "   dump heap [--force] <domain> <file>"
     echo "   dump threads [<domain>...]"
+    echo "   dump zklog [<path>]"
     echo "   kpwd <command> [-debug] [<command argument>]..."
     echo "   ports"
     echo "   pool convert <name> <target-type>"
@@ -665,6 +666,24 @@ case "$1" in
                 done
                 ;;
 
+            zklog)
+                [ $# -gt 1 ] && usage
+                dir="$(getProperty zookeeper.data-log-dir)/version-2"
+                [ $# -eq 1 ] && dir="$1"
+
+                [ ! -d "$dir" ] && fail 1 "No such directory: $dir"
+
+                ls -r -t "$dir/log."* 2>/dev/null \
+                   | while read f; do
+                         name=$(basename "$f")
+                         echo
+                         echo "TRANSACTION LOG FILE $name:"
+                         echo
+                         CLASSPATH="$(getProperty dcache.paths.classpath)" quickJava org.apache.zookeeper.server.LogFormatter "$f" \
+                             | awk '{print "  "$0}'
+                     done
+                ;;
+
             *)
                 usage
                 ;;

skel/man/man8/dcache.8

@@ -344,6 +344,13 @@ SERVICE. If no services or domains are specified, stack traces of all
 running domains will be dumped. The information is written to the log
 files of the respective domains.
 
+.TP
+.B dump zklog [PATH]
+List the transactions recorded in the ZooKeeper server log files.
+These are 'log.' files located within PATH.  If PATH is omitted then
+the expanded value of '${zookeeper.data-log-dir}/version-2' is used.
+The transaction information is written to standard out.
+
 .SH DEFINITIONS
 The following definitions are used throughout this document: