some work on the docs re: 26 & 29

DEVELOPMENT.md

@@ -0,0 +1,86 @@
+# Riaknostic Development
+
+Riaknostic requires a sane GNU build system and a recent version of
+Erlang.  It has `lager` and `getopt` as dependencies, so those must be
+compatible with your version of Erlang. Release versions are currently
+built with Erlang version R14B03, while development versions are targeted at Erlang version R14B04.
+
+See the `rebar.config` file for more details.
+
+To build Riaknostic, simply run `make`:
+
+```bash
+$ make
+./rebar get-deps
+==> riaknostic (get-deps)
+Pulling lager from {git,"git://github.com/basho/lager",{branch,"master"}}
+Cloning into lager...
+Pulling getopt from {git,"git://github.com/jcomellas/getopt.git","2981dfe"}
+Cloning into getopt...
+==> lager (get-deps)
+==> getopt (get-deps)
+./rebar compile
+==> lager (compile)
+Compiled src/lager_util.erl
+Compiled src/lager_transform.erl
+Compiled src/lager_sup.erl
+Compiled src/lager_mochiglobal.erl
+Compiled src/lager_stdlib.erl
+Compiled src/lager_handler_watcher_sup.erl
+Compiled src/lager_handler_watcher.erl
+Compiled src/lager_trunc_io.erl
+Compiled src/lager_crash_log.erl
+Compiled src/lager_file_backend.erl
+Compiled src/lager_app.erl
+Compiled src/lager.erl
+Compiled src/lager_console_backend.erl
+Compiled src/lager_format.erl
+Compiled src/error_logger_lager_h.erl
+==> getopt (compile)
+Compiled src/getopt.erl
+==> riaknostic (compile)
+Compiled src/riaknostic_check.erl
+Compiled src/riaknostic_util.erl
+Compiled src/riaknostic_node.erl
+Compiled src/riaknostic_check_ring_size.erl
+Compiled src/riaknostic_check_ring_membership.erl
+Compiled src/riaknostic_config.erl
+Compiled src/riaknostic_check_memory_use.erl
+Compiled src/riaknostic_check_nodes_connected.erl
+Compiled src/riaknostic_check_dumps.erl
+Compiled src/riaknostic.erl
+Compiled src/riaknostic_check_disk.erl
+./rebar escriptize
+==> lager (escriptize)
+==> getopt (escriptize)
+==> riaknostic (escriptize)
+```
+
+Now you can invoke the script manually via the below command:
+
+```bash
+$ ./riaknostic --etc ~/code/riak/rel/riak/etc --base ~/code/riak/rel/riak --user `whoami` [other options]
+```
+
+To generate the edoc reference, use `make docs` and then open the
+`doc/index.html` file in your browser. Detailed discussion of the
+internal APIs that you can use in developing new diagnostics is found
+in the edocs.
+
+## Contributing
+
+Have an idea for a diagnostic? Want to improve the way Riaknostic works? Fork the [github repository](https://github.com/basho/riaknostic) and send us a pull-request with your changes! The code is documented with `edoc`, so give the [API Docs](http://riaknostic.basho.com/edoc/index.html) a read before you contribute.
+
+### Developing for Riaknostic Without a Riak Instance
+
+If you want to run the `riaknostic` script while developing, and you don't have it hooked up to your local Riak, you can invoke it directly like so:
+
+```bash
+./riaknostic --etc ~/code/riak/rel/riak/etc --base ~/code/riak/rel/riak --user `whoami` [other options]
+```
+
+The extra options are usually assigned by the `riak-admin` script for you, but here's how to set them:
+
+* `--etc`:	Where your Riak configuration directory is, in the example above it's in the generated directory of a source checkout of Riak.
+* `--base`:	The "base" directory of Riak, usually the root of the generated directory or `/usr/lib/riak` on Linux, for example. Scan the `riak-admin` script for how the `RUNNER_BASE_DIR` variable is assigned on your platform.
+* `--user`:	What user/UID the Riak node runs as. In a source checkout, it's the current user, on most systems, it's `riak`.

README.md

@@ -1,78 +1,134 @@
 # Riaknostic
 
-Riaknostic performs basic diagnostics on Riak nodes. See the
-[Riaknostic homepage](http://riaknostic.basho.com) for user-oriented
-information.
+`riaknostic` is an escript and set of tools that diagnoses common problems which could affect a Riak node or cluster. When experiencing any problem with Riak, `riaknostic` should be the first thing run during troubleshooting. The tool is integrated with Riak via the `riak-admin` script.
 
-## Development
+## Overview
 
-Riaknostic requires a sane GNU build system and a recent version of
-Erlang.  It has `lager` and `getopt` as dependencies, so those must be
-compatible with your version of Erlang. Release versions are currently
-built with R14B03.  See the `rebar.config` file for more details.
+To diagnose problems with Riak, Riaknostic uses a series of checks which are derived from the experience of the Basho Client Services Team as well as numerous public discussions on the mailing list, IRC room, and other online media.
 
-To build Riaknostic, simply run `make`:
+Here is a basic example of using `riaknostic` followed immediately by the command's output:
 
 ```bash
-$ make
-./rebar get-deps
-==> riaknostic (get-deps)
-Pulling lager from {git,"git://github.com/basho/lager",{branch,"master"}}
-Cloning into lager...
-Pulling getopt from {git,"git://github.com/jcomellas/getopt.git","2981dfe"}
-Cloning into getopt...
-==> lager (get-deps)
-==> getopt (get-deps)
-./rebar compile
-==> lager (compile)
-Compiled src/lager_util.erl
-Compiled src/lager_transform.erl
-Compiled src/lager_sup.erl
-Compiled src/lager_mochiglobal.erl
-Compiled src/lager_stdlib.erl
-Compiled src/lager_handler_watcher_sup.erl
-Compiled src/lager_handler_watcher.erl
-Compiled src/lager_trunc_io.erl
-Compiled src/lager_crash_log.erl
-Compiled src/lager_file_backend.erl
-Compiled src/lager_app.erl
-Compiled src/lager.erl
-Compiled src/lager_console_backend.erl
-Compiled src/lager_format.erl
-Compiled src/error_logger_lager_h.erl
-==> getopt (compile)
-Compiled src/getopt.erl
-==> riaknostic (compile)
-Compiled src/riaknostic_check.erl
-Compiled src/riaknostic_util.erl
-Compiled src/riaknostic_node.erl
-Compiled src/riaknostic_check_ring_size.erl
-Compiled src/riaknostic_check_ring_membership.erl
-Compiled src/riaknostic_config.erl
-Compiled src/riaknostic_check_memory_use.erl
-Compiled src/riaknostic_check_nodes_connected.erl
-Compiled src/riaknostic_check_dumps.erl
-Compiled src/riaknostic.erl
-Compiled src/riaknostic_check_disk.erl
-./rebar escriptize
-==> lager (escriptize)
-==> getopt (escriptize)
-==> riaknostic (escriptize)
+$ riak-admin diag
+15:34:52.736 [warning] Riak crashed at Wed, 07 Dec 2011 21:47:50 GMT, leaving
+crash dump in /srv/riak/log/erl_crash.dump. Please inspect or remove the file.
+15:34:52.736 [notice] Data directory /srv/riak/data/bitcask is not mounted with 'noatime'. Please remount its disk with the 'noatime' flag to improve
+performance.
 ```
 
-Now you can invoke the script manually via the below command:
+As shown in the above output, Riaknostic tells us about two problems right away. First, an Erlang crash dump is present, indicating that Riak has experienced a crash. Second, a performance problem is mentioned (disk mounted without `noatime` argument)along with a helpful tip to resolve the issue.
+
+## Installation
+
+NOTE: Riaknostic depends on features introduced by Erlang version R14B04, so verify that you've installed this version of Erlang before proceeding with installation.
+
+To install `riaknostic`, download the latest package version, and extract it within the directory shown for your operating system in the following table:
+
+<table class="bordered-table zebra-striped">
+    <thead>
+        <tr><th>Platform</th><th>Directory</th></tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td>Linux (Redhat, CentOS, Debian, Ubuntu)</td>
+            <td><code>/usr/lib/riak/lib</code></td>
+        </tr>
+        <tr>
+            <td>Linux (Fedora)</td>
+            <td><code>/usr/lib64/riak/lib</code></td>
+        </tr>
+        <tr>
+            <td>Solaris, OpenSolaris</td>
+            <td><code>/opt/riak/lib</code></td>
+        </tr>
+        <tr>
+            <td>Mac OS/X or Self-built</td>
+            <td><code>$RIAK/lib</code>
+                (where <code>$RIAK=rel/riak</code> for source installs,
+                or the directory where you unpacked the package)</td>
+        </tr>
+    </tbody>
+</table>
+
+An example Riaknostic installation for Linux looks like this:
+
+```bash
+wget https://github.com/basho/riaknostic/downloads/riaknostic-1.0.2.tar.gz -P /tmp
+cd /usr/lib/riak/lib
+sudo tar xzvf /tmp/riaknostic-1.0.2.tar.gz
+```
+
+The package will expand to a `riaknostic/` directory which contains the `riaknostic` script, source code in the `src/` directory, and documentation.
+
+Now try it out!
+
+## Usage
+
+For most cases, you can just run the `riak-admin diag` command as given at the top of this README. However, sometimes you might want to know some extra detail or run only specific checks. For that, there are command-line options. Execute `riaknostic --help` to learn more about these options:
 
 ```bash
-$ ./riaknostic --etc ~/code/riak/rel/riak/etc --base ~/code/riak/rel/riak --user `whoami` [other options]
+riak-admin diag --help
+Usage: riak-admin diag [-d <level>] [-l] [-h] [check_name ...]
+
+  -d, --level		Minimum message severity level (default: notice)
+  -l, --list		Describe available diagnostic tasks
+  -h, --help		Display help/usage
+  check_name		A specific check to run
 ```
 
-To generate the edoc reference, use `make docs` and then open the
-`doc/index.html` file in your browser.  Detailed discussion of the
-internal APIs that you can use in developing new diagnostics is found
-in the edocs.
+To get an idea of what checks will be run, use the `--list` option:
+
+```bash
+riak-admin diag --list
+Available diagnostic checks:
+
+  disk                 Data directory permissions and atime
+  dumps                Find crash dumps
+  memory_use           Measure memory usage
+  nodes_connected      Cluster node liveness
+  ring_membership      Cluster membership validity
+  ring_size            Ring size valid
+```
+
+If you want all the gory details about what Riaknostic is doing, you can run the checks at a more verbose logging level with the --level option:
+
+```bash
+riak-admin diag --level debug
+18:34:19.708 [debug] Lager installed handler lager_console_backend into lager_event
+18:34:19.720 [debug] Lager installed handler error_logger_lager_h into error_logger
+18:34:19.720 [info] Application lager started on node nonode@nohost
+18:34:20.736 [debug] Not connected to the local Riak node, trying to connect. alive:false connect_failed:undefined
+18:34:20.737 [debug] Starting distributed Erlang.
+18:34:20.740 [debug] Supervisor net_sup started erl_epmd:start_link() at pid <0.42.0>
+18:34:20.742 [debug] Supervisor net_sup started auth:start_link() at pid <0.43.0>
+18:34:20.771 [debug] Supervisor net_sup started net_kernel:start_link(['riak_diag87813@127.0.0.1',longnames]) at pid <0.44.0>
+18:34:20.771 [debug] Supervisor kernel_sup started erl_distribution:start_link(['riak_diag87813@127.0.0.1',longnames]) at pid <0.41.0>
+18:34:20.781 [debug] Supervisor inet_gethost_native_sup started undefined at pid <0.49.0>
+18:34:20.782 [debug] Supervisor kernel_safe_sup started inet_gethost_native:start_link() at pid <0.48.0>
+18:34:20.834 [debug] Connected to local Riak node 'riak@127.0.0.1'.
+18:34:20.939 [debug] Local RPC: os:getpid([]) [5000]
+18:34:20.939 [debug] Running shell command: ps -o pmem,rss,command -p 83144
+18:34:20.946 [debug] Shell command output:
+%MEM    RSS COMMAND
+ 0.4  31004 /srv/riak/erts-5.8.4/bin/beam.smp -K true -A 64 -W w -- -root /srv/riak/rel/riak -progname riak -- -home /Users/sean -- -boot /srv/riak/releases/1.0.2/riak -embedded -config /srv/riak/etc/app.config -name riak@127.0.0.1 -setcookie riak -- console
+
+18:34:20.960 [warning] Riak crashed at Wed, 07 Dec 2011 21:47:50 GMT, leaving crash dump in /srv/riak/log/erl_crash.dump. Please inspect or remove the file.
+18:34:20.961 [notice] Data directory /srv/riak/data/bitcask is not mounted with 'noatime'. Please remount its disk with the 'noatime' flag to improve performance.
+18:34:20.961 [info] Riak process is using 0.4% of available RAM, totalling 31004 KB of real memory.
+```
+
+Most times you'll want to use the defaults, but any Syslog severity name will do (from most to least verbose): `debug, info, notice, warning, error, critical, alert, emergency`.
+
+Finally, if you want to run just a single diagnostic or a list of specific ones, you can pass their name(s):
+
+```bash
+riak-admin diag dumps
+18:41:24.083 [warning] Riak crashed at Wed, 07 Dec 2011 21:47:50 GMT, leaving crash dump in /srv/riak/log/erl_crash.dump. Please inspect or remove the file.
+```
 
 ## Contributing
 
+0. Read DEVELOPMENT.md
 1. Fork the project on [Github](https://github.com/basho/riaknostic).
 2. Make your changes or additions on a "topic" branch, test and
    document them. If you are making a new diagnostic, make sure you
@@ -81,4 +137,4 @@ in the edocs.
    Riak, only inspect things.
 3. Push to your fork and send a pull-request.
 4. A Basho Developer Advocate or Engineer will review your
-   pull-request and get back to you.
+   pull-request and get back to you.

doc/overview.edoc

@@ -2,14 +2,11 @@
 @copyright 2011 Basho Technologies, Inc.
 @version 1.0.0
 @title riaknostic: Automated diagnostic tools for Riak
-@doc <p>riaknostic is an escript and set of tools that diagnoses common problems that may
-affect a Riak node or cluster.  riaknostic should be the first thing run when experiencing
-any problem with Riak. The tool is integrated with Riak via the <code>riak-admin</code> script.</p>
+@doc <p>riaknostic is an escript and set of tools that diagnoses common problems which could affect a Riak node or cluster. When experiencing any problem with Riak, riaknostic should be the first thing run during troubleshooting. The tool is integrated with Riak via the <code>riak-admin</code> script.</p>
 
 <pre>$ riak-admin diag</pre>
 
-<p>This documentation serves to describe the API of the <code>riaknostic</code> application and
-script that powers the above command.  The core of the application consists of 5 modules:</p>
+<p>This documentation describes the <code>riaknostic</code> API and user interface commands. The application's core consists of 5 modules:</p>
 
 <ul>
 <li><strong>riaknostic</strong> - the core of the script, including CLI parsing and dispatching commands.</li>
@@ -23,4 +20,4 @@ script that powers the above command.  The core of the application consists of 5
 <p>All other included modules are generally prefixed with <code>riaknostic_check_</code> and are individual diagnostics that can be run.</p>
 
 <p>riaknostic is licensed under the Apache v2 license.</p>
-@end
+@end