Rename 'custom scripts' to 'plugin scripts'. Cleanup readme.

PLUGIN_SCRIPTS.md

@@ -0,0 +1,66 @@
+# Plugin Scripts
+
+You may have specific pieces of architecture that you would like `instrument_server` to monitor. Pass the `-e` flag to `instrument_server` on startup to enable this script plugins. There are many [existing plugins](examples/) that you may use for your infrastructure, or you can [write your own](#writing-plugin-scripts). We also work with customers to develop plugins on an as-needed basis. Please contact us at [Support@InstrumentalApp.com](mailto:support@instrumentalapp.com) if we can assist with plugin development.
+
+## Installing Plugin Scripts
+
+To install scripts, place them in the directory `$HOME/.instrumental_scripts`. The `instrument_server` process will create this directory if it doesn't exist the first time you run the process with script functionality enabled (`-e`). You may also specify a specific directory to the `instrument_server` process with the `-s` (`--script-location`) flag.
+
+### Permission Requirements
+
+The directory you use for plugin scripts and the scripts themselves must be readable/writable only by owner (`0700`), which must be the same user that the `instrument_server` process runs as. The `instrument_server` process will exit with an error message alerting you to the fact that it cannot use the directory otherwise.
+
+## Writing Plugin Scripts
+
+A script may be a binary or shell script that exists in the scripts directory (`$HOME/.instrumental_scripts`). Each time the `instrument_server` process collects system metrics, it will also execute your script with the following arguments:
+
+* **Argument 1:** The Unix timestamp of the last time this script had been executed, in seconds. If the process has not successfully run by `instrument_server` before, this value will be 0.
+* **Argument 2:** The exit status of the process the last time this script had been executed. If the process has not successfully ran by `instrument_server` before, this value will not be present.
+* **STDIN:**
+ The `STDIN` pipe to your process will contain the output of your script the last time it had been executed. You may use this data to compute differences between the last time your script ran and the current execution. (_The [MySQL script](examples/mysql/mysql_status.rb) uses this to compute rate metrics_)
+* Environment: Any environment variables set for the `instrument_server` process will be available to your process.
+
+Your script is expected to output data in the following format on `STDOUT` in order to be sent back to Instrumental:
+
+```
+METRIC_NAME METRIC_VALUE
+```
+
+or
+
+```
+METRIC_NAME METRIC_VALUE UNIX_TIME_IN_SECONDS
+```
+
+For example, if a script named `application_load` were to report two metrics, `memory` and `load`, to the `instrument_server` process, its output should be:
+
+```
+memory 1024.0
+load 0.7
+```
+
+The `instrument_server` process will submit each metric to Instrumental in the following form:
+
+```
+HOST_NAME.SCRIPT_NAME.METRIC_NAME
+```
+
+Using the previous example, if the `application_load` script ran on a host named `app-0001`, its `memory` and `load` metrics would be submitted to Instrumental as `app-0001.application_load.memory` and `app-0001.application_load.load`.
+
+The optional third parameter of the above formats, `UNIX_TIME_IN_SECONDS`, represents the time under which the submitted metric should be measured. Generally you do not need to provide this value, as `instrument_server` will default to recording the time when it receives the metric from your script as the time under which the measurement should be recorded.
+
+### Exit Codes
+
+If you do not want the output of your script submitted to Instrumental, your process should exit with a non-zero exit code. Its `STDOUT` output will still be provided to your script on the next iteration.
+
+### Errors
+
+You may output error information on `STDERR` of your process, and it will be output to the `instrument_server` log to aid in debugging script behavior.
+
+### Timeouts
+
+Your script is responsible for managing timeouts. The `instrument_server` process will not attempt to terminate your process for you.
+
+### Powershell Integration
+
+Windows administrators may create Powershell scripts following the above guidelines. You should write your output to the STDOUT stream using `Write-Output`. The file extension of the Powershell script must be `.ps1` in order for it to be executd by the Instrument Server process.

README.md

@@ -1,25 +1,21 @@
 # Instrumental Tools
 
-A collection of tools for monitoring servers with Instrumental ([www.instrumentalapp.com](http://www.instrumentalapp.com/)).
-
-## Operating System Support
-
-`instrumental_tools` is currently officially supported on 32-bit and 64-bit Linux, Windows systems and Mac OS X. There are prebuilt packages available for Debian, RHEL and Win32 systems.
+Provides [server monitoring](https://instrumentalapp.com/docs/server-monitoring) for [Instrumental](https://instrumentalapp.com/) through the `instrumental_server` daemon.
 
 ## Installation
+`instrumental_tools` is currently officially supported on 32-bit and 64-bit Linux, Windows systems and Mac OS X. There are prebuilt packages available for Debian, Ubuntu, RHEL and Win32 systems.
 
 Installation instructions for supported platforms is available in [INSTALL.md](INSTALL.md). The recommended installation method is to use a prebuilt package, which will automatically install the application as a service in your operating system's startup list.
 
 Once you've installed the package, you will want to edit the `/etc/instrumental.yml` file with your Instrumental API key. Example `/etc/instrumental.yml` (replace `YOUR_API_KEY` with the API key for your Instrumental project):
 
 ```yaml
----
-api_key: YOUR_API_KEY
+api_key: YOUR_PROJECT_API_KEY
 ```
 
-## Sent Metrics
+## Metrics
 
-The default `instrument_server` behavior will collect metrics on the following data:
+By default, Instrumental Tools will collect metrics on the following server data:
 
 * CPU (`user`, `nice`, `system`, `idle`, `iowait` and `total in use`)
 * Load (at 1 minute, 5 minute and 15 minute intervals)
@@ -29,6 +25,15 @@ The default `instrument_server` behavior will collect metrics on the following d
 * Disk Usage (`percent_utilization` for all mounted disks)
 * Filesystem stats (`open_files`, `max_open_files`)
 
+#### Monitoring Services & Other Processes
+
+Instrumental Tools monitors other processes through a powerful plugin system built on binary and shell scripts. Plugin installation and development instructions are listed in [PLUGIN_SCRIPTS.md](PLUGIN_SCRIPTS.md). Existing plugins include:
+
+* [MySQL](examples/mysql)
+* [Mongo](examples/mongo)
+* [Docker](examples/docker)
+* [Redis](examples/redis)
+
 ## Command Line Usage
 
 Basic usage:
@@ -37,41 +42,34 @@ Basic usage:
 instrument_server -k <API_KEY>
 ```
 
-The API key can also be provided by setting the INSTRUMENTAL_TOKEN environment variable, which eliminates the need to supply the key via command line option.
-
-By default, instrument_server will use the hostname of the current host when reporting metrics, e.g. 'hostname.cpu.in_use'. To specify a different hostname:
+To start `instrument_server` as a daemon:
 
 ```sh
-instrument_server -k <API_KEY> -H <HOSTNAME>
+instrument_server -k <API_KEY> start
 ```
 
-### Running as a Daemon
+The API key can also be provided by setting the INSTRUMENTAL_TOKEN environment variable, which eliminates the need to supply the key via command line option.
 
-To start instrument_server as a daemon:
+By default, instrument_server will use the hostname of the current host when reporting metrics, e.g. 'hostname.cpu.in_use'. To specify a different hostname:
 
-```
-instrument_server -k <API_KEY> start
+```sh
+instrument_server -k <API_KEY> -H <HOSTNAME>
 ```
 
 The `start` command will start and detach the process. You may issue additional commands to the process like:
 
-* `start` - start and detach the process
 * `stop` - stop the currently running `instrument_server` process
 * `restart` - restart the currently running `instrument_server` process
 * `foreground` - run the process in the foreground instead of detaching
 * `status` - display daemon status (running, stopped)
 * `clean` - remove any files created by the daemon
 * `kill` - forcibly halt the daemon and remove its pid file
 
-### Custom Metrics
-
-You can create custom scripts whose output will be sent to Instrumental every time `instrument_server` checks in. You can read more about how to create these scripts at [CUSTOM_METRICS.md](CUSTOM_METRICS.md).
-
 
 ### Capistrano Integration
 
 Add `require "instrumental_tools/capistrano"` to your capistrano
-configuration and instrument_server will be restarted after your
+configuration and `instrument_server` will be restarted after your
 deploy is finished. Additionally, you will need to add a new variable
 to your capistrano file.
 
@@ -86,7 +84,7 @@ after "deploy", "instrumental:restart_instrument_server"
 after "deploy:migrations", "instrumental:restart_instrument_server"
 ```
 
-By default, this will attempt to restart the instrument_server command
+By default, this will attempt to restart the `instrument_server` command
 on all the servers specified in your configuration. If you need to
 limit the servers on which you restart the server, you can do
 something like this in your capistrano configuration:
@@ -95,10 +93,6 @@ something like this in your capistrano configuration:
 namespaces[:instrumental].tasks[:restart_instrument_server].options[:roles] = [:web, :worker]
 ```
 
-### NOTES
-
-Mac OS users: Due to a bug in Ruby, instrument_server can occasionally deadlock ([bug report](http://bugs.ruby-lang.org/issues/5811)).
-
 ## Troubleshooting & Help
 
-We are here to help. Email us at [support@instrumentalapp.com](mailto:support@instrumentalapp.com).
+We are here to help! Email us at [Support@InstrumentalApp.com](mailto:support@instrumentalapp.com).