DEPRECATED

This project is no longer in active development.

NAD is obsolete and has been replaced by the circonus-agent.

---

NAD - Node Agent Daemon

• Overview
• Installation
  • Automated recommended
  • Manual
    • Windows
  • Source
  • Updating
• Running NAD
  • Command line
  • As a service
  • NAD HTTP interface
• Configuration options
  • General
  • Reverse
  • API
  • SSL
  • Miscellaneous
• StatsD
  • Configuration
  • Group metrics
• Plugins
  • Enable
  • Disable
  • Verify
  • Run State Directory
  • Creating custom plugins
• NAD Development

---

Overview

NAD is a portable, extensible, lightweight metric collection agent. It is the recommended way to collect system metrics for the Circonus monitoring platform.

NAD comes with a rich set of plugins which collect:

• System metrics on Linux, Illumos (Solaris), FreeBSD and OpenBSD
• Application metrics for MySQL, PostgreSQL, HAProxy, Cassandra and more

Further, applications can be easily added using a simple but powerful plugin system. We welcome further contributions by the community. Just submit a pull request against this repository.

NAD Features:

• Simple HTTP interface for metric collection
• Metrics exposed in easy to parse JSON format
• Supports SSL for securing HTTP interface
• Full support for histogram metrics
• Support for Circonus real-time (1s) dashboards and graphing
• Provides local StatsD interface for application metric submission
• Multiple data submission paradigms:
  • pull - Circonus collects metrics using HTTP interface
  • reverse - Function behind NAT. NAD initiates secure TCP connection, Circonus uses connection to collect metrics
• Self-configure with Circonus via the command line with a user-provided JSON configuration file

Installation

Automated Install

The easiest, and recommended, method to install NAD for Linux and Illumos based systems is via the Circonus one-step Installer (COSI). See the COSI documentation for details.

Benefits of using COSI:

• one command install (fully automated, install NAD, create checks, graphs, worksheets and dashboards)
• or download the install script and customize to your needs
• customizable templates for checks, graphs, dashboards and worksheets
• supports automation via orchestration systems (e.g. ansible, puppet, shell scripts, etc.)
• cosi-site can be installed and run locally for complete control

Manual Install

For convenience and flexibility, pre-built packages are available for selected platforms from updates.circonus.net. These are self-contained omnibus packages built for the target OS. Packages contain the correct version of NodeJS, binaries for platform-specific plugins, and applicable service configuration. These packages will install NAD in /opt/circonus/nad and configure and start NAD as a service.

At the time of this writing, these are:

• deb packages - Ubuntu: 14.04, 16.04
• rpm packages - CentOS: EL6, EL7

An up-to-date list of currently supported platforms is available from COSI (list returned in JSON).

Windows Install

1. Download from NAD repository releases
2. Unpack NAD file downloaded from releases
3. Ensure node v6+ installed
4. Change to directory where NAD was unpacked
5. Create the default plugin directory etc/node-agent.d e.g. mkdir etc\node-agent.d
6. Run npm install npm install
7. Create nad directory in node_modules e.g. mkdir node_modules\nad
8. Copy lib/* into node_modules/nad e.g. xcopy lib\*.* node_modules\nad /s /e
9. Run NAD node sbin\nad.js

Updating

For RPM or DEB based installs, go to updates.circonus.net and retrieve most recent package.

curl -O http://updates.circonus.net/node-agent/packages/<name of rpm file>
rpm -Uvh <name of rpm file>

curl -O http://updates.circonus.net/node-agent/packages/<name of deb file>
dpkg -i <name of deb file>

For pkg based systems (e.g. OmniOS) run pkg update field/nad.

Source Install

Build requirements

• NodeJS v4.4.5+ must be installed, node and npm available in the PATH.
• A basic development environment (compiler, GNU make, etc.) in order to build certain plugins. For OmniOS/FreeBSD/etc. you must install and use gmake.

Basic install target

A basic install from source installs NAD in /opt/circonus/nad.

For CentOS/Ubuntu:

git clone https://github.com/circonus-labs/nad.git
cd nad
sudo make install

For Illumos/FreeBSD/OpenBSD:

git clone https://github.com/circonus-labs/nad.git
cd nad
sudo gmake install

OS install targets

In addition to the basic install target, there are OS-specific installation targets. Which will build certain plugins for the specific OS platform, enable default plugins, and install an OS-specific service configuration.

• make install-ubuntu
• make install-rhel
• gmake install-illumos
• gmake install-freebsd
• gmake install-openbsd

Install files and directories

path	description
Core Directories
/opt/circous	base directory
/opt/circonus/nad	default installation location
/opt/circonus/nad/bin	nad utilities, if applicable
/opt/circonus/nad/etc	configurations
/opt/circonus/nad/etc/node-agent.d	plugin directory
/opt/circonus/nad/node_agent	nad module packages
/opt/circonus/nad/log	nad log directory (if applicable)
/opt/circonus/nad/man	nad man page
/opt/circonus/nad/sbin	nad daemon
/opt/circonus/nad/var/run	Plugin state directory
Core Files
/opt/circonus/nad/etc/nad.conf	main nad configuration (see Options)
/opt/circonus/nad/sbin/nad	nad startup script
/opt/circonus/nad/sbin/nad.js	main nad process
Miscellaneous Files
/opt/circonus/nad/bin/nad-log	nad log viewer script, if applicable
/opt/circonus/nad/log/nad.log	nad log, if applicable
/var/run/nad.pid	running nad pid file, if applicable
/lib/systemd/system/nad.service	systemd service configuration, if applicable
/etc/init/nad.conf	upstart service configuration, if applicable
/var/svc/manifest/network/circonus/nad.xml	smf service configuration, if applicable
/var/svc/method/circonus-nad	smf method script, if applicable
/etc/rc.d/nad	FreeBSD service configuration, if applicable

Running

Command line

/opt/circonus/nad/sbin/nad [options]

As a service

• Systemd based systems - CentOS 7.x and Ubuntu 16.04
  • Configuration: /lib/systemd/system/nad.service
  • Enable: systemctl enable nad
  • Start: systemctl start nad
• Upstart based systems - CentOS 6.x and Ubuntu 14.04
  • Configuration: /etc/init/nad.conf
  • Enable: presence of configuration
  • Start: initctl start nad
• SMF based systems - OmniOS/Illumos/etc.
  • Configuration: /var/svc/manifest/network/circonus/nad.xml
  • Enable: svccfg import /var/svc/manifest/network/circonus/nad.xml
  • Start: svcadm enable nad
• FreeBSD
  • Configuration: /etc/rc.d/nad
  • Enable: add nad_enable="YES" to /etc/rc.conf
  • Start: service nad start
• OpenBSD - manual service configuration/installation required

  For example, add the following to your /etc/rc.local:

  if [ -x /opt/circonus/nad/sbin/nad ]; then
     echo -n ' nad'
     /opt/circonus/nad/sbin/nad --daemon --syslog
  fi

  Will start NAD and redirect logging to syslog via the logger command. To redirect logging to a file or elsewhere, replace the --syslog option with redirection e.g. > /tmp/my.log 2>&1.

Interface

NAD exposes an HTTP endpoint, the default is to listen to TCP:2609 (e.g. curl http://127.0.0.1:2609/). The output from all requests is JSON.

URI	description
/	run all plugins, consolidate output, return metrics.
/run	run all plugins, consolidate output, return metrics.
/run/plugin	run a single plugin and return metrics. plugin is file name minus extension. e.g. the vm.sh plugin becomes /run/vm
/inventory	return list of currently enabled and loaded plugins.
/inventory?full	return list of currently enabled and loaded plugins with full details for each plugin.

Options

Options are configured in /opt/circonus/nad/etc/nad.conf. Options can be set using their individual environment variables or added to a single NAD_OPTS variable (for backwards compatibility).

Option	Description
General
--plugin-dir <dir>	Plugin directory. Default: /opt/circonus/nad/etc/node-agent.d NAD_PLUGIN_DIR="<dir>"
--listen <spec>	Listening IP address and port. (ip|port|ip:port) Default: 2609 NAD_LISTEN="<spec>"
--no-statsd	Disable built-in StatsD interface. NAD_STATSD="no"
--statsd-config <file>	Configuration file for StatsD interface. Default: none NAD_STATSD_CONFIG="<file>"
Reverse
-r, --reverse	Use reverse connection to broker. Default: false NAD_REVERSE="yes"
--cid <cid>	Check bundle ID for reverse connection. Default: from cosi NAD_REVERSE_CID="<cid>"
--broker-ca <file>	CA file for broker reverse connection. Default: fetch from API NAD_REVERSE_BROKER_CA="<file>"
--target <target>	Target host -- see Target below. Default: os.hostname() NAD_REVERSE_TARGET="<target>"
API
--api-key <key>	Circonus API Token key. Default: none NAD_API_KEY="<key>"
--api-app <app>	Circonus API Token app. Default: nad NAD_API_APP="<app>"
--api-url <url>	Circonus API URL. Default: https://api.circonus.com/v2/ NAD_API_URL="<url>"
--api-ca <file>	CA file for API URL. Default: none NAD_API_CA="<file>"
SSL
--ssl-listen <spec>	SSL listening IP address and port. (ip|port|ip:port) Default: none NAD_SSL_LISTEN="<spec>"
--ssl-cert <file>	SSL certificate PEM file, required for SSL. Default: /opt/circonus/nad/etc/na.crt NAD_SSL_CERT="<file>"
--ssl-key <file>	SSL certificate key PEM file, required for SSL. Default: /opt/circonus/nad/etc/na.key NAD_SSL_KEY="<file>"
--ssl-ca <file>	SSL CA certificate PEM file, required for SSL w/verify. Default: /opt/circonus/nad/etc/na.ca NAD_SSL_CA="<file>"
--ssl-verify	Enable SSL verification. Default: false NAD_SSL_VERIFY="yes"
Miscellaneous
-u, --uid <id>	User id to drop privileges to on start. Default: nobody NAD_UID="<id>"
-g, --gid <id>	Group id to drop privileges to on start. Default: nobody NAD_GID="<id>"
--log-level <level>	Log level (trace, debug, info, warn, error, fatal). Default: info NAD_LOG_LEVEL="<level>"
-d, --debug	Enable debug logging (verbose). Default: false
-t, --trace	Enable trace logging (very verbose). Default: false
--no-watch	Disable automatic plugin-dir rescan on changes. Send SIGHUP to force rescan.
-h, --help	Output usage information and exit.
-V, --version	Output the version number and exit.
--debugdir	Create debug files for each plugin and write to this directory. Default: none
--wipedebugdir	Wipe debug directory clean before each write. Default: false
-i, --inventory	Offline inventory and exit.
Self-configure
--hostname <host>	Hostname self-configure to use in check and graph names. Default: os.hostname()
--brokerid <id>	Broker ID for self-configure to use for creating check. Default: required
--configfile <file>	File in plugin-dir for self-configure. Default: required
DEPRECATED	Obsolescence 1/2018
-c <dir>	DEPRECATED use --plugin-dir
-p <spec>	DEPRECATED use --listen
-s <spec>	DEPRECATED use --ssl-listen
-v	DEPRECATED use --ssl-verify
--authtoken <token>	DEPRECATED use --api-key
--apihost <host>	DEPRECATED use --api-url
--apiport <port>	DEPRECATED use --api-url
--apipath <path>	DEPRECATED use --api-url
--apiprotocol <proto>	DEPRECATED use --api-url
--apiverbose	DEPRECATED NOP, see --debug
--sslcert <file>	DEPRECATED use --ssl-cert
--sslkey <file>	DEPRECATED use --ssl-key
--sslca <file>	DEPRECATED use --ssl-ca
--cafile <file>	DEPRECATED use --broker-ca

Target

Is used by both Reverse and Self-configure.

1. Reverse will use it to search for a check if a cid is not provided and cosi was not used to setup the host.
2. Self-configure will use it to configure the check on the broker - it is the host (IP or FQDN) the broker will connect to in order to pull metrics.

Reverse mode

Set up reverse connection for metric collection.

If the host was registered with COSI then the only required parameter is --reverse, the rest of the information will be retrieved from the COSI configuration.

If the host was not registered with COSI then a valid API Token Key must be supplied. If an explicit Check Bundle ID is supplied, NAD will use the check if it is still active. If no Check Bundle ID is supplied, NAD will search for a json:nad check where the check target matches the supplied (or default) --target.

Required:

• --reverse flag signals nad to setup a reverse connection to the broker.
• --api-key - optional if cosi configuration exists on host, otherwise, api key is required.

Optional:

• --cid - will pull from cosi configuration, if available.
• --target - to enable searching for a check (e.g. on a host not registered by cosi).

Self-configure

DEPRECATED -- use of COSI is recommended.

Providing an API token key without the reverse flag will initiate a self-configuration attempt.

Required:

• --api-key
• --target - Note: environment variable NAD_REVERSE_TARGET is not used for self-configure, --target must be specified on command line.
• --brokerid
• --configfile

Optional:

• --hostname

StatsD

NAD-StatsD (nad-statsd) provides support for applications on the local system to send metrics using the StatsD line protocol <metricname>:<value>|<type>. The core StatsD metric types (c, g, s, ms) are supported, as well as, additional types specific to Circonus.

• h histogram, treated exactly the same as timing (ms) metrics.
• t text metrics.

Additionally, nad-statsd does not automatically generate derivative metrics from timings since they are represented as histograms in Circonus offering much more flexibility for analysis.

Note: nad-statsd uses a push method for submitting metrics to Circonus, as such, it is not fully compatible with real-time graphing (graphs will update as metrics are received rather than at the normal one second cadence).

Configuration

Place configuration options in a file, the default is /opt/circonus/nad/etc/statsd.json. If configuration saved to a different location, use either the NAD --statsd-config command line option or set NAD_STATSD_CONFIG in nad.conf or environment to indicate where the configuration file is located. e.g. nad --statsd-config=/etc/project_configs/nad_statsd.json.

The default nad-statsd configuration is:

{
    "servers": [
        {
            "server": "udp",
            "address": "127.0.0.1",
            "port": 8125
        }
    ],
    "flush_interval": 10000,
    "group_check_id": null,
    "group_key": "group.",
    "group_counter_op": "sum",
    "group_gauge_op": "average",
    "group_set_op": "sum",
    "host_key": null,
    "host_category": "statsd",
    "send_process_stats": true    
}

• servers - array of objects - is the same as the StatsD servers list
• flush_interval - milliseconds - is the same as the StatsD flushInterval
• group_check_id - /^[0-9]+$/ - the ID (numeric portion of CID) for the group check, default is to retrieve from COSI installation
• group_key - string - metrics prefixed with this key will be sent to the group check (if enabled)
• group_counter_op - string - sum or average group counter metrics
• group_gauge_op - string - sum or average group gauge metrics
• group_set_op - string - sum or average group set metrics
• host_key - string - metrics prefixed with this key will be sent to NAD. Show up in Circonus in the host check
• host_category - string - the category to hold the host metrics e.g. with the default host_category of 'statsd', a metric named 'my_metric' would appear in Circonus as 'statsd`my_metric'
• send_process_stats - boolean - send nad-statsd module's processing statistics

Group metrics

The nad-statsd module can bifurcate metrics - sending some metrics to NAD (host) and some to a group check (intended to be used by multiple hosts - e.g. a group of web servers). If the --group parameter is provided to COSI when the system is registered, it will create a group check (or use the existing group check if one has already been created from another system registration with the same --group). Additional systems which use the same --group parameter when COSI registers them will also send group metrics to the same group check. This allows application metrics to go to either the host, the group, or both - providing more flexibility in viewing, aggregating and analytics. If an HTTPTrap group check is manually created using the UI, set group_check_id in the nad-statsd configuration file. Additionally, if an HTTPTrap check is created manually it must have asynchronous set to disabled, in the Advanced Configuration section, in order for metrics submitted by systems in the group to be handled correctly.

host_key and group_key

The host_key and group_key are metric name prefixes which determine the disposition of a given metric. The host and group key prefix is removed from the metric name when it is submitted to Circonus.

metric name	host_key	group_key	disposition
foo	null	null	all metrics go to host
			preference group
host.foo	host.	null	foo goes to host
blah	host.	null	metrics not prefixed with host. go to group
DEFAULT			preference host
group.foo	null	group.	foo goes to group
bar.yadda	null	group.	metrics not prefixed with group. go to host
			explicit only
host.foo	host.	group.	foo goes to host
group.foo	host.	group.	foo goes to group
drop_me	host.	group.	all other metrics are ignored

Note: If a group check is not enabled and a group_key is configured -- all metrics destined for group, prefixed with the group_key, will be ignored.

how group metrics are handled

type	default	option
c	sum	group_counter_op ('sum' or 'average')
g	average	group_gauge_op ('sum' or 'average')
ms	n/a	all samples received are represented
h	n/a	all samples received are represented
s	sum	group_set_op ('sum' or 'average')
t	n/a	note: with text metrics the last one received is used

Plugins

NAD plugins are located in the plugin directory (default: /opt/circonus/nad/etc/node-agent.d, configurable with --plugin-dir option). If the automated or manual install were used, platform specific plugins for the current OS are already built. If the source installation method was used - change to the appropriate directory for the current OS and run make or gmake to build the platform specific plugins for the OS. (e.g. cd /opt/circonus/nad/etc/node-agent.d/linux && make)

Enabling

When NAD starts it scans the plugin directory for plugins to enable. Rudimentary filters are used to determine what is a plugin and what is not. e.g. entry is not a directory, entry has a name in the format name.ext, entry is executable, entry is not a configuration file (extension of .json or .conf), etc. It is recommended that plugins be stored in subdirectories of the plugin directory. Subdirectories are not scanned, those plugins will not be loaded and enabled without an additional step.

To enable a plugin, create a symlink in the plugin directory. For example:

cd /opt/circonus/nad/etc/node-agent.d  # change to plugin directory
ln -s linux/vm.sh .                    # create symlink

The plugin will be automatically found and loaded if file watching is enabled (the default). If file watching is disabled (--no-watch), send a SIGHUP to the NAD process to trigger scanning for plugins.

Disabling

To disable a plugin, delete the symlink in the plugin directory. For example:

cd /opt/circonus/nad/etc/node-agent.d  # change to plugin directory
rm vm.sh                               # delete symlink

The plugin will automatically be purged from the loaded plugins if file watching is enabled (the default). If file watching is disabled (--no-watch), send a SIGHUP to the NAD process to trigger scanning for plugins.

Verify

The output from a plugin can be verified/inspected at any time by making a request for that specific plugin:

curl http://localhost:2609/run/name

where name is the name of the plugin without the extension. NAD will respond with the metrics from that plugin in JSON format.

Inventory

The currently loaded plugin inventory can be seen by making a request to the inventory endpoint.

curl http://localhost:2609/inventory

NAD will respond with a list of the currently loaded plugins. The inventory endpoint supports one argument, ?full, which includes additional details on each plugin. The output of the inventory endpoint is JSON, enabling it to be used by orchestration and monitoring tooling.

Run State Directory

Installation will create a run-state directory in the application directory that should be writable by the non-privileged user that NAD runs as. Plugins can use this directory as scratch space to save local state, such as caching the output of an expensive command or query.

The default location is $PREFIX/nad/var/run.

Custom

For information on creating custom plugins see the Plugin section of DEVELOPMENT.md.