A very thin, simply managed host agent written in Node.js
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
etc
examples
freebsd-init
lib
linux-init
man
packaging
plugins
sbin
smf
.eslintrc.js
.gitignore
.node_version
DEVELOPMENT.md
LICENSE
Makefile
README.md
Vagrantfile
install-sh
mkinstalldirs
package.json

README.md

NAD - Node Agent Daemon


Overview

NAD is a portable, extensible, lightweight metric collection agent. It is the recommended way to collect system metrics for the Circonus monitoring platform.

NAD comes with a rich set of plugins which collect:

Further, applications can be easily added using a simple but powerful plugin system. We welcome further contributions by the community. Just submit a pull request against this repository.

NAD Features:

  • Simple HTTP interface for metric collection
  • Metrics exposed in easy to parse JSON format
  • Supports SSL for securing HTTP interface
  • Full support for histogram metrics
  • Support for Circonus real-time (1s) dashboards and graphing
  • Provides local StatsD interface for application metric submission
  • Multiple data submission paradigms:
    • pull - Circonus collects metrics using HTTP interface
    • reverse - Function behind NAT. NAD initiates secure TCP connection, Circonus uses connection to collect metrics
  • Self-configure with Circonus via the command line with a user-provided JSON configuration file

Installation

Automated Install

The easiest, and recommended, method to install NAD for Linux and Illumos based systems is via the Circonus one-step Installer (COSI). See the COSI documentation for details.

Benefits of using COSI:

  • one command install (fully automated, install NAD, create checks, graphs, worksheets and dashboards)
  • or download the install script and customize to your needs
  • customizable templates for checks, graphs, dashboards and worksheets
  • supports automation via orchestration systems (e.g. ansible, puppet, shell scripts, etc.)
  • cosi-site can be installed and run locally for complete control

Manual Install

For convenience and flexibility, pre-built packages are available for selected platforms from updates.circonus.net. These are self-contained omnibus packages built for the target OS. Packages contain the correct version of NodeJS, binaries for platform-specific plugins, and applicable service configuration. These packages will install NAD in /opt/circonus/nad and configure and start NAD as a service.

At the time of this writing, these are:

  • deb packages - Ubuntu: 14.04, 16.04
  • rpm packages - CentOS: EL6, EL7

An up-to-date list of currently supported platforms is available from COSI (list returned in JSON).

Windows Install

  1. Download from NAD repository releases
  2. Unpack NAD file downloaded from releases
  3. Ensure node v6+ installed
  4. Change to directory where NAD was unpacked
  5. Create the default plugin directory etc/node-agent.d e.g. mkdir etc\node-agent.d
  6. Run npm install npm install
  7. Create nad directory in node_modules e.g. mkdir node_modules\nad
  8. Copy lib/* into node_modules/nad e.g. xcopy lib\*.* node_modules\nad /s /e
  9. Run NAD node sbin\nad.js

Updating

For RPM or DEB based installs, go to updates.circonus.net and retrieve most recent package.

curl -O http://updates.circonus.net/node-agent/packages/<name of rpm file>
rpm -Uvh <name of rpm file>
curl -O http://updates.circonus.net/node-agent/packages/<name of deb file>
dpkg -i <name of deb file>

For pkg based systems (e.g. OmniOS) run pkg update field/nad.

Source Install

Build requirements

  • NodeJS v4.4.5+ must be installed, node and npm available in the PATH.
  • A basic development environment (compiler, GNU make, etc.) in order to build certain plugins. For OmniOS/FreeBSD/etc. you must install and use gmake.

Basic install target

A basic install from source installs NAD in /opt/circonus/nad.

For CentOS/Ubuntu:

git clone https://github.com/circonus-labs/nad.git
cd nad
sudo make install

For Illumos/FreeBSD/OpenBSD:

git clone https://github.com/circonus-labs/nad.git
cd nad
sudo gmake install

OS install targets

In addition to the basic install target, there are OS-specific installation targets. Which will build certain plugins for the specific OS platform, enable default plugins, and install an OS-specific service configuration.

  • make install-ubuntu
  • make install-rhel
  • gmake install-illumos
  • gmake install-freebsd
  • gmake install-openbsd

Install files and directories

path description
Core Directories
/opt/circous base directory
/opt/circonus/nad default installation location
/opt/circonus/nad/bin nad utilities, if applicable
/opt/circonus/nad/etc configurations
/opt/circonus/nad/etc/node-agent.d plugin directory
/opt/circonus/nad/node_agent nad module packages
/opt/circonus/nad/log nad log directory (if applicable)
/opt/circonus/nad/man nad man page
/opt/circonus/nad/sbin nad daemon
/opt/circonus/nad/var/run                 Plugin state directory    
Core Files
/opt/circonus/nad/etc/nad.conf main nad configuration (see Options)
/opt/circonus/nad/sbin/nad nad startup script
/opt/circonus/nad/sbin/nad.js main nad process
Miscellaneous Files
/opt/circonus/nad/bin/nad-log nad log viewer script, if applicable
/opt/circonus/nad/log/nad.log nad log, if applicable
/var/run/nad.pid running nad pid file, if applicable
/lib/systemd/system/nad.service systemd service configuration, if applicable
/etc/init/nad.conf upstart service configuration, if applicable
/var/svc/manifest/network/circonus/nad.xml smf service configuration, if applicable
/var/svc/method/circonus-nad smf method script, if applicable
/etc/rc.d/nad FreeBSD service configuration, if applicable

Running

Command line

/opt/circonus/nad/sbin/nad [options]

As a service

  • Systemd based systems - CentOS 7.x and Ubuntu 16.04
    • Configuration: /lib/systemd/system/nad.service
    • Enable: systemctl enable nad
    • Start: systemctl start nad
  • Upstart based systems - CentOS 6.x and Ubuntu 14.04
    • Configuration: /etc/init/nad.conf
    • Enable: presence of configuration
    • Start: initctl start nad
  • SMF based systems - OmniOS/Illumos/etc.
    • Configuration: /var/svc/manifest/network/circonus/nad.xml
    • Enable: svccfg import /var/svc/manifest/network/circonus/nad.xml
    • Start: svcadm enable nad
  • FreeBSD
    • Configuration: /etc/rc.d/nad
    • Enable: add nad_enable="YES" to /etc/rc.conf
    • Start: service nad start
  • OpenBSD - manual service configuration/installation required

    For example, add the following to your /etc/rc.local:

    if [ -x /opt/circonus/nad/sbin/nad ]; then
       echo -n ' nad'
       /opt/circonus/nad/sbin/nad --daemon --syslog
    fi

    Will start NAD and redirect logging to syslog via the logger command. To redirect logging to a file or elsewhere, replace the --syslog option with redirection e.g. > /tmp/my.log 2>&1.

Interface

NAD exposes an HTTP endpoint, the default is to listen to TCP:2609 (e.g. curl http://127.0.0.1:2609/). The output from all requests is JSON.

URI description
/ run all plugins, consolidate output, return metrics.
/run run all plugins, consolidate output, return metrics.
/run/plugin run a single plugin and return metrics. plugin is file name minus extension.
e.g. the vm.sh plugin becomes /run/vm
/inventory return list of currently enabled and loaded plugins.
/inventory?full return list of currently enabled and loaded plugins with full details for each plugin.

Options

Options are configured in /opt/circonus/nad/etc/nad.conf. Options can be set using their individual environment variables or added to a single NAD_OPTS variable (for backwards compatibility).

Option Description
General
--plugin-dir <dir> Plugin directory.
Default: /opt/circonus/nad/etc/node-agent.d
NAD_PLUGIN_DIR="<dir>"
--listen <spec> Listening IP address and port. (ip|port|ip:port)
Default: 2609
NAD_LISTEN="<spec>"
--no-statsd Disable built-in StatsD interface.
NAD_STATSD="no"
--statsd-config <file> Configuration file for StatsD interface.
Default: none
NAD_STATSD_CONFIG="<file>"
Reverse
-r, --reverse Use reverse connection to broker.
Default: false
NAD_REVERSE="yes"
--cid <cid> Check bundle ID for reverse connection.
Default: from cosi
NAD_REVERSE_CID="<cid>"
--broker-ca <file> CA file for broker reverse connection.
Default: fetch from API
NAD_REVERSE_BROKER_CA="<file>"
--target <target> Target host -- see Target below.
Default: os.hostname()
NAD_REVERSE_TARGET="<target>"
API
--api-key <key> Circonus API Token key.
Default: none
NAD_API_KEY="<key>"
--api-app <app> Circonus API Token app.
Default: nad
NAD_API_APP="<app>"
--api-url <url> Circonus API URL.
Default: https://api.circonus.com/v2/
NAD_API_URL="<url>"
--api-ca <file> CA file for API URL.
Default: none
NAD_API_CA="<file>"
SSL
--ssl-listen <spec> SSL listening IP address and port. (ip|port|ip:port)
Default: none
NAD_SSL_LISTEN="<spec>"
--ssl-cert <file> SSL certificate PEM file, required for SSL.
Default: /opt/circonus/nad/etc/na.crt
NAD_SSL_CERT="<file>"
--ssl-key <file> SSL certificate key PEM file, required for SSL.
Default: /opt/circonus/nad/etc/na.key
NAD_SSL_KEY="<file>"
--ssl-ca <file> SSL CA certificate PEM file, required for SSL w/verify.
Default: /opt/circonus/nad/etc/na.ca
NAD_SSL_CA="<file>"
--ssl-verify Enable SSL verification.
Default: false
NAD_SSL_VERIFY="yes"
Miscellaneous
-u, --uid <id> User id to drop privileges to on start.
Default: nobody
NAD_UID="<id>"
-g, --gid <id> Group id to drop privileges to on start.
Default: nobody
NAD_GID="<id>"
--log-level <level> Log level (trace, debug, info, warn, error, fatal).
Default: info
NAD_LOG_LEVEL="<level>"
-d, --debug Enable debug logging (verbose).
Default: false
-t, --trace Enable trace logging (very verbose).
Default: false
--no-watch Disable automatic plugin-dir rescan on changes. Send SIGHUP to force rescan.
-h, --help Output usage information and exit.
-V, --version Output the version number and exit.
--debugdir Create debug files for each plugin and write to this directory.
Default: none
--wipedebugdir Wipe debug directory clean before each write.
Default: false
-i, --inventory Offline inventory and exit.
Self-configure
--hostname <host> Hostname self-configure to use in check and graph names.
Default: os.hostname()
--brokerid <id> Broker ID for self-configure to use for creating check.
Default: required
--configfile <file> File in plugin-dir for self-configure.
Default: required
DEPRECATED Obsolescence 1/2018
-c <dir> DEPRECATED use --plugin-dir
-p <spec> DEPRECATED use --listen
-s <spec> DEPRECATED use --ssl-listen
-v DEPRECATED use --ssl-verify
--authtoken <token> DEPRECATED use --api-key
--apihost <host> DEPRECATED use --api-url
--apiport <port> DEPRECATED use --api-url
--apipath <path> DEPRECATED use --api-url
--apiprotocol <proto> DEPRECATED use --api-url
--apiverbose DEPRECATED NOP, see --debug
--sslcert <file> DEPRECATED use --ssl-cert
--sslkey <file> DEPRECATED use --ssl-key
--sslca <file> DEPRECATED use --ssl-ca
--cafile <file> DEPRECATED use --broker-ca

Target

Is used by both Reverse and Self-configure.

  1. Reverse will use it to search for a check if a cid is not provided and cosi was not used to setup the host.
  2. Self-configure will use it to configure the check on the broker - it is the host (IP or FQDN) the broker will connect to in order to pull metrics.

Reverse mode

Set up reverse connection for metric collection.

If the host was registered with COSI then the only required parameter is --reverse, the rest of the information will be retrieved from the COSI configuration.

If the host was not registered with COSI then a valid API Token Key must be supplied. If an explicit Check Bundle ID is supplied, NAD will use the check if it is still active. If no Check Bundle ID is supplied, NAD will search for a json:nad check where the check target matches the supplied (or default) --target.

Required:

  • --reverse flag signals nad to setup a reverse connection to the broker.
  • --api-key - optional if cosi configuration exists on host, otherwise, api key is required.

Optional:

  • --cid - will pull from cosi configuration, if available.
  • --target - to enable searching for a check (e.g. on a host not registered by cosi).

Self-configure

DEPRECATED -- use of COSI is recommended.

Providing an API token key without the reverse flag will initiate a self-configuration attempt.

Required:

  • --api-key
  • --target - Note: environment variable NAD_REVERSE_TARGET is not used for self-configure, --target must be specified on command line.
  • --brokerid
  • --configfile

Optional:

  • --hostname

StatsD

NAD-StatsD (nad-statsd) provides support for applications on the local system to send metrics using the StatsD line protocol <metricname>:<value>|<type>. The core StatsD metric types (c, g, s, ms) are supported, as well as, additional types specific to Circonus.

  • h histogram, treated exactly the same as timing (ms) metrics.
  • t text metrics.

Additionally, nad-statsd does not automatically generate derivative metrics from timings since they are represented as histograms in Circonus offering much more flexibility for analysis.

Note: nad-statsd uses a push method for submitting metrics to Circonus, as such, it is not fully compatible with real-time graphing (graphs will update as metrics are received rather than at the normal one second cadence).

Configuration

Place configuration options in a file, the default is /opt/circonus/nad/etc/statsd.json. If configuration saved to a different location, use either the NAD --statsd-config command line option or set NAD_STATSD_CONFIG in nad.conf or environment to indicate where the configuration file is located. e.g. nad --statsd-config=/etc/project_configs/nad_statsd.json.

The default nad-statsd configuration is:

{
    "servers": [
        {
            "server": "udp",
            "address": "127.0.0.1",
            "port": 8125
        }
    ],
    "flush_interval": 10000,
    "group_check_id": null,
    "group_key": "group.",
    "group_counter_op": "sum",
    "group_gauge_op": "average",
    "group_set_op": "sum",
    "host_key": null,
    "host_category": "statsd",
    "send_process_stats": true    
}
  • servers - array of objects - is the same as the StatsD servers list
  • flush_interval - milliseconds - is the same as the StatsD flushInterval
  • group_check_id - /^[0-9]+$/ - the ID (numeric portion of CID) for the group check, default is to retrieve from COSI installation
  • group_key - string - metrics prefixed with this key will be sent to the group check (if enabled)
  • group_counter_op - string - sum or average group counter metrics
  • group_gauge_op - string - sum or average group gauge metrics
  • group_set_op - string - sum or average group set metrics
  • host_key - string - metrics prefixed with this key will be sent to NAD. Show up in Circonus in the host check
  • host_category - string - the category to hold the host metrics e.g. with the default host_category of 'statsd', a metric named 'my_metric' would appear in Circonus as 'statsd`my_metric'
  • send_process_stats - boolean - send nad-statsd module's processing statistics

Group metrics

The nad-statsd module can bifurcate metrics - sending some metrics to NAD (host) and some to a group check (intended to be used by multiple hosts - e.g. a group of web servers). If the --group parameter is provided to COSI when the system is registered, it will create a group check (or use the existing group check if one has already been created from another system registration with the same --group). Additional systems which use the same --group parameter when COSI registers them will also send group metrics to the same group check. This allows application metrics to go to either the host, the group, or both - providing more flexibility in viewing, aggregating and analytics. If an HTTPTrap group check is manually created using the UI, set group_check_id in the nad-statsd configuration file. Additionally, if an HTTPTrap check is created manually it must have asynchronous set to disabled, in the Advanced Configuration section, in order for metrics submitted by systems in the group to be handled correctly.

host_key and group_key

The host_key and group_key are metric name prefixes which determine the disposition of a given metric. The host and group key prefix is removed from the metric name when it is submitted to Circonus.

metric name host_key group_key disposition
foo null null all metrics go to host
preference group
host.foo host. null foo goes to host
blah host. null metrics not prefixed with host. go to group
DEFAULT preference host
group.foo null group. foo goes to group
bar.yadda null group. metrics not prefixed with group. go to host
explicit only
host.foo host. group. foo goes to host
group.foo host. group. foo goes to group
drop_me host. group. all other metrics are ignored

Note: If a group check is not enabled and a group_key is configured -- all metrics destined for group, prefixed with the group_key, will be ignored.

how group metrics are handled

type default option
c sum group_counter_op ('sum' or 'average')
g average group_gauge_op ('sum' or 'average')
ms n/a all samples received are represented
h n/a all samples received are represented
s sum group_set_op ('sum' or 'average')
t n/a note: with text metrics the last one received is used

Plugins

NAD plugins are located in the plugin directory (default: /opt/circonus/nad/etc/node-agent.d, configurable with --plugin-dir option). If the automated or manual install were used, platform specific plugins for the current OS are already built. If the source installation method was used - change to the appropriate directory for the current OS and run make or gmake to build the platform specific plugins for the OS. (e.g. cd /opt/circonus/nad/etc/node-agent.d/linux && make)

Enabling

When NAD starts it scans the plugin directory for plugins to enable. Rudimentary filters are used to determine what is a plugin and what is not. e.g. entry is not a directory, entry has a name in the format name.ext, entry is executable, entry is not a configuration file (extension of .json or .conf), etc. It is recommended that plugins be stored in subdirectories of the plugin directory. Subdirectories are not scanned, those plugins will not be loaded and enabled without an additional step.

To enable a plugin, create a symlink in the plugin directory. For example:

cd /opt/circonus/nad/etc/node-agent.d  # change to plugin directory
ln -s linux/vm.sh .                    # create symlink

The plugin will be automatically found and loaded if file watching is enabled (the default). If file watching is disabled (--no-watch), send a SIGHUP to the NAD process to trigger scanning for plugins.

Disabling

To disable a plugin, delete the symlink in the plugin directory. For example:

cd /opt/circonus/nad/etc/node-agent.d  # change to plugin directory
rm vm.sh                               # delete symlink

The plugin will automatically be purged from the loaded plugins if file watching is enabled (the default). If file watching is disabled (--no-watch), send a SIGHUP to the NAD process to trigger scanning for plugins.

Verify

The output from a plugin can be verified/inspected at any time by making a request for that specific plugin:

curl http://localhost:2609/run/name

where name is the name of the plugin without the extension. NAD will respond with the metrics from that plugin in JSON format.

Inventory

The currently loaded plugin inventory can be seen by making a request to the inventory endpoint.

curl http://localhost:2609/inventory

NAD will respond with a list of the currently loaded plugins. The inventory endpoint supports one argument, ?full, which includes additional details on each plugin. The output of the inventory endpoint is JSON, enabling it to be used by orchestration and monitoring tooling.

Run State Directory

Installation will create a run-state directory in the application directory that should be writable by the non-privileged user that NAD runs as. Plugins can use this directory as scratch space to save local state, such as caching the output of an expensive command or query.

The default location is $PREFIX/nad/var/run.

Custom

For information on creating custom plugins see the Plugin section of DEVELOPMENT.md.