Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
src
.dbpmdarc
.gitignore
GNUmakefile
Install
README.md
Remove
help
pmdastatsd.1
pmdastatsd.ini

README.md

pmdastatsd - Performance Metric Domain Agent for StatsD

This agent collects StatsD data, aggregates them and makes them available to any Performance Co-Pilot client, which is ideal for easily tracking stats in your application.

Features

  • Counter metric type
  • Gauge metric type
  • Duration metric type, available instances
    • Minimum
    • Maximum
    • Median
    • Average
    • 90th Percentile
    • 95th Percentile
    • 99th Percentile
    • Count
    • Standard deviation
  • Parsing of datagrams either with Ragel or Basic parser (with very simple tests available as of right now)
  • Aggregation of duration metrics either with basic histogram or HDR histogram
  • Labels
  • Logging
  • Stats about agent itself
  • Configuration using
    • .ini files
    • Command line arguments

Installation

Dependencies

Installation steps

  1. Put contents of this repo into $PCP_PMDAS_DIR/statsd/ ($PCP_PMDAS_DIR is sourced from /etc/pcp.conf, which should be available if you have PCP installed)
  2. First make sure you have "STATSD" namespace set to "510" in stdpmid file. How-to
  3. Compile with make
  4. Activate agent with sudo make activate

Uninstallation steps

  1. Run sudo make deactivate within $PCP_PMDAS_DIR/statsd/ directory
  2. Remove the statsd folder

Configuration

Ini file

Agent looks for pmdastatsd.ini within it's root directory by default. It accepts following parameters:

  • max_udp_packet_size - Maximum allowed packet size
    default: 1472
  • port - On which port is agent listening for incoming trafic
    default: 8125
  • verbose - Verbosity level. Prints info about agent execution into logfile. Valid values are 0-2. 0 = Default value, shows config information, read socket state, and first 100 dropped messages. 1 = Shows PMNS and related information. 2 = Most detailed verbosity level, also shows dropped messages above 100
    default: 0 All levels include those belows.
  • debug_output_filename - You can send USR1 signal that 'asks' agent to output basic information about all aggregated metric into a $PCP_LOG_DIR/pmcd/statsd_{name} file.
    default: debug
  • version - Flag controlling whether or not to log current agent version on start
    default: 0
  • parser_type - Flag specifying which algorithm to use for parsing incoming datagrams, 0 = basic, 1 = Ragel
    default: 0
  • duration_aggregation_type - Flag specifying which aggregation scheme to use for duration metrics, 0 = basic, 1 = hdr histogram
    default: 1
  • max_unprocessed_packets - Maximum size of packet queue that the agent will save in memory. There are 2 queues: one for packets that are waiting to be parsed and one for parsed packets before they are aggregated
    default: 2048

Command line arguments

Agent accepts all arguments that any PMDA accepts by default, including those specified above, in following form:

  • --max-udp, -Z
  • --port, -P
  • --verbose, -v
  • --debug, -g
  • --debug-output-filename, -o
  • --version, -s
  • --parser-type, -r
  • --duration-aggregation-type, -a
  • --max-unprocessed-packets-size, -z

In case when an argument is included in both an .ini file and in command line, the values passed via command line take precedence.

Usage

Once started, pmdastatsd will listed on specified address and port for any content in a form of:

<metricname>:<value>|<type>

There may be multiple such messages in single datagram, split by a newline character, so this:

<metricname>:<value>|<type>\n<metricname>:<value>|<type>

is valid as well.

<metricname> = [a-z][a-zA-Z0-9_.]*
<value>      = described further in each metric type
<type>       = 'c'|'g'|'ms'

If debug logging is turned on, agent will log every message parsed and related failures.

All recorded metrics will be available under statsd.* namespace.

Counter metric

Stores metrics as simple counters, adding any incoming values to already existing ones.

<metricname>:<value>|c

Where value is positive number.

Example

After aggregating following messages:

metric:20|c
metric:10|c
metric:3.3|c

Value available to PCP will be:

pminfo -f statsd.metric
-> inst[0 or "/"] value 33.3

Gauge metric

Stores metrics as modifiable values, with an option to either set, increment or decrement values.

<metricname>:<value>|g

Where value can be in a form of:

  • '-{value}', when negative value is supplied agent will substract stored value with value passed
  • '+{value}', when positive value with leading plus sign is supplied agent will add passed value to the value stored
  • '{value}', when value without any leading sign is supplied, agent will set the metric to the passed value

Initial value for metric of gauge type is 0.

Example

After aggregating following messages:

metric:20|g
metric:+10|g
metric:-3.3|g

Value available to PCP will be:

pminfo -f statsd.metric
-> inst [0 or "/"] value 26.7

Duration metric

Aggregates values either via HDR Histogram or simply stores all values and then calculates inst ors from all values received.

<metricname>:<value>|ms

Where value is a positive number.

Example

With larger message count, the values may vary based on selected duration aggregation scheme.

metric:10|ms
metric:20|ms

Values available to PCP will be:

pminfo -f statsd.metric
->
inst [0 or "/min"] value 10
inst [1 or "/max"] value 20
inst [2 or "/median"] value 10
inst [3 or "/average"] value 15
inst [4 or "/percentile90"] value 20
inst [5 or "/percentile95"] value 20
inst [6 or "/percentile99"] value 20
inst [7 or "/count"] value 2
inst [8 or "/std_deviation"] value 5 

Note

Once you send given metricname with specified type, agent will no longer aggregate any messages with same metricname but different type and will throw them away.

Labels

StatsD datagrams may also contain key:value pairs separated by commas like so:

metric,tagX=X,tagW=W:5|c

or so:

metric:5|c|#tagX:X,tagW:W

Where:

  • tagX is key, X is value
  • tagW is key, W is value

Both key and value of such pair are [a-zA-Z0-9_.]{1,}.

Both formats are interchangeble and you may combine them together. When key is not unique, right-most value takes precedence. This is valid:

metric,tagX=1:5|c|#tagX:2

Pair with key tagX will have value of 2.

You may use these labels to map specific values to some PCP instances. PCP labels are also assigned to these PCP instances. Pairs are ordered by key in resulting instance name and label descriptor.

Single label:

metric,tagX=X:5|c

Such payload would map to PCP as follows (non-related labels were ommited):

pminfo -f --labels statsd.metric
->
inst [0 or "/tagX=X"] value 5 
inst [0 or "/tagX=X"] labels {"tagX":"X"}

As shown earlier you may also send payload with multiple labels. When multiple labels are supplied they are split in instance name by '::'. Example:

metric,tagX=X,tagW=W:5|c

This resolves to:

pminfo -f --labels statsd.metric
->
inst [0 or "/tagX=X::tagW=W"] value 5
inst [0 or "/tagX=X::tagW=W"] labels {"tagX":"X","tagW":"W"}

Note

Be mindful of the fact that duration metric type already maps to instances even without any labels. Sending labeled value to a such metric creates another 9 (as there are that many hardcoded) instances.

Example:

metric:200|ms
metric:100|ms
metric:200|ms
metric,target=cpu0:10|ms   
metric,target=cpu0:100|ms
metric,target=cpu0:1000|ms

Creates 18 instances. Duration data type and label name compose instance name in following manner:

pminfo -f --labels statsd.metric
->
...
inst [10 or "/max::target=cpu0"] value 1000
inst [10 or "/max::target=cpu0"] labels {"target":"cpu0"}
...

Hardcoded stats

Agent also exports metrics about itself:

statsd.pmda.received Number of datagrams that the agent has received
statsd.pmda.parsed Number of datagrams that were successfully parsed
statsd.pmda.dropped Number of datagrams that were dropped
statsd.pmda.aggregated Number of datagrams that were aggregated
statsd.pmda.metrics_tracked
  • counter - Number of tracked counter metrics
  • gauge - Number of tracked gauge metrics
  • duration - Number of tracked duration metrics
  • total - Number of tracked metrics total
statsd.pmda.settings.max_udp_packet_size Maximum UDP packet size
statsd.pmda.settings.max_unprocessed_packets Maximum size of unprocessed packets Q
statsd.pmda.settings.verbose Verbosity flag
statsd.pmda.settings.debug Debug flag
statsd.pmda.settings.debug_output_filename Debug output filename
statsd.pmda.settings.port Port that is listened to
statsd.pmda.settings.parser_type Used parser type
statsd.pmda.settings.duration_aggregation_type Used duration aggregation type

These names are blacklisted for user usage. No messages with these names will processed. While not yet reserved, whole statsd.pmda.* namespace is not recommended to use for user metrics.

Roadmap

  • Make sure code is optimized

FAQ

I installed both chan and HdrHistogram_c yet the program won't run... there seem to be .so missing.

You may need to make sure that /usr/local is actually looked into. You may need to add the directory to /etc/ld.so.conf yourself:

tee /etc/ld.so.conf.d/local.conf <<EOF
/usr/local/lib
/usr/local/lib64
EOF

Next, run as root:

ldconfig

to clear linker cache.

Simplified inner logic flow:

Listener -> Parser -> Aggregator => Data structures (PMDA stats and StatsD metrics) guarded by locks <= PCP

Legend:

  • "->" = channel
  • "=>" = shared data structures
You can’t perform that action at this time.