Aberrant Behavior Detection support. A brief overview added to rrdtoo…

…l.pod.

Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
This is backwards compatible! But new files using the Aberrant stuff are not readable
by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
-- Jake Brutlag <jakeb@corp.webtv.net>


git-svn-id: svn://svn.oetiker.ch/rrdtool/trunk/program@26 a5681a0c-68f1-0310-ab6d-d61299d08faa

NEWS

@@ -0,0 +1,18 @@
+RRDTOOL NEWS
+============
+
+In this file I am noting the Major changes to rrdtool
+for details check the cvs ChangeLog
+
+2001/03/21 Tobias Oetiker <oetiker@ee.ethz.ch>
+  Added Aberrant Patch from Jake Brutlag <jakeb@corp.webtv.net>
+  From now one, new rrd files use version tag 0002. They can
+  NOT be read by the old 1.0.x rrdtools
+
+  Jack:
+  Aberrant Behavior Detection support. A brief overview added to
+  rrdtool.pod.  Major updates to rrd_update.c, rrd_create.c. Minor update to
+  other core files.  Updated documentation: rrdcreate.pod, rrdgraph.pod,
+  rrdtune.pod.  This is backwards compatible!  See
+  http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+

doc/rrdcreate.pod

@@ -10,7 +10,7 @@ B<rrdtool> B<create> I<filename>
 S<[B<--start>|B<-b> I<start time>]> 
 S<[B<--step>|B<-s> I<step>]> 
 S<[B<DS:>I<ds-name>B<:>I<DST>B<:>I<heartbeat>B<:>I<min>B<:>I<max>]>
-S<[B<RRA:>I<CF>B<:>I<xff>B<:>I<steps>B<:>I<rows>]>
+S<[B<RRA:>I<CF>B<:>I<cf arguments>]>
 
 =head1 DESCRIPTION
 
@@ -106,30 +106,174 @@ I<If information on minimal/maximal expected values is available,
 always set the min and/or max properties. This will help RRDtool in
 doing a simple sanity check on the data supplied when running update.>
 
-=item B<RRA:>I<CF>B<:>I<xff>B<:>I<steps>B<:>I<rows>
-
+=item B<RRA:>I<CF>B<:>I<cf arguments>
+  
 The purpose of an B<RRD> is to store data in the round robin archives
-(B<RRA>). An archive consists of a number of data values from all the
-defined data-sources (B<DS>) and is defined with an B<RRA> line.
+(B<RRA>). An archive consists of a number of data values or statistics for 
+each of the defined data-sources (B<DS>) and is defined with an B<RRA> line.
 
 When data is entered into an B<RRD>, it is first fit into time slots of
 the length defined with the B<-s> option becoming a I<primary data point>.
 
-The data is also consolidated with the consolidation function (I<CF>)
-of the archive. The following consolidation functions are defined:
-B<AVERAGE>, B<MIN>, B<MAX>, B<LAST>.
+The data is also processed with the consolidation function (I<CF>)
+of the archive. There are several consolidation functions that consolidate
+primary data points via an aggregate function: B<AVERAGE>, B<MIN>, B<MAX>, B<LAST>.
+The format of B<RRA> line for these consolidation function is:
+
+B<RRA:>I<AVERAGE | MIN | MAX | LAST>B<:>I<xff>B<:>I<steps>B<:>I<rows>
 
 I<xff> The xfiles factor defines what part of a consolidation interval may
 be made up from I<*UNKNOWN*> data while the consolidated value is still
 regarded as known.
 
-I<steps> defines how many of these I<primary data points> are used to
-build a I<consolidated data point> which then goes into the archive.
+I<steps> defines how many of these I<primary data points> are used to build
+a I<consolidated data point> which then goes into the archive.
 
 I<rows> defines how many generations of data values are kept in an B<RRA>.
 
 =back
 
+=head1 Aberrant Behaviour detection with Holt-Winters forecasting
+
+by Jake Brutlag E<lt>jakeb@corp.webtv.netE<gt>
+
+In addition to the aggregate functions, there are a set of specialized
+functions that enable B<RRDtool> to provide data smoothing (via the
+Holt-Winters forecasting algorithm), confidence bands, and the flagging
+aberrant behavior in the data source time series:
+
+=over 4
+
+=item B<RRA:>I<HWPREDICT>B<:>I<rows>B<:>I<alpha>B<:>I<beta>B<:>I<seasonal period>B<:>I<rra num>
+
+=item B<RRA:>I<SEASONAL>B<:>I<seasonal period>B<:>I<gamma>B<:>I<rra num>
+
+=item B<RRA:>I<DEVSEASONAL>B<:>I<seasonal period>B<:>I<gamma>B<:>I<rra num>
+
+=item B<RRA:>I<DEVPREDICT>B<:>I<rows>B<:>I<rra num>
+
+=item B<RRA:>I<FAILURES>B<:>I<rows>B<:>I<threshold>B<:>I<window length>B<:>I<rra num>
+
+=back
+
+These B<RRAs> differ from the true consolidation functions in several ways.
+First, each of the B<RRA>s is updated once for every primary data point.
+Second, these B<RRAs> are interdependent. To generate real-time confidence
+bounds, then a matched set of HWPREDICT, SEASONAL, DEVSEASONAL, and
+DEVPREDICT must exist. Generating smoothed values of the primary data points
+requires both a HWPREDICT B<RRA> and SEASONAL B<RRA>. Aberrant behavior
+detection requires FAILURES, HWPREDICT, DEVSEASONAL, and SEASONAL.
+
+The actual predicted, or smoothed, values are stored in the HWPREDICT
+B<RRA>. The predicted deviations are store in DEVPREDICT (think a standard
+deviation which can be scaled to yield a confidence band). The FAILURES
+B<RRA> stores binary indicators. A 1 marks the indexed observation a
+failure; that is, the number of confidence bounds violations in the
+preceding window of observations met or exceeded a specified threshold. An
+example of using these B<RRAs> to graph confidence bounds and failures
+appears in L<rrdgraph>.
+
+The SEASONAL and DEVSEASONAL B<RRAs> store the seasonal coefficients for the
+Holt-Winters Forecasting algorithm and the seasonal deviations respectively.
+There is one entry per observation time point in the seasonal cycle. For
+example, if primary data points are generated every five minutes, and the
+seasonal cycle is 1 day, both SEASONAL and DEVSEASONAL with have 288 rows.
+
+In order to simplify the creation for the novice user, in addition to
+supporting explicit creation the HWPREDICT, SEASONAL, DEVPREDICT,
+DEVSEASONAL, and FAILURES B<RRAs>, the B<rrdtool> create command supports
+implicit creation of the other four when HWPREDICT is specified alone and
+the final argument I<rra num> is omitted.
+
+I<rows> specifies the length of the B<RRA> prior to wrap around. Remember
+that there is a one-to-one correspondence between primary data points and
+entries in these RRAs. For the HWPREDICT CF, I<rows> should be larger than
+the I<seasonal period>. If the DEVPREDICT B<RRA> is implicity created, the
+default number of rows is the same as the HWPREDICT I<rows> argument. If the
+FAILURES B<RRA> is implicitly created, I<rows> will be set to the I<seasonal
+period> argument of the HWPREDICT B<RRA>. Of course, the B<rrdtool>
+I<resize> command is available if these defaults are not sufficient and the
+create wishes to avoid explicit creations of the other specialized function
+B<RRAs>.
+
+I<seasonal period> specifies the number of primary data points in a seasonal
+cycle. If SEASONAL and DEVSEASONAL are implicitly created, this argument for
+those B<RRAs> is set automatically to the value specified by HWPREDICT. If
+they are explicity created, the creator should verify that all three
+I<seasonal period> arguments agree.
+
+I<alpha> is the adaptation parameter of the intercept (or baseline)
+coefficient in the Holt-Winters Forecasting algorithm. See L<rrdtool> for a
+description of this algorithm. I<alpha> must lie between 0 and 1. A value
+closer to 1 means that more recent observations carry greater weight in
+predicting the baseline component of the forecast. A value closer to 0 mean
+that past history carries greater weight in predicted the baseline
+component.
+
+I<beta> is the adaption parameter of the slope (or linear trend) coefficient
+in the Holt-Winters Forecating algorihtm. I<beta> must lie between 0 and 1
+and plays the same role as I<alpha> with respect to the predicted linear
+trend.
+
+I<gamma> is the adaption parameter of the seasonal coefficients in the
+Holt-Winters Forecasting algorithm (HWPREDICT) or the adaption parameter in
+the exponential smoothing update of the seasonal deviations. It must lie
+between 0 and 1. If the SEASONAL and DEVSEASONAL B<RRAs> are created
+implicitly, they will both have the same value for I<gamma>: the value
+specified for the HWPREDICT I<alpha> argument. Note that because there is
+one seasonal coefficient (or deviation) for each time point during the
+seasonal cycle, the adaption rate is much slower than the baseline. Each
+seasonal coefficient is only updated (or adapts) when the observed value
+occurs at the offset in the seasonal cycle corresponding to that
+coefficient.
+
+If SEASONAL and DEVSEASONAL B<RRAs> are created explicity, I<gamma> need not
+be the same for both. Note that I<gamma> can also be changed via the
+B<rrdtool> I<tune> command.
+
+I<rra num> provides the links between related B<RRAs>. If HWPREDICT is
+specified alone and the other B<RRAs> created implicitly, then there is no
+need to worry about this argument. If B<RRAs> are created explicitly, then
+pay careful attention to this argument. For each B<RRA> which includes this
+argument, there is a dependency between that B<RRA> and another B<RRA>. The
+I<rra num> argument is the 1-based index in the order of B<RRA> creation
+(that is, the order they appear in the I<create> command). The dependent
+B<RRA> for each B<RRA> requiring the I<rra num> argument is listed here:
+
+=over 4
+
+=item *
+
+HWPREDICT I<rra num> is the index of the SEASONAL B<RRA>.
+
+=item * 
+
+SEASONAL I<rra num> is the index of the HWPREDICT B<RRA>.
+
+=item * 
+
+DEVPREDICT I<rra num> is the index of the DEVSEASONAL B<RRA>.
+
+=item *
+
+DEVSEASONAL I<rra num> is the index of the HWPREDICT B<RRA>.
+
+=item * 
+
+FAILURES I<rra num> is the index of the DEVSEASONAL B<RRA>.
+
+=back
+
+I<threshold> is the minimum number of violations (observed values outside
+the confidence bounds) within a window that constitutes a failure. If the
+FAILURES B<RRA> is implicitly created, the default value is 7.
+
+I<window length> is the number of time points in the window. Specify an
+integer greater than or equal to the threshold and less than or equal to 28.
+The time interval this window represents depends on the interval between
+primary data points. If the FAILURES B<RRA> is implicity created, the
+default value is 9.
+
 =head1 The HEARTBEAT and the STEP
 
 Here is an explanation by Don Baarda on the inner workings of rrdtool.
@@ -232,6 +376,46 @@ every hour (12 * 300 seconds = 1 hour), for 100 days (2400 hours). The
 third and the fourth RRA's do the same with the for the maximum and
 average temperature, respectively.
 
+=head1 EXAMPLE 2
+
+C<rrdtool create monitor.rrd --step 300 
+DS:ifOutOctets:COUNTER:1800:0:4294967295
+RRA:AVERAGE:0.5:1:2016
+RRA:HWPREDICT:1440:0.1:0.0035:288>
+
+This example is a monitor of a router interface. The first B<RRA> tracks the
+traffic flow in octects; the second B<RRA> generates the specialized
+functions B<RRAs> for aberrant behavior detection. Note that the I<rra num>
+argument of HWPREDICT is missing, so the other B<RRAs> will be implicitly be
+created with default parameter values. In this example, the forecasting
+algorithm baseline adapts quickly; in fact the most recent one hour of
+observations (each at 5 minute intervals) account for 75% of the baseline
+prediction. The linear trend forecast adapts much more slowly. Observations
+made in during the last day (at 288 observations per day) account for only
+65% of the predicted linear trend. Note: these computations rely on an
+exponential smoothing formula described in a forthcoming LISA 2000 paper.
+
+The seasonal cycle is one day (288 data points at 300 second intervals), and
+the seasonal adaption paramter will be set to 0.1. The RRD file will store 5
+days (1440 data points) of forecasts and deviation predictions before wrap
+around. The file will store 1 day (a seasonal cycle) of 0-1 indicators in
+the FAILURES B<RRA>.
+
+The same RRD file and B<RRAs> are created with the following command, which explicitly
+creates all specialized function B<RRAs>.
+
+C<rrdtool create monitor.rrd --step 300 
+DS:ifOutOctets:COUNTER:1800:0:4294967295
+RRA:AVERAGE:0.5:1:2016
+RRA:HWPREDICT:1440:0.1:0.0035:288:3
+RRA:SEASONAL:288:0.1:2
+RRA:DEVPREDICT:1440:5
+RRA:DEVSEASONAL:288:0.1:2
+RRA:FAILURES:288:7:9:5>
+
+Of course, explicit creation need not replicate implicit create, a number of arguments 
+could be changed.
+
 =head1 AUTHOR
 
 Tobias Oetiker E<lt>oetiker@ee.ethz.chE<gt>

doc/rrdgraph.pod

@@ -41,6 +41,7 @@ S<[B<VRULE:>I<time>B<#>I<rrggbb>[B<:>I<legend>]]>
 S<[B<LINE>{B<1>|B<2>|B<3>}B<:>I<vname>[B<#>I<rrggbb>[B<:>I<legend>]]]>
 S<[B<AREA:>I<vname>[B<#>I<rrggbb>[B<:>I<legend>]]]>
 S<[B<STACK:>I<vname>[B<#>I<rrggbb>[B<:>I<legend>]]]>
+S<[B<TICK:>I<vname>B<#>I<rrggbb>[B<:>I<axis-fraction>[B<:>I<legend>]]]>
 
 =head1 DESCRIPTION
 
@@ -472,10 +473,18 @@ B<AREA> or B<LINE?> -- you need something to stack something onto in
 the first place ;) 
 
 Note, that when you STACK onto *UNKNOWN* data, rrdtool will not draw
-any graphics ... *UNKNOWN* is not zero ... if you want it to zero
+any graphics ... *UNKNOWN* is not zero ... if you want it to be zero
 then you might want to use a CDEF argument with IF and UN functions to
 turn *UNKNOWN* into zero ...
 
+=item B<TICK:>I<vname>B<#>I<rrggbb>[B<:>I<axis-fraction>[B<:>I<legend>]]
+
+Plot a tick mark (a vertical line) for each value of I<vname> that is
+non-zero and not *UNKNOWN*. The I<axis-fraction> argument specifies the
+length of the tick mark as a fraction of the y-axis; the default value
+is 0.1 (10% of the axis). Note that the color specification is not
+optional.
+
 =back
 
 =head1 NOTES on legend arguments
@@ -587,6 +596,61 @@ Note that this example assumes that your data is in the positive half of the y-a
 otherwhise you would would have to add NEGINF in order to extend the coverage
 of the rea to whole graph.
 
+=head1 EXAMPLE 4
+
+If the specialized function B<RRAs> exist for aberrant behavior detection, they
+can be used to generate the graph of a time series with confidence bands and
+failures.
+
+   rrdtool graph example.gif \
+          DEF:obs=monitor.rrd:ifOutOctets:AVERAGE \
+          DEF:pred=monitor.rrd:ifOutOctets:HWPREDICT \
+          DEF:dev=monitor.rrd:ifOutOctets:DEVPREDICT \
+          DEF:fail=monitor.rrd:ifOutOctets:FAILURES \
+          TICK:fail#ffffa0:1.0:"Failures\: Average bits out" \
+          CDEF:scaledobs=obs,8,* \
+          CDEF:upper=pred,dev,2,*,+ \
+          CDEF:lower=pred,dev,2,*,- \
+          CDEF:scaledupper=upper,8,* \
+          CDEF:scaledlower=lower,8,* \
+          LINE2:scaledobs#0000ff:"Average bits out" \
+          LINE1:scaledupper#ff0000:"Upper Confidence Bound: Average bits out" \
+          LINE1:scaledlower#ff0000:"Lower Confidence Bound: Average bits out"
+
+This example generates a graph of the data series in blue (LINE2 with the scaledobs
+virtual data source), confidence bounds in red (scaledupper and scaledlower virtual
+data sources), and potential failures (i.e. potential aberrant aberrant behavior)
+marked by vertical yellow lines (the fail data source).
+
+The raw data comes from an AVERAGE B<RRA>, the finest resolution of the observed
+time series (one consolidated data point per primary data point). The predicted
+(or smoothed) values are stored in the HWPREDICT B<RRA>. The predicted deviations
+(think standard deviation) values are stored in the DEVPREDICT B<RRA>. Finally,
+the FAILURES B<RRA> contains indicators, with 1 denoting a potential failure.
+
+All of the data is rescaled to bits (instead of Octets) by multiplying by 8.
+The confidence bounds are computed by an offset of 2 deviations both above
+and below the predicted values (the CDEFs upper and lower). Vertical lines
+indicated potential failures are graphed via the TICK graph element, which
+converts non-zero values in an B<RRA> into tick marks. Here an axis-fraction
+argument of 1.0 means the tick marks span the entire y-axis, and hence become
+vertical lines on the graph.
+
+The choice of 2 deviations (a scaling factor) matches the default used internally
+by the FAILURES B<RRA>. If the internal value is changed (see L<rrdtune>), this
+graphing command should be changed to be consistent.
+
+=head2 A note on data reduction:
+
+The B<rrdtool> I<graph> command is designed to plot data at a specified temporal
+resolution, regardless of the actually resolution of the data in the RRD file.
+This can present a problem for the specialized consolidation functions which
+maintain a one-to-one mapping between primary data points and consolidated
+data points. If a graph insists on viewing the contents of these B<RRAs> on a
+coarser temporal scale, the I<graph> command tries to do something intelligent,
+but the confidence bands and failures no longer have the same meaning and may
+be misleading.
+
 =head1 AUTHOR
 
 Tobias Oetiker E<lt>oetiker@ee.ethz.chE<gt>