From 58dcd7d0aa47e01b72b125ae1362c77698a39b29 Mon Sep 17 00:00:00 2001 From: Jon Zeolla Date: Sat, 31 Dec 2016 09:08:49 -0500 Subject: [PATCH 1/2] METRON-643: Stellar function documentation needs to be updated --- metron-analytics/metron-statistics/README.md | 60 ++++++++++---------- metron-platform/metron-common/README.md | 38 ++++++------- 2 files changed, 49 insertions(+), 49 deletions(-) diff --git a/metron-analytics/metron-statistics/README.md b/metron-analytics/metron-statistics/README.md index fa21e0875d..554e34dbb4 100644 --- a/metron-analytics/metron-statistics/README.md +++ b/metron-analytics/metron-statistics/README.md @@ -8,126 +8,126 @@ aimed at capturing a robust set of statistical functions and statistical-based algorithms in the form of Stellar functions. These functions can be used from everywhere where Stellar is used. -##Stellar Functions +## Stellar Functions ### Mathematical Functions -* `ABS` - * Description: Returns the absolute value of a number. - * Input: - * number - The number to take the absolute value of - * Returns: The absolute value of the number passed in. +#### `ABS` +* Description: Returns the absolute value of a number. +* Input: + * number - The number to take the absolute value of +* Returns: The absolute value of the number passed in. ### Distributional Statistics -* `STATS_ADD` +#### `STATS_ADD` * Description: Adds one or more input values to those that are used to calculate the summary statistics. * Input: * stats - The Stellar statistics object. If null, then a new one is initialized. * value+ - One or more numbers to add * Returns: A Stellar statistics object -* `STATS_COUNT` +#### `STATS_COUNT` * Description: Calculates the count of the values accumulated (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The count of the values in the window or NaN if the statistics object is null. -* `STATS_GEOMETRIC_MEAN` +#### `STATS_GEOMETRIC_MEAN` * Description: Calculates the geometric mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The geometric mean of the values in the window or NaN if the statistics object is null. -* `STATS_INIT` +#### `STATS_INIT` * Description: Initializes a statistics object * Input: * window_size - The number of input data values to maintain in a rolling window in memory. If window_size is equal to 0, then no rolling window is maintained. Using no rolling window is less memory intensive, but cannot calculate certain statistics like percentiles and kurtosis. * Returns: A Stellar statistics object -* `STATS_KURTOSIS` +#### `STATS_KURTOSIS` * Description: Calculates the kurtosis of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The kurtosis of the values in the window or NaN if the statistics object is null. -* `STATS_MAX` +#### `STATS_MAX` * Description: Calculates the maximum of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The maximum of the accumulated values in the window or NaN if the statistics object is null. -* `STATS_MEAN` +#### `STATS_MEAN` * Description: Calculates the mean of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The mean of the values in the window or NaN if the statistics object is null. -* `STATS_MERGE` +#### `STATS_MERGE` * Description: Merges statistics objects. * Input: * statistics - A list of statistics objects * Returns: A Stellar statistics object -* `STATS_MIN` +#### `STATS_MIN` * Description: Calculates the minimum of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The minimum of the accumulated values in the window or NaN if the statistics object is null. -* `STATS_PERCENTILE` +#### `STATS_PERCENTILE` * Description: Computes the p'th percentile of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * p - a double where 0 <= p < 1 representing the percentile * Returns: The p'th percentile of the data or NaN if the statistics object is null -* `STATS_POPULATION_VARIANCE` +#### `STATS_POPULATION_VARIANCE` * Description: Calculates the population variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The population variance of the values in the window or NaN if the statistics object is null. -* `STATS_QUADRATIC_MEAN` +#### `STATS_QUADRATIC_MEAN` * Description: Calculates the quadratic mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The quadratic mean of the values in the window or NaN if the statistics object is null. -* `STATS_SD` +#### `STATS_SD` * Description: Calculates the standard deviation of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The standard deviation of the values in the window or NaN if the statistics object is null. -* `STATS_SKEWNESS` +#### `STATS_SKEWNESS` * Description: Calculates the skewness of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The skewness of the values in the window or NaN if the statistics object is null. -* `STATS_SUM` +#### `STATS_SUM` * Description: Calculates the sum of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The sum of the values in the window or NaN if the statistics object is null. -* `STATS_SUM_LOGS` +#### `STATS_SUM_LOGS` * Description: Calculates the sum of the (natural) log of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The sum of the (natural) log of the values in the window or NaN if the statistics object is null. -* `STATS_SUM_SQUARES` +#### `STATS_SUM_SQUARES` * Description: Calculates the sum of the squares of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The sum of the squares of the values in the window or NaN if the statistics object is null. -* `STATS_VARIANCE` +#### `STATS_VARIANCE` * Description: Calculates the variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The variance of the values in the window or NaN if the statistics object is null. - -### Statistical Outlier Detection - -* `OUTLIER_MAD_STATE_MERGE` + + ### Statistical Outlier Detection + +#### `OUTLIER_MAD_STATE_MERGE` * Description: Update the statistical state required to compute the Median Absolute Deviation. * Input: * [state] - A list of Median Absolute Deviation States to merge. Generally these are states across time. * currentState? - The current state (optional) * Returns: The Median Absolute Deviation state -* `OUTLIER_MAD_ADD` +#### `OUTLIER_MAD_ADD` * Description: Add a piece of data to the state. * Input: * state - The MAD state * value - The numeric value to add * Returns: The MAD state -* `OUTLIER_MAD_SCORE` +#### `OUTLIER_MAD_SCORE` * Description: Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf * Input: * state - The MAD state diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md index a6eec3dc55..cf67b5243d 100644 --- a/metron-platform/metron-common/README.md +++ b/metron-platform/metron-common/README.md @@ -86,25 +86,25 @@ Using parens such as: "foo" : "\" requires escaping; "foo": "\'\\'" | [ `REGEXP_MATCH`](#regexp_match)| | [ `SPLIT`](#split)| | [ `STARTS_WITH`](#starts_with)| -| [ `STATS_ADD`](#stats_add)| -| [ `STATS_COUNT`](#stats_count)| -| [ `STATS_GEOMETRIC_MEAN`](#stats_geometric_mean)| -| [ `STATS_INIT`](#stats_init)| -| [ `STATS_KURTOSIS`](#stats_kurtosis)| -| [ `STATS_MAX`](#stats_max)| -| [ `STATS_MEAN`](#stats_mean)| -| [ `STATS_MERGE`](#stats_merge)| -| [ `STATS_MIN`](#stats_min)| -| [ `STATS_PERCENTILE`](#stats_percentile)| -| [ `STATS_POPULATION_VARIANCE`](#stats_population_variance)| -| [ `STATS_QUADRATIC_MEAN`](#stats_quadriatic_mean)| -| [ `STATS_SD`](#stats_sd)| -| [ `STATS_SKEWNESS`](#stats_skewness)| -| [ `STATS_SUM`](#stats_sum)| -| [ `STATS_SUM_LOGS`](#stats_sum_logs)| -| [ `STATS_SUM_SQUARES`](#stats_sum_squares)| -| [ `STATS_VARIANCE`](#stats_variance)| -| [ `SYSTEM_ENV_GET`](#stats_env_get)| +| [ `STATS_ADD`](../../metron-analytics/metron-statistics#stats_add)| +| [ `STATS_COUNT`](../../metron-analytics/metron-statistics#stats_count)| +| [ `STATS_GEOMETRIC_MEAN`](../../metron-analytics/metron-statistics#stats_geometric_mean)| +| [ `STATS_INIT`](../../metron-analytics/metron-statistics#stats_init)| +| [ `STATS_KURTOSIS`](../../metron-analytics/metron-statistics#stats_kurtosis)| +| [ `STATS_MAX`](../../metron-analytics/metron-statistics#stats_max)| +| [ `STATS_MEAN`](../../metron-analytics/metron-statistics#stats_mean)| +| [ `STATS_MERGE`](../../metron-analytics/metron-statistics#stats_merge)| +| [ `STATS_MIN`](../../metron-analytics/metron-statistics#stats_min)| +| [ `STATS_PERCENTILE`](../../metron-analytics/metron-statistics#stats_percentile)| +| [ `STATS_POPULATION_VARIANCE`](../../metron-analytics/metron-statistics#stats_population_variance)| +| [ `STATS_QUADRATIC_MEAN`](../../metron-analytics/metron-statistics#stats_quadratic_mean)| +| [ `STATS_SD`](../../metron-analytics/metron-statistics#stats_sd)| +| [ `STATS_SKEWNESS`](../../metron-analytics/metron-statistics#stats_skewness)| +| [ `STATS_SUM`](../../metron-analytics/metron-statistics#stats_sum)| +| [ `STATS_SUM_LOGS`](../../metron-analytics/metron-statistics#stats_sum_logs)| +| [ `STATS_SUM_SQUARES`](../../metron-analytics/metron-statistics#stats_sum_squares)| +| [ `STATS_VARIANCE`](../../metron-analytics/metron-statistics#stats_variance)| +| [ `SYSTEM_ENV_GET`](#system_env_get)| | [ `SYSTEM_PROPERTY_GET`](#system_property_get)| | [ `TO_DOUBLE`](#to_double)| | [ `TO_EPOCH_TIMESTAMP`](#to_epoch_timestamp)| From 792acb93d4a412209e3930fb517cc4bc69c22c12 Mon Sep 17 00:00:00 2001 From: Jon Zeolla Date: Sat, 31 Dec 2016 09:12:01 -0500 Subject: [PATCH 2/2] METRON-643: Stellar function documentation needs to be updated --- metron-analytics/metron-statistics/README.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/metron-analytics/metron-statistics/README.md b/metron-analytics/metron-statistics/README.md index 554e34dbb4..edc68e1f74 100644 --- a/metron-analytics/metron-statistics/README.md +++ b/metron-analytics/metron-statistics/README.md @@ -26,87 +26,104 @@ functions can be used from everywhere where Stellar is used. * stats - The Stellar statistics object. If null, then a new one is initialized. * value+ - One or more numbers to add * Returns: A Stellar statistics object + #### `STATS_COUNT` * Description: Calculates the count of the values accumulated (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The count of the values in the window or NaN if the statistics object is null. + #### `STATS_GEOMETRIC_MEAN` * Description: Calculates the geometric mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The geometric mean of the values in the window or NaN if the statistics object is null. + #### `STATS_INIT` * Description: Initializes a statistics object * Input: * window_size - The number of input data values to maintain in a rolling window in memory. If window_size is equal to 0, then no rolling window is maintained. Using no rolling window is less memory intensive, but cannot calculate certain statistics like percentiles and kurtosis. * Returns: A Stellar statistics object + #### `STATS_KURTOSIS` * Description: Calculates the kurtosis of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The kurtosis of the values in the window or NaN if the statistics object is null. + #### `STATS_MAX` * Description: Calculates the maximum of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The maximum of the accumulated values in the window or NaN if the statistics object is null. + #### `STATS_MEAN` * Description: Calculates the mean of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The mean of the values in the window or NaN if the statistics object is null. + #### `STATS_MERGE` * Description: Merges statistics objects. * Input: * statistics - A list of statistics objects * Returns: A Stellar statistics object + #### `STATS_MIN` * Description: Calculates the minimum of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The minimum of the accumulated values in the window or NaN if the statistics object is null. + #### `STATS_PERCENTILE` * Description: Computes the p'th percentile of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * p - a double where 0 <= p < 1 representing the percentile * Returns: The p'th percentile of the data or NaN if the statistics object is null + #### `STATS_POPULATION_VARIANCE` * Description: Calculates the population variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The population variance of the values in the window or NaN if the statistics object is null. + #### `STATS_QUADRATIC_MEAN` * Description: Calculates the quadratic mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The quadratic mean of the values in the window or NaN if the statistics object is null. + #### `STATS_SD` * Description: Calculates the standard deviation of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The standard deviation of the values in the window or NaN if the statistics object is null. + #### `STATS_SKEWNESS` * Description: Calculates the skewness of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The skewness of the values in the window or NaN if the statistics object is null. + #### `STATS_SUM` * Description: Calculates the sum of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The sum of the values in the window or NaN if the statistics object is null. + #### `STATS_SUM_LOGS` * Description: Calculates the sum of the (natural) log of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: * stats - The Stellar statistics object * Returns: The sum of the (natural) log of the values in the window or NaN if the statistics object is null. + #### `STATS_SUM_SQUARES` * Description: Calculates the sum of the squares of the accumulated values (or in the window if a window is used). * Input: * stats - The Stellar statistics object * Returns: The sum of the squares of the values in the window or NaN if the statistics object is null. + #### `STATS_VARIANCE` * Description: Calculates the variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics * Input: @@ -121,12 +138,14 @@ functions can be used from everywhere where Stellar is used. * [state] - A list of Median Absolute Deviation States to merge. Generally these are states across time. * currentState? - The current state (optional) * Returns: The Median Absolute Deviation state + #### `OUTLIER_MAD_ADD` * Description: Add a piece of data to the state. * Input: * state - The MAD state * value - The numeric value to add * Returns: The MAD state + #### `OUTLIER_MAD_SCORE` * Description: Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf * Input: