Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

METRON-643: Stellar function documentation needs to be updated #407

Closed
wants to merge 3 commits into from
Closed

METRON-643: Stellar function documentation needs to be updated #407

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
58dcd7d
METRON-643: Stellar function documentation needs to be updated
JonZeolla Dec 31, 2016
792acb9
METRON-643: Stellar function documentation needs to be updated
JonZeolla Dec 31, 2016
9e395ce
Merge branch 'master' of https://github.com/apache/incubator-metron i…
JonZeolla Jan 5, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
79 changes: 49 additions & 30 deletions metron-analytics/metron-statistics/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,126 +8,145 @@ aimed at capturing a robust set of statistical functions and
statistical-based algorithms in the form of Stellar functions. These
functions can be used from everywhere where Stellar is used.

##Stellar Functions
## Stellar Functions

### Mathematical Functions
* `ABS`
* Description: Returns the absolute value of a number.
* Input:
* number - The number to take the absolute value of
* Returns: The absolute value of the number passed in.
#### `ABS`
* Description: Returns the absolute value of a number.
* Input:
* number - The number to take the absolute value of
* Returns: The absolute value of the number passed in.


### Distributional Statistics

* `STATS_ADD`
#### `STATS_ADD`
* Description: Adds one or more input values to those that are used to calculate the summary statistics.
* Input:
* stats - The Stellar statistics object. If null, then a new one is initialized.
* value+ - One or more numbers to add
* Returns: A Stellar statistics object
* `STATS_COUNT`

#### `STATS_COUNT`
* Description: Calculates the count of the values accumulated (or in the window if a window is used).
* Input:
* stats - The Stellar statistics object
* Returns: The count of the values in the window or NaN if the statistics object is null.
* `STATS_GEOMETRIC_MEAN`

#### `STATS_GEOMETRIC_MEAN`
* Description: Calculates the geometric mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The geometric mean of the values in the window or NaN if the statistics object is null.
* `STATS_INIT`

#### `STATS_INIT`
* Description: Initializes a statistics object
* Input:
* window_size - The number of input data values to maintain in a rolling window in memory. If window_size is equal to 0, then no rolling window is maintained. Using no rolling window is less memory intensive, but cannot calculate certain statistics like percentiles and kurtosis.
* Returns: A Stellar statistics object
* `STATS_KURTOSIS`

#### `STATS_KURTOSIS`
* Description: Calculates the kurtosis of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The kurtosis of the values in the window or NaN if the statistics object is null.
* `STATS_MAX`

#### `STATS_MAX`
* Description: Calculates the maximum of the accumulated values (or in the window if a window is used).
* Input:
* stats - The Stellar statistics object
* Returns: The maximum of the accumulated values in the window or NaN if the statistics object is null.
* `STATS_MEAN`

#### `STATS_MEAN`
* Description: Calculates the mean of the accumulated values (or in the window if a window is used).
* Input:
* stats - The Stellar statistics object
* Returns: The mean of the values in the window or NaN if the statistics object is null.
* `STATS_MERGE`

#### `STATS_MERGE`
* Description: Merges statistics objects.
* Input:
* statistics - A list of statistics objects
* Returns: A Stellar statistics object
* `STATS_MIN`

#### `STATS_MIN`
* Description: Calculates the minimum of the accumulated values (or in the window if a window is used).
* Input:
* stats - The Stellar statistics object
* Returns: The minimum of the accumulated values in the window or NaN if the statistics object is null.
* `STATS_PERCENTILE`

#### `STATS_PERCENTILE`
* Description: Computes the p'th percentile of the accumulated values (or in the window if a window is used).
* Input:
* stats - The Stellar statistics object
* p - a double where 0 <= p < 1 representing the percentile
* Returns: The p'th percentile of the data or NaN if the statistics object is null
* `STATS_POPULATION_VARIANCE`

#### `STATS_POPULATION_VARIANCE`
* Description: Calculates the population variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The population variance of the values in the window or NaN if the statistics object is null.
* `STATS_QUADRATIC_MEAN`

#### `STATS_QUADRATIC_MEAN`
* Description: Calculates the quadratic mean of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The quadratic mean of the values in the window or NaN if the statistics object is null.
* `STATS_SD`

#### `STATS_SD`
* Description: Calculates the standard deviation of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The standard deviation of the values in the window or NaN if the statistics object is null.
* `STATS_SKEWNESS`

#### `STATS_SKEWNESS`
* Description: Calculates the skewness of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The skewness of the values in the window or NaN if the statistics object is null.
* `STATS_SUM`

#### `STATS_SUM`
* Description: Calculates the sum of the accumulated values (or in the window if a window is used).
* Input:
* stats - The Stellar statistics object
* Returns: The sum of the values in the window or NaN if the statistics object is null.
* `STATS_SUM_LOGS`

#### `STATS_SUM_LOGS`
* Description: Calculates the sum of the (natural) log of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The sum of the (natural) log of the values in the window or NaN if the statistics object is null.
* `STATS_SUM_SQUARES`

#### `STATS_SUM_SQUARES`
* Description: Calculates the sum of the squares of the accumulated values (or in the window if a window is used).
* Input:
* stats - The Stellar statistics object
* Returns: The sum of the squares of the values in the window or NaN if the statistics object is null.
* `STATS_VARIANCE`

#### `STATS_VARIANCE`
* Description: Calculates the variance of the accumulated values (or in the window if a window is used). See http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics
* Input:
* stats - The Stellar statistics object
* Returns: The variance of the values in the window or NaN if the statistics object is null.

### Statistical Outlier Detection

* `OUTLIER_MAD_STATE_MERGE`
### Statistical Outlier Detection
#### `OUTLIER_MAD_STATE_MERGE`
* Description: Update the statistical state required to compute the Median Absolute Deviation.
* Input:
* [state] - A list of Median Absolute Deviation States to merge. Generally these are states across time.
* currentState? - The current state (optional)
* Returns: The Median Absolute Deviation state
* `OUTLIER_MAD_ADD`

#### `OUTLIER_MAD_ADD`
* Description: Add a piece of data to the state.
* Input:
* state - The MAD state
* value - The numeric value to add
* Returns: The MAD state
* `OUTLIER_MAD_SCORE`

#### `OUTLIER_MAD_SCORE`
* Description: Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf
* Input:
* state - The MAD state
Expand Down
38 changes: 19 additions & 19 deletions metron-platform/metron-common/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,25 +86,25 @@ Using parens such as: "foo" : "\<ok\>" requires escaping; "foo": "\'\<ok\>\'"
| [ `REGEXP_MATCH`](#regexp_match)|
| [ `SPLIT`](#split)|
| [ `STARTS_WITH`](#starts_with)|
| [ `STATS_ADD`](#stats_add)|
| [ `STATS_COUNT`](#stats_count)|
| [ `STATS_GEOMETRIC_MEAN`](#stats_geometric_mean)|
| [ `STATS_INIT`](#stats_init)|
| [ `STATS_KURTOSIS`](#stats_kurtosis)|
| [ `STATS_MAX`](#stats_max)|
| [ `STATS_MEAN`](#stats_mean)|
| [ `STATS_MERGE`](#stats_merge)|
| [ `STATS_MIN`](#stats_min)|
| [ `STATS_PERCENTILE`](#stats_percentile)|
| [ `STATS_POPULATION_VARIANCE`](#stats_population_variance)|
| [ `STATS_QUADRATIC_MEAN`](#stats_quadriatic_mean)|
| [ `STATS_SD`](#stats_sd)|
| [ `STATS_SKEWNESS`](#stats_skewness)|
| [ `STATS_SUM`](#stats_sum)|
| [ `STATS_SUM_LOGS`](#stats_sum_logs)|
| [ `STATS_SUM_SQUARES`](#stats_sum_squares)|
| [ `STATS_VARIANCE`](#stats_variance)|
| [ `SYSTEM_ENV_GET`](#stats_env_get)|
| [ `STATS_ADD`](../../metron-analytics/metron-statistics#stats_add)|
| [ `STATS_COUNT`](../../metron-analytics/metron-statistics#stats_count)|
| [ `STATS_GEOMETRIC_MEAN`](../../metron-analytics/metron-statistics#stats_geometric_mean)|
| [ `STATS_INIT`](../../metron-analytics/metron-statistics#stats_init)|
| [ `STATS_KURTOSIS`](../../metron-analytics/metron-statistics#stats_kurtosis)|
| [ `STATS_MAX`](../../metron-analytics/metron-statistics#stats_max)|
| [ `STATS_MEAN`](../../metron-analytics/metron-statistics#stats_mean)|
| [ `STATS_MERGE`](../../metron-analytics/metron-statistics#stats_merge)|
| [ `STATS_MIN`](../../metron-analytics/metron-statistics#stats_min)|
| [ `STATS_PERCENTILE`](../../metron-analytics/metron-statistics#stats_percentile)|
| [ `STATS_POPULATION_VARIANCE`](../../metron-analytics/metron-statistics#stats_population_variance)|
| [ `STATS_QUADRATIC_MEAN`](../../metron-analytics/metron-statistics#stats_quadratic_mean)|
| [ `STATS_SD`](../../metron-analytics/metron-statistics#stats_sd)|
| [ `STATS_SKEWNESS`](../../metron-analytics/metron-statistics#stats_skewness)|
| [ `STATS_SUM`](../../metron-analytics/metron-statistics#stats_sum)|
| [ `STATS_SUM_LOGS`](../../metron-analytics/metron-statistics#stats_sum_logs)|
| [ `STATS_SUM_SQUARES`](../../metron-analytics/metron-statistics#stats_sum_squares)|
| [ `STATS_VARIANCE`](../../metron-analytics/metron-statistics#stats_variance)|
| [ `SYSTEM_ENV_GET`](#system_env_get)|
| [ `SYSTEM_PROPERTY_GET`](#system_property_get)|
| [ `TO_DOUBLE`](#to_double)|
| [ `TO_EPOCH_TIMESTAMP`](#to_epoch_timestamp)|
Expand Down