initial checkin

LICENSE

@@ -1,3 +1,4 @@
+
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
@@ -178,15 +179,15 @@
    APPENDIX: How to apply the Apache License to your work.
 
       To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "{}"
+      boilerplate notice, with the fields enclosed by brackets "[]"
       replaced with your own identifying information. (Don't include
       the brackets!)  The text should be enclosed in the appropriate
       comment syntax for the file format. We also recommend that a
       file or class name and description of purpose be included on the
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright {yyyy} {name of copyright owner}
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
@@ -199,4 +200,3 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-

MANIFEST.in

@@ -0,0 +1 @@
+global-include VERSION requirements.txt

NOTICE

@@ -0,0 +1,52 @@
+Copyright 2013 LinkedIn Corp. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+List of other open-source software used/depended on by Naarad:
+
+PrintGCStats
+https://java.net/projects/printgcstats/
+Copyright 2013 Oracle
+License: BSD
+
+dygraphs
+http://dygraphs.com/
+Copyright 2011 Dan Vanderkam
+License: MIT
+
+numpy
+http://www.numpy.org/
+Copyright 2005-2013 NumPy Developers
+License: BSD
+
+matplotlib
+http://matplotlib.org/
+Copyright 2012-2013 Matplotlib Development Team
+License: PSF (http://matplotlib.org/users/license.html)
+
+pytz
+http://pytz.sourceforge.net/
+Copyright 2008-2013 Stuart Bishop
+License: MIT
+
+sorttable.js
+http://www.kryogenix.org/code/browser/sorttable/#licence
+License: X11 License
+
+pygal
+http://pygal.org
+License: GNU LGPL v3+ (https://www.gnu.org/licenses/lgpl.html)
+
+bootstrap
+http://getbootstrap.com/
+License: Apache License v2.0 (http://www.apache.org/licenses/LICENSE-2.0)

README.md

@@ -0,0 +1,204 @@
+# luminol #
+
+### Overview
+Luminol is a light weight python library for time series data analysis. The two major functionalities it supports are anomaly detection and correlation. It can be used to investigate possible causes of anomaly. You collect time series data and Luminol can:
+* Given a time series, detect if the data contains any anomaly and gives you back a time window where the anomaly happened in, a time stamp where the anomaly reaches its severity, and a score indicating how severe is the anomaly compare to others in the time series.
+* Given two time series, help find their correlation coefficient. Since the correlation mechanism allows a shift room, you are able to correlate two peaks that are slightly apart in time.
+
+Luminol is configurable in a sense that you can choose which specific algorithm you want to use for anomaly detection or correlation. In addition, the library does not rely on any predefined threshold on the values of a time series. Instead, it assigns each data point an anomaly score and identifies anomalies using the scores.  
+
+By using the library, we can establish a logic flow for root cause analysis. For example, suppose there is a spike in network latency:
+* Anomaly detection discovers the spike in network latency time series
+* Get the anomaly period of the spike, and correlate with other system metrics(GC, IO, CPU, etc.) in the same time range
+* Get a ranked list of correlated metrics, and the root cause candidates are likely to be on the top.
+
+Investigating the possible ways to automate root cause analysis is one of the main reasons we developed this library and it will be a fundamental part of the future work.
+
+***
+
+### Installation
+make sure you have python, pip, numpy, and install directly through pip:
+```
+pip install luminol
+```
+the most up-to-date version of the library is 0.1.
+
+***
+
+### Quick Start
+This is a quick start guide for using luminol for time series analysis.
+
+1. import the library
+   ```python
+   import luminol
+   ```
+
+2. conduct anomaly detection on a single time series ts.
+   ```python
+   detector = luminol.anomaly_detector.AnomalyDetector(ts)
+   anomalies = detector.get_anomalies()
+   ```
+
+3. if there is anomaly, correlate the first anomaly period with a secondary time series ts2.
+   ```python
+   if anomalies:
+     time_period = anomalies[0].get_time_window()
+     correlator = luminol.correlator.Correlator(ts, ts2, time_period)
+   ```
+
+4. print the correlation coefficient
+   ```python
+   print correlator.get_correlation_result().coefficient
+   ```
+
+These are really simple use of luminol. For information of the parameter types, return types and optional parameters, please refer to the API.
+
+***
+
+### Modules
+Modules in Luminol refers to customized classes developed for better data representation, which are `Anomaly`, `CorrelationResult` and `TimeSeries`.
+####Anomaly
+_class_ luminol.modules.anomaly.**Anomaly**
+<br/> It contains these attributes:
+```python
+self.start_timestamp: # epoch seconds represents the start of the anomaly period.
+self.end_timestamp: # epoch seconds represents the end of the anomaly period.
+self.anomaly_score: # a score indicating how severe is this anomaly.
+self.exact_timestamp: # epoch seconds indicates when the anomaly reaches its severity.
+```
+It has these public methods:
+* `get_time_window()`: returns a tuple (start_timestamp, end_timestamp).
+
+####CorrelationResult
+_class_ luminol.modules.correlation_result.**CorrelationResult**
+<br/> It contains these attributes:
+```python
+self.coefficient: # correlation coefficient.
+self.shift: # the amount of shift needed to get the above coefficient.
+self.shifted_coefficient: # a correlation coefficient with shift taken into account.
+```
+
+####TimeSeries
+_class_ luminol.modules.time_series.**TimeSeries**
+```python
+__init__(self, series)
+```
+* `series(dict)`: timestamp -> value
+
+It has a various handy methods for manipulating time series, including generator `iterkeys`, `itervalues`, and `iteritems`. It also supports binary operations such as add and subtract. Please refer to the [code](https://github.com/linkedin/naarad/blob/master/lib/luminol/src/luminol/modules/time_series.py) and inline comments for more information.
+
+***
+
+### API
+The library contains two classes: `AnomalyDetector` and `Correlator`, and there are two sets of APIs, one corresponding to each class. There are also customized modules for better data representation. The [Modules](#modules) section in this documentation may provide useful information as you walk through the APIs.
+####AnomalyDetector
+_class_ luminol.anomaly_detector.**AnomalyDetecor**
+```python
+__init__(self, time_series, baseline_time_series=None, score_only=False, score_threshold=None,
+score_percentile_threshold=None, algorithm_name=None, algorithm_params=None, 
+refine_algorithm_name=None, refine_algorithm_params=None)
+```
+*  `time_series`: The metric you want to conduct anomaly detection on. It can have the following three types:
+
+   ```python
+   1. string: # path to a csv file
+   2. dict: # timestamp -> value
+   3. lumnol.modules.time_series.TimeSeries
+   ```
+* `baseline_time_series`: an optional baseline time series of one the types mentioned above.
+* `score only(bool)`: if asserted, anomaly scores for the time series will be available, while anomaly periods will not be identified.
+* `score_threshold`: if passed, anomaly scores above this value will be identified as anomaly. It can overrides score_percentile_threshold.
+* `score_precentile_threshold`: if passed, anomaly scores above this percentile will be identified as anomaly. It can not overrides score_threshold.
+* `algorithm_name(string)`: if passed, the specific algorithm will be used to compute anomaly scores.
+* `algorithm_params(dict)`: additional parameters for algorithm specified by algorithm_name.
+* `refine_algorithm_name(string)`: if passed, the specific algorithm will be used to compute the time stamp of severity within each anomaly period.
+* `refine_algorithm_params(dict)`: additional parameters for algorithm specified by refine_algorithm_params.
+
+Available algorithms and their additional parameters are:
+
+  ```python
+  1. 'bitmap_detector': # behaves well for huge data sets, and it is the default detector.
+     {
+        'precision'(4): # how many sections to categorize values,
+        'lag_window_size'(2% of the series length): # lagging window size,
+        'future_window_size'(2% of the series length): # future window size,
+        'chunk_size'(2): # chunk size.
+     }
+  2. 'default_detector': # used when other algorithms fails, not meant to be explicitly used.
+  3. 'derivative_detector': # meant to be used when abrupt changes of value are of main interest.
+     {
+        'smoothing factor'(0.2): # smoothing factor used to compute exponential moving averages
+                                  # of derivatives.
+     }
+  4. 'exp_avg_detector': # meant to be used when values are in a roughly stationary range.
+                          # and it is the default refine algorithm.
+     {
+        'smoothing factor'(0.2): # smoothing factor used to compute exponential moving averages.
+        'lag_window_size'(20% of the series length): # lagging window size.
+        'use_lag_window'(False): # if asserted, a lagging window of size lag_window_size will be used.
+     }
+  ```
+
+It may seem vague for the meanings of some parameters above. Here are some useful insights:
+* [Bitmap](http://alumni.cs.ucr.edu/~ratana/SSDBM05.pdf)
+* [Exponential Moving Avg](http://en.wikipedia.org/wiki/Exponential_smoothing)
+
+The **AnomalyDetector** class has the following public methods:
+* `get_all_scores()`: returns a anomaly score time series of type [TimeSeries](#modules).
+* `get_anomalies()`: return a list of [Anomaly](#modules) objects.
+
+####Correlator
+_class_ luminol.correlator.**Correlator**
+```python
+__init__(self, time_series_a, time_series_b, time_period=None, use_anomaly_score=False,
+algorithm_name=None, algorithm_params=None)
+```
+* `time_series_a`: a time series, for its type, please refer to time_series for AnomalyDetector above.
+* `time_series_b`: a time series, for its type, please refer to time_series for AnomalyDetector above.
+* `time_period(tuple)`: a time period where to correlate the two time series.
+* `use_anomaly_score(bool)`: if asserted, the anomaly scores of the time series will be used to compute correlation coefficient instead of the original data in the time series.
+* `algorithm_name`: if passed, the specific algorithm will be used to calculate correlation coefficient.
+* `algorithm_params`: any additional parameters for the algorithm specified by algorithm_name.
+
+Available algorithms and their additional parameters are:
+
+   ```python
+   1. 'cross_correlator': # when correlate two time series, it tries to shift the series around so that it
+                           # can catch spikes that are slightly apart in time.
+      {
+         'max_shift_seconds'(60): # maximal allowed shift room in seconds,
+         'shift_impact'(0.05): # weight of shift in the shifted coefficient.
+      }
+   ```
+
+The **Correlator** class has the following public methods:
+* `get_correlation_result()`: return a [CorrelationResult](#modules) object.
+* `is_correlated(threshold=0.7)`: if coefficient above the passed in threshold, return a [CorrelationResult](#modules) object. Otherwise, return false.
+
+### Example
+1. Put anomaly scores in a list.
+
+   ```python
+   from luminol.anomaly_detector import AnomalyDetector
+   my_detector = AnomalyDetector(ts)
+   score = my_detector.get_all_scores()
+   anom_score = list()
+   for (timestamp, value) in score.iteritems():
+     t_str = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(timestamp))
+     anom_score.append([t_str, value])
+   ```
+
+2. Correlate with ts2 on every anomaly.
+
+   ```python
+   from luminol.anomaly_detector import AnomalyDetector
+   from luminol.correlator import Correlator
+
+   my_detector = detector.AnomalyDetector(ts)
+   anomalies = my_detector.get_anomalies()
+   for a in anomalies:
+     time_period = a.get_time_window()
+     my_correlator = Correlator(ts, ts2, time_period)
+     if my_correlator.is_correlated(threshold=0.8):
+       print "ts2 correlate with ts at time period (%d, %d)" % time_period 
+   ```

VERSION

@@ -0,0 +1 @@
+0.2

demo/requirements.txt

@@ -0,0 +1 @@
+flask

demo/src/rca.py

@@ -0,0 +1,92 @@
+# coding=utf-8
+"""
+© 2014 LinkedIn Corp. All rights reserved.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+"""
+
+from collections import defaultdict
+import os
+import sys
+
+from luminol import utils, exceptions
+from luminol.anomaly_detector import AnomalyDetector
+from luminol.correlator import Correlator
+from luminol.modules.correlation_result import CorrelationResult
+from luminol.modules.time_series import TimeSeries
+
+
+class RCA(object):
+  def __init__(self, metrix, related_metrices):
+    """
+    Initializer
+    :param metrix: a TimeSeries, a dictionary or a path to a csv file(str)
+    :param list related_metrixes: a list of time series.
+    """
+    self.metrix = self._load(metrix)
+    self.anomaly_detector = AnomalyDetector(metrix)
+    self.related_metrices = related_metrices
+    self.anomalies = self.anomaly_detector.get_anomalies()
+    self._analyze()
+
+  def _load(self, metrix):
+    """
+    Load time series.
+    :param timeseries: a TimeSeries, a dictionary or a path to a csv file(str).
+    :return TimeSeries: a TimeSeries object.
+    """
+    if isinstance(metrix, TimeSeries):
+      return metrix
+    if isinstance(metrix, dict):
+      return TimeSeries(metrix)
+    return TimeSeries(utils.read_csv(metrix))
+
+  def _analyze(self):
+    """
+    Analyzes if a matrix has anomalies.
+    If any anomaly is found, determine if the matrix correlates with any other matrixes.
+    To be implemented.
+    """
+    output = defaultdict(list)
+    output_by_name = defaultdict(list)
+    scores = self.anomaly_detector.get_all_scores()
+
+    if self.anomalies:
+      for anomaly in self.anomalies:
+        metrix_scores = scores
+        start_t, end_t = anomaly.get_time_window()
+        t = anomaly.exact_timestamp
+
+        # Compute extended start timestamp and extended end timestamp.
+        room = (end_t - start_t) / 2
+        if not room:
+          room = 30
+        extended_start_t = start_t - room
+        extended_end_t = end_t + room
+        metrix_scores_cropped = metrix_scores.crop(extended_start_t, extended_end_t)
+
+        # Adjust the two timestamps if not enough data points are included.
+        while len(metrix_scores_cropped) < 2:
+          extended_start_t = extended_start_t - room
+          extended_end_t = extended_end_t + room
+          metrix_scores_cropped = metrix_scores.crop(extended_start_t, extended_end_t)
+
+        # Correlate with other metrics
+        for entry in self.related_metrices:
+          try:
+            entry_correlation_result = Correlator(self.metrix, entry, time_period=(extended_start_t, extended_end_t),
+                                                  use_anomaly_score=True).get_correlation_result()
+            record = extended_start_t, extended_end_t, entry_correlation_result.__dict__, entry
+            record_by_name = extended_start_t, extended_end_t, entry_correlation_result.__dict__
+            output[t].append(record)
+            output_by_name[entry].append(record_by_name)
+          except exceptions.NotEnoughDataPoints:
+            pass
+
+    self.output = output
+    self.output_by_name = output_by_name