Copy project from internal repository

.gitignore

@@ -0,0 +1 @@
+/.project

README.md

@@ -0,0 +1,91 @@
+# Spark Streaming job for monitoring metrics
+
+This Spark Streaming job is desinged for monitoring metrics. 
+
+[User's manual](doc/users-manual.md)
+
+### Key feautres
+
+- Several monitors can be declared, each monitor can have a metric filter, a metric pre-analysis, a metric analysis and notificators. 
+- Only a metric source, analysis result sink and notifications sink can be declared. They are shared by all monitors. 
+- Components: metrics source, pre-analysis, analysis, analysis results sink, notificator and notification sink. They can be replaced. 
+- Some built-in components: Kafka source, different pre-analysis and analyis, Elastic sink, notificators, ...
+- Metrics at different frequencies.
+- Monitors configuration can be updated while running.
+- Detection of missing metrics.
+
+## Monitors
+
+Metrics are consumed by a source and sent to all monitors.
+
+Many monitors can be declared. Each monitor has a filter to determine to which metrics it should be applied.
+Filtered metrics are pre-analyzed if a pre-analysis is configured. Once pre-analyzed, an analysis is applied to determine the current status of the metric.
+Several notificators can be configured to produce notifications.
+
+Results from analysis and notifications can be sinked to an external storage.
+
+## Components
+
+They are considered parts of the processing pipeline that can be easily replaced by other built-in components or by an externally developed component.
+
+If you are willing to develop any component, look at the [developers guide](doc/developers-guide.md).
+
+### Metric source
+
+This componenet is ment to consume metrics from a source and generate an stream of metrics. 
+
+Only one source is declared for the job. All monitors consume from this source.
+
+Built-in metric sources:
+- Kafka.
+
+### Metric pre-analysis
+
+This componenet is ment to transform incoming metrics before the analysis. If a pre-analysis is applied, the produced value will be the value used by the analysis.
+
+Each monitor configures or not its own pre-analysis.
+
+Built-in metric pre-analysis:
+- Average value during certain period.
+- Weighted average value during certain period. The closer in time the metri is to current time, the more influence is has over analyzed value.
+- Difference: difference with previous value.
+
+### Metric analysis
+
+This component is ment to determine the status (error, warning, exception, ok) of each of the incoming metrics (pre-analyzed or not).
+
+Each monitor configures or not its own analysis.
+
+Built-in metric analysis:
+- Fixed threshold: error and warning thresholds.
+- Recent activity: error and warning thresholds are computed using average and variance from recent activity.
+- Percentile: error and warning thresholds are computed based on percentiles from recent activity.
+- Seasonal: a season is configured (hour, day or week), using a learning coeficient, average and variance are computed along the season, these two values are used to calculate error and warning thresholds.  
+
+### Analysis results sink
+
+Analysis produce results for each of the incoming metrics. These results can be sinked to an external storage for watching the metric and analysis results.
+
+Only one analysis results sink is declared for the job. All monitors use this sink.
+
+Built-in analysis results sink:
+- Elastic.
+
+### Notificator
+
+A notificator determine when to raise a notifications based on analysis results.
+
+Several notificators can be configured in a monitor.
+
+Built-in notificators:
+- Constant status: if a metric has been in configured statuses during a certain period.
+- Percentage status: if a metric has been in configured statuses during a percentage of a certain period.
+
+### Notifications sink
+
+Notifications produced by notificators are sink using this componenet. Notifications can be sinked to an external storage, sent by email, used to trigger actions, etc.
+
+Only one notifications sink is declared for the job. All monitors use this sink.
+
+Built-in notifications sink:
+- Elastic.

bin/run

@@ -0,0 +1,60 @@
+#!/bin/bash
+
+# Copyright (C) 2016, CERN
+# This software is distributed under the terms of the GNU General Public
+# Licence version 3 (GPL Version 3), copied verbatim in the file "LICENSE".
+# In applying this license, CERN does not waive the privileges and immunities
+# granted to it by virtue of its status as Intergovernmental Organization
+# or submit itself to any jurisdiction.
+
+# Reference: http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "$SOURCE" ]; do
+  DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
+  SOURCE="$(readlink "$SOURCE")"
+  [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
+done
+SCRIPT_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
+
+HOME=$SCRIPT_DIR/..
+
+CONFIG_PATH=$HOME/conf/config.properties
+CONFIG_PATH=/tmp/config.properties
+
+$HOME/bin/update_config
+
+export JAVA_HOME=/root/jdk1.8.0_112/
+export SPARK_HOME=/root/spark-2.1.0-bin-hadoop2.7/
+
+export HADOOP_CONF_DIR=$HOME/conf/
+export SPARK_CONF_DIR=$HOME/conf/
+
+[ -f $HOME/elasticsearch-hadoop-5.2.2.jar ] || \
+    wget http://central.maven.org/maven2/org/elasticsearch/elasticsearch-hadoop/5.2.2/elasticsearch-hadoop-5.2.2.jar -O $HOME/elasticsearch-hadoop-5.2.2.jar
+
+# Stop if runnning
+kill `ps -ax | grep "ch.cern.spark.metrics.Driver" | grep -v grep | awk '{$1=$1};1' | cut -d' ' -f 1`
+echo `date` Restarting... >> $HOME/log/restarts
+
+# STart
+$SPARK_HOME/bin/spark-submit \
+	--master yarn \
+	--driver-cores 2 \
+	--driver-memory 2G \
+	--executor-memory 3G \
+	--num-executors 5 \
+	--executor-cores 1 \
+	--class ch.cern.spark.metrics.Driver \
+	--packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0,org.apache.spark:spark-streaming-flume_2.11:2.1.0 \
+	--exclude-packages org.apache.flume:flume-ng-auth \
+	--jars $HOME/elasticsearch-hadoop-5.2.2.jar \
+	--repositories https://repository.cloudera.com/artifactory/cloudera-repos/ \
+	--conf spark.yarn.am.attemptFailuresValidityInterval=1h \
+	--conf spark.yarn.max.executor.failures=48 \
+	--conf spark.yarn.executor.failuresValidityInterval=1h \
+	--conf spark.task.maxFailures=8 \
+	--conf spark.hadoop.fs.hdfs.impl.disable.cache=true \
+	--principal dblogs@CERN.CH \
+	--keytab $HOME/conf/principal.keytab \
+	$HOME/target/spark-metrics-*.jar \
+	$CONFIG_PATH

bin/update_config

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# Copyright (C) 2016, CERN
+# This software is distributed under the terms of the GNU General Public
+# Licence version 3 (GPL Version 3), copied verbatim in the file "LICENSE".
+# In applying this license, CERN does not waive the privileges and immunities
+# granted to it by virtue of its status as Intergovernmental Organization
+# or submit itself to any jurisdiction.
+
+# Enviorenment variables for configuration
+# 
+# FORCE_JAVAC = true if you do not want to use Maven (default Maven is used)
+# JAVA_BIN_PATH = path to java bin (default: /usr/bin/)
+
+# Reference: http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
+SOURCE="${BASH_SOURCE[0]}"
+while [ -h "$SOURCE" ]; do
+  DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
+  SOURCE="$(readlink "$SOURCE")"
+  [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
+done
+SCRIPT_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
+
+HOME=$SCRIPT_DIR/..
+
+CONFIG_PATH=sparkStreaming/config.properties
+
+hdfs dfs -put -f $HOME/conf/config.properties $CONFIG_PATH

doc/developers-guide.md

@@ -0,0 +1,63 @@
+# Spark Streaming job for monitoring metrics: developers guide
+
+## Components
+
+External components can be developed by extending the corresponging classes (look at component type below for details).
+
+Internal components should use the two parameters constructor (super(Class, String)) specifying own class and type (for configuring). 
+They also need to be registered at ch.cern.spark.ComponentManager. 
+External components can use the no-parametrized constructor and they do not need to be registered.
+
+Any component can override the config() method. Properties parameter will contain only the corresponding configuration for the component.
+
+### Store for stateful components
+
+Pre-analysis, analysis and notificators may need to keep some historical data. If so, implemented component can implement the interface ch.cern.spark.metrics.store.HasStore.
+
+The save() method must return an object which implements the interface Store and contains only the data that needs to be stored.
+
+The load() method receives the previous saved store.
+
+Data contained in Store will be serialized and may thousands of these are stored. Taking into account that, the Store should contain as less data as possible.
+
+### Metric source
+
+This component is ment to consume metrics from a source and generate an stream of metrics. 
+
+Externally developed sources must extend ch.cern.spark.metrics.source.MetricsSource.
+
+### Metric pre-analysis
+
+This component is ment to transform incoming metrics before the analysis. If a pre-analysis is applied, the produced value will be the value used by the analysis.
+
+Externally developed pre-analysis must extend ch.cern.spark.metrics.preanalysis.PreAnalysis.
+
+If same data need to be kept, this component can make use of an [Store](#store-for-stateful-components). 
+
+### Metric analysis
+
+This component is ment to determine the status (error, warning, exception, ok) of each of the incoming metrics (pre-analyzed or not).  
+
+Externally developed analysis must extend ch.cern.spark.metrics.analysis.Analysis.
+
+If same data need to be kept, this component can make use of an [Store](#store-for-stateful-components).
+
+### Analysis results sink
+
+This component produce results for each of the incoming metrics. These results can be sinked to an external storage for watching the metric and analysis results.
+
+Externally developed analysis results sinks must extend ch.cern.spark.metrics.results.sink.AnalysisResultsSink.
+
+### Notificator
+
+This component determines when to raise a notifications based on analysis results.
+
+Externally developed notificators must extend ch.cern.spark.metrics.notificator.Notificator.
+
+If same data need to be kept, this component can make use of an [Store](#store-for-stateful-components).
+
+### Notifications sink
+
+Notifications produced by notificators are sink using this component. Notifications can be sinked to an external storage, sent by email, used to trigger actions, etc. 
+
+Externally developed analysis results sinks must extend ch.cern.spark.metrics.results.sink.AnalysisResultsSink.