Merge branch 'master' of https://github.com/apache/falcon

common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java

@@ -35,7 +35,7 @@
 
 //SUSPEND CHECKSTYLE CHECK LineLengthCheck
 /**
- * Feed SLA monitoring.
+ * Entity SLA monitoring.
  * */
 @Entity
 @NamedQueries({

docs/src/site/twiki/EntitySLAAlerting.twiki

@@ -0,0 +1,57 @@
+---++Entity SLA Alerting
+
+Falcon supports SLA in feed and process.
+
+Types of SLA supported for feed:
+
+   1.slaLow
+   1.slaHigh
+
+To know more about feedSla look at [[EntitySpecification][Feed Specification]]
+
+Types of SLA supported for process:
+
+   1.shouldStartIn
+   1.shouldEndIn
+
+To know more about processSla look at [[EntitySpecification][Process Specification]]
+
+Falcon Entity Alerting service do the following things:
+
+   1.Monitor instances of feed and process and send notifications to all the listeners attached to it.
+   1.In case of feed it notifies when an *slaHigh* miss happens. slaLow is not supported.
+   1.In case of process it notifies when an SLA miss for *shouldEndIn* happens. shouldStartIn is not supported.
+
+Entity SLA Alert service depends upon [[EntitySLAMonitoring][Falcon Entity SLA Monitoring]] to know which process and feed instances are to be monitored.
+
+*How to attach listeners:*
+
+You can write custom listeners to do some action whenever a process or feed instance misses its SLA.
+To attach listeners please add below property in startup.properties:
+
+<verbatim>
+
+*.entityAlert.listeners=org.apache.customPath.customListener
+
+</verbatim>
+
+Currently Falcon natively supports [[BacklogMetricEmitterService][Back Log Emitter Service]] as a listener to EntitySLAAlert service
+
+---++Dependencies:
+
+*Other Services:*
+
+To enable Enity SLA Alerting service you need to enable [[EntitySLAMonitoring][Falcon Entity SLA Monitoring]]
+
+ Following properties are needed in startup.properties:
+
+<verbatim>
+
+*.application.services=org.apache.falcon.service.EntitySLAAlertService
+
+*.entity.sla.statusCheck.frequency.seconds=600
+</verbatim>
+
+*Falcon Database:*
+
+Entity SLA Alerting service maintains its state in the database.It needs one table *ENTITY_SLA_ALERTS* please have a look at [[FalconDatabase]] to know how to create it.

docs/src/site/twiki/EntitySLAMonitoring.twiki

@@ -0,0 +1,25 @@
+---++Falcon Entity SLA Monitoring
+
+Entity SLA monitoring allows you to monitor the entity (process and feed) .It keeps track of the instances of the entity that are running and stores them in the db.
+
+
+---++Dependencies:
+
+*Other Services:*
+
+Entity SLA monitoring service requires FalconJPAService to be up. Following are the values you need to set to run EntitySLAMonitoring.
+In startup.properties:
+
+<verbatim>
+*.application.services= org.apache.falcon.state.store.service.FalconJPAService,
+                        org.apache.falcon.service.EntitySLAMonitoringService
+</verbatim>
+
+
+*Falcon Database:*
+
+Entity SLA monitoring service maintains its state in the database.It needs two tables:
+
+   1.MONITORED_ENTITY
+   1.PENDING_INSTANCES
+please have a look at [[FalconDatabase]] to know how to create them.

docs/src/site/twiki/FalconDatabase.twiki

@@ -0,0 +1,74 @@
+---+++ Configuring the state store for Falcon
+You can configure statestore by making changes to __$FALCON_HOME/conf/statestore.properties__ as follows. You will need to restart Falcon Server for the changes to take effect.
+
+Falcon Server needs to maintain state of the entities and instances in a persistent store for the system to be recoverable. Since Prism only federates, it does not need to maintain any state information. Following properties need to be set in statestore.properties of Falcon Servers:
+<verbatim>
+######### StateStore Properties #####
+*.falcon.state.store.impl=org.apache.falcon.state.store.jdbc.JDBCStateStore
+*.falcon.statestore.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
+*.falcon.statestore.jdbc.url=jdbc:derby:data/falcon.db
+# StateStore credentials file where username,password and other properties can be stored securely.
+# Set this credentials file permission 400 ;the user who starts falcon should only have read permission.
+# Give Absolute path to credentials file along with file name or put in classpath with file name statestore.credentials.
+# Credentials file should be present either in given location or class path, otherwise falcon won't start.
+*.falcon.statestore.credentials.file=
+*.falcon.statestore.jdbc.username=sa
+*.falcon.statestore.jdbc.password=
+*.falcon.statestore.connection.data.source=org.apache.commons.dbcp.BasicDataSource
+# Maximum number of active connections that can be allocated from this pool at the same time.
+*.falcon.statestore.pool.max.active.conn=10
+*.falcon.statestore.connection.properties=
+# Indicates the interval (in milliseconds) between eviction runs.
+*.falcon.statestore.validate.db.connection.eviction.interval=300000
+## The number of objects to examine during each run of the idle object evictor thread.
+*.falcon.statestore.validate.db.connection.eviction.num=10
+## Creates Falcon DB.
+## If set to true, Falcon creates the DB schema if it does not exist. If the DB schema exists is a NOP.
+## If set to false, Falcon does not create the DB schema. If the DB schema does not exist it fails start up.
+*.falcon.statestore.create.db.schema=true
+</verbatim>
+
+The _*.falcon.statestore.jdbc.url_ property in statestore.properties determines the DB and data location. All other properties are common across RDBMS.
+
+*NOTE : Although multiple Falcon Servers can share a DB (not applicable for Derby DB), it is recommended that you have different DBs for different Falcon Servers for better performance.*
+
+You will need to create the state DB and tables before starting the Falcon Server. To create tables, a tool comes bundled with the Falcon installation. You can use the _falcon-db.sh_ script to create tables in the DB. The script needs to be run only for Falcon Servers and can be run by any user that has execute permission on the script. The script picks up the DB connection details from __$FALCON_HOME/conf/statestore.properties__. Ensure that you have granted the right privileges to the user mentioned in statestore.properties_, so the tables can be created.
+
+You can use the help command to get details on the sub-commands supported:
+<verbatim>
+./bin/falcon-db.sh help
+usage:
+      Falcon DB initialization tool currently supports Derby DB/ Mysql
+
+      falcondb help : Display usage for all commands or specified command
+
+      falcondb version : Show Falcon DB version information
+
+      falcondb create <OPTIONS> : Create Falcon DB schema
+                      -run             Confirmation option regarding DB schema creation/upgrade
+                      -sqlfile <arg>   Generate SQL script instead of creating/upgrading the DB
+                                       schema
+
+      falcondb upgrade <OPTIONS> : Upgrade Falcon DB schema
+                       -run             Confirmation option regarding DB schema creation/upgrade
+                       -sqlfile <arg>   Generate SQL script instead of creating/upgrading the DB
+                                        schema
+
+</verbatim>
+Currently, MySQL, postgreSQL and Derby are supported as state stores. We may extend support to other DBs in the future. Falcon has been tested against MySQL v5.5 and PostgreSQL v9.5. If you are using MySQL ensure you also copy mysql-connector-java-<version>.jar under __$FALCON_HOME/server/webapp/falcon/WEB-INF/lib__ and __$FALCON_HOME/client/lib__
+
+---++++ Using Derby as the State Store
+Using Derby is ideal for QA and staging setup. Falcon comes bundled with a Derby connector and no explicit setup is required (although you can set it up) in terms creating the DB or tables.
+For example,
+ <verbatim> *.falcon.statestore.jdbc.url=jdbc:derby:data/falcon.db;create=true </verbatim>
+
+ tells Falcon to use the Derby JDBC connector, with data directory, $FALCON_HOME/data/ and DB name 'falcon'. If _create=true_ is specified, you will not need to create a DB up front; a database will be created if it does not exist.
+
+---++++ Using MySQL as the State Store
+The jdbc.url property in statestore.properties determines the DB and data location.
+For example,
+ <verbatim> *.falcon.statestore.jdbc.url=jdbc:mysql://localhost:3306/falcon </verbatim>
+
+ tells Falcon to use the MySQL JDBC connector, which is accessible @localhost:3306, with DB name 'falcon'.
+
+ Note: First time we have to manually create the schema in production as we have set falcon.statestore.create.db.schema = false

docs/src/site/twiki/FalconDocumentation.twiki

@@ -823,7 +823,7 @@ Extensions is detailed in [[Extensions][Extensions]].
 
 ---++ Monitoring
 
-Monitoring and Operationalizing Falcon is detailed in [[Operability][Operability]] and [[FeedSLAMonitoring][Falcon Feed SLA Monitoring]].
+Monitoring and Operationalizing Falcon is detailed in [[Operability][Operability]] and [[EntitySLAMonitoring][Falcon Entity SLA Monitoring]].
 
 ---++ Email Notification
 Notification for instance completion in Falcon is defined in [[FalconEmailNotification][Falcon Email Notification]].

docs/src/site/twiki/GraphiteMetricCollection.twiki

@@ -8,15 +8,17 @@ Graphite Metric Collection currently allows to collect the following metrics at
 
 To send data to graphite we need to intialize metricNotificationService in startup.properties:
 
+<verbatim>
 *.application.services= org.apache.falcon.metrics.MetricNotificationService,
-
+</verbatim>
 
 Add following properties for graphiteNotificationPlugin :
 
-Graphite properties
+*Graphite properties*
+<verbatim>
    * *.falcon.graphite.hostname=localhost
    * *.falcon.graphite.port=2003
    * *.falcon.graphite.frequency=1
    * *.falcon.graphite.prefix=falcon
-
+</verbatim>
 The falcon.graphite.frequency is in seconds and all the time that is being sent to graphite is in seconds.

docs/src/site/twiki/Operability.twiki

@@ -207,13 +207,16 @@ public class FalconUserJMSClient {
 
 ---++ Alerts
 
-Falcon generates alerts for unrecoverable errors into a log file by default.
+Falcon generates two type of alerts:
+
+1. By default it logs unrecoverable errors into a log file
 Users can view these alerts in the alerts.log file, by default this file is created
 under ${user.dir}/logs/ directory.
 
 Users may also extend the Falcon Alerting plugin to send events to systems like Nagios, etc. by
 extending org.apache.falcon.plugin.AlertingPlugin interface.
 
+2. Alerts on *SLA misses* for feeds and process is detailed in [[EntitySLAAlerting][Entity SLA Alerting]].
 
 ---++ Audits
 

prism/src/main/java/org/apache/falcon/service/EntitySLAAlertService.java

@@ -148,8 +148,8 @@ void processSLACandidates(){
                     }
                     store.updateSLAAlertInstance(entityName, clusterName, nominalTime, entityType);
                     LOG.info("Entity :{} EntityType : {} Cluster: {} Nominal Time: {} missed SLAHigh", entityName,
-                            entityType , clusterName , nominalTime);
-                    highSLAMissed(entityName, clusterName, EntityType.valueOf(entityType), nominalTime);
+                            entityType, clusterName , nominalTime);
+                    highSLAMissed(entityName, clusterName, EntityType.getEnum(entityType), nominalTime);
                 }
             }
         } catch (FalconException e){