Hawkular Alerting Demo

This repository contains resources to prepare a standalone demo for Hawkular Alerting.
This demo will build a standalone distribution of Hawkular Alerting.

It will also download and install a secondary Wildfly Server that will be monitored
sending data and logs into Hawkular Alerting for evaluation.

Note	This demo has been tested on Linux platforms. Bash scripts can be easily translated into other platforms.

Step 0: Prepare Hawkular Alerting

On this step we will build and run a Hawkular Alerting standalone distribution:

Build a Hawkular Alerting standalone distribution:

    git clone https://github.com/hawkular/hawkular-alerts.git
    cd hawkular-alerts
    mvn clean install -DskipTests
    cd hawkular-alerts-rest-tests/target/wildfly-10.0.0.Final/
    bin/standalone.sh

Tip

Test Email server By default, Hawkular Alerting will send email notifications using a SMTP server on localhost:25, for demo purposes a test smtp server can be used to validate the reception of the emails. Hawkular Alerting has been tested using FakeSMTP. A GUI SMTP server can be set up with these steps: git clone https://github.com/Nilhcem/FakeSMTP cd FakeSMTP mvn clean install -DskipTests cd target sudo java -jar fakeSMTP-*.jar We use sudo java -jar fakeSMTP-*.jar to set the SMTP into the 25 port which is protected. If the JDK is only defined locally but not per all the users you might get an error related the GUI can not be correctly started. Try to set up your jdk correctly using alternatives --config java and selecting your jdk for all users.

Step 1: Install and start a monitored Wildfly Server

Now we are going to prepare a secondary Wildfly Server that will be monitored
during this demo.
This Wildfly Server will be started with port-offset=150.
In a new bash session:

    git clone https://github.com/lucasponce/hawkular-alerts-demo.git
    cd hawkular-alerts-demo
    ./01_install.sh
    ./02_start-wildfly.sh

Step 2: Import trigger definitions

In a new bash session, import the definitions using the script:

    ./03_import-definitions.sh

The definitions are structured on the following files.

import-actions.json

List of actions defined.

Actions can be shared across several triggers definitions.

import-availability.json

It defines one trigger to detect when the Wildfly Server is down.
This trigger uses AUTORESOLVE feature.
This functionality uses a FIRING condition to detect when the server is down and fire a CRITICAL Alert.
But this trigger also defines an AUTORESOLVE condition to detect with the server is up again.

Then Alerts opened will be automatically resolved once the Wildfly Server is up again.

import-deployments.json

Two triggers are defined to detect when new applications are deployed or undeployed into the Wildfly Server.
These triggers do not generate Alerts but Events.
This means that there is not lifecycle or pending tasks to resolve on Events.
Events can be also notified using the actions defined on import-actions.json.

import-applications.json

Two more triggers are defined to detect WARN and ERROR log messages of an specific demo application.
The trigger that detects WARN messages has defined a Dampening rule to control when the alert will be generated.
On this case, it will fire when there are at least 3 WARN messages on a sequence of 10 log messages.

On the contrary, the trigger that detects ERROR message will fire on a single ERROR message.
These triggers do not AUTORESOLVE the issue, so this means that these applications alerts need human intervention to be resolved.

Step 3: Start a process agent

One we have our definitions stored in Hawkular Alerting we need to feed data and events to start evaluating.
On regular scenarios this task is performed by specific agents.
For this demo, we have simulated very simple scripts that monitors the Wildfly Server.

This first script monitors when the Wildfly Server process is present and it sends and availability data via an endpoint.
In normal scenarios, the monitoring data storage is handled by Hawkular Metrics component, and data is pulled from this.

For this demo, we wanted to show how it is possible to send directly data into Hawkular Alerting for evaluation.
In a new bash session, executes the script:

    ./04_agent-process.sh

Now, we are ready to test the availability.

Step 3.1: Stop the Wildfly Server

Try to stop the Wildfly Server started on Step 1 with a CTRL+C.
Check that in two seconds Hawkular Alerting should receive an Alert like:

21:16:43,555 INFO  [org.hawkular.alerts.actions.api] (standalone-action-0) HAWKALERT240001: Plugin [email] has received an action message: [StandaloneActionMessage[action=Action[eventId='wildfly-availability-1478549803040-a3de9345-8a97-4cd9-a18f-3e32b1791f70', ctime=1478549803042, event=Alert{severity=CRITICAL, status=OPEN, notes=[], lifecycle=[LifeCycle{user='system', status=OPEN, stime=1478549803040}], resolvedEvalSets=null}, result='WAITING']]]

If an smtp email server is configured you should received a proper open Alert email.

Step 3.2: Re-start the Wildfly Server

Re-start the server with the script

    ./02_start-wildfly.sh

Check that Hawkular Alerting should automatically resolve the open Alert.

21:16:59,549 INFO  [org.hawkular.alerts.actions.api] (standalone-action-1) HAWKALERT240001: Plugin [email] has received an action message: [StandaloneActionMessage[action=Action[eventId='wildfly-availability-1478549803040-a3de9345-8a97-4cd9-a18f-3e32b1791f70', ctime=1478549819082, event=Alert{severity=CRITICAL, status=RESOLVED, notes=[Note{user='AutoResolve', ctime=1478549819066, text='Trigger AutoResolve=True'}], lifecycle=[LifeCycle{user='system', status=OPEN, stime=1478549803040}, LifeCycle{user='AutoResolve', status=RESOLVED, stime=1478549819066}], resolvedEvalSets=[[AvailabilityConditionEval [condition=AvailabilityCondition [triggerId='wildfly-availability', triggerMode=AUTORESOLVE, dataId='demo-avail', operator='UP'], value=UP, match=true, evalTimestamp=1478549819038, dataTimestamp=1478549817727]]]}, result='WAITING']]]

If an smtp email server is configured you should received a proper resolved Alert email.

Step 4: Start a log agent

We can also to monitor the Wildfly Server log file and send specific log lines like events into Hawkular Alerting.
In a new bash session:

    ./05_agent-log.sh

Now, we are ready to test the deployments and applications triggers.

Step 4.1: Deploy a demo app

In a new bash session:

    ./06_deploy-app.sh

Check that Hawkular Alerting trigger events for deployed applications.

21:33:21,441 INFO  [org.hawkular.alerts.actions.api] (standalone-action-4) HAWKALERT240001: Plugin [email] has received an action message: [StandaloneActionMessage[action=Action[eventId='wildfly-deployments-1478550801091-4c88ec46-69fa-4a48-b36f-f77d5a5d5534', ctime=1478550801091, event=Event [tenantId=my-organization, id=wildfly-deployments-1478550801091-4c88ec46-69fa-4a48-b36f-f77d5a5d5534, ctime=1478550801091, category=TRIGGER, dataId=wildfly-deployments, dataSource=_none_, text=Generate events on deployments, context={}, tags={}, trigger=Trigger [tenantId=my-organization, id=wildfly-deployments, type=STANDARD, eventType=EVENT, name=Deployments on Wildfly Server, description=Generate events on deployments, eventCategory=null, eventText=null, severity=MEDIUM, context={}, actions=[TriggerAction[tenantId='my-organization', actionPlugin='email', actionId='notify-to-developers', states=[], calendar='null']], autoDisable=false, autoEnable=false, autoResolve=false, autoResolveAlerts=true, autoResolveMatch=ALL, memberOf=null, dataIdMap={}, enabled=true, firingMatch=ALL, mode=FIRING, tags={}]], result='WAITING']]]

If an smtp email server is configured you should received a proper Event email.

Step 4.2: Undeploy demo app

In a bash session:

    ./07_undeploy-app.sh

Check that Hawkular Alerting trigger events for undeployed applications.

21:32:51,448 INFO  [org.hawkular.alerts.actions.api] (standalone-action-3) HAWKALERT240001: Plugin [email] has received an action message: [StandaloneActionMessage[action=Action[eventId='wildfly-undeployments-1478550771090-54c62ab9-16dc-4999-99cf-806e343158a2', ctime=1478550771090, event=Event [tenantId=my-organization, id=wildfly-undeployments-1478550771090-54c62ab9-16dc-4999-99cf-806e343158a2, ctime=1478550771090, category=TRIGGER, dataId=wildfly-undeployments, dataSource=_none_, text=Generate events on undeployments, context={}, tags={}, trigger=Trigger [tenantId=my-organization, id=wildfly-undeployments, type=STANDARD, eventType=EVENT, name=Undeployments on Wildfly Server, description=Generate events on undeployments, eventCategory=null, eventText=null, severity=MEDIUM, context={}, actions=[TriggerAction[tenantId='my-organization', actionPlugin='email', actionId='notify-to-developers', states=[], calendar='null']], autoDisable=false, autoEnable=false, autoResolve=false, autoResolveAlerts=true, autoResolveMatch=ALL, memberOf=null, dataIdMap={}, enabled=true, firingMatch=ALL, mode=FIRING, tags={}]], result='WAITING']]]

If an smtp email server is configured you should received a proper Event email.

Step 4.3: Interact with the demo app

Repeat Step 4.1 to deploy again the demo app.

Open a browser on:

	http://localhost:8230/wildfly-helloworld-html5/

This demo app is configured to print INFO messages every time the form is submitted.

Using name1 as a name will generate an app WARN log.
Using name2 will generate an app ERROR log.
Check how Dampening is working and only and alert will be generated when there are 3 WARN messages
per 10 total messages.
Check that an alert will be generated on every single app ERROR message.

Step 5: Acknowledge and resolve open Alerts

Triggers without AUTORESOLVE feature enabled fire Alerts that need manual intervention.
Alerts are designed to indicate that someone needs to review or perform some task on it.

Alerts support a lifecycle of OPEN, ACKNOWLEDGED and RESOLVED states.

To acknowledged pending open alerts wen can use the following script:

    ./08_acknowledge_alerts.sh

To resolve acknowledge alerts we can use the following script:

    ./09_resolve_alerts.sh