Elasticsearch data performance data collection

This repository contains the configuration to use OpenNMS collecting performance metrics provided through the Elasticsearch Cluster Stats API. With the provided configuration the metrics will be collected every 5 minutes and persisted in OpenNMS using the configured storage strategy like RRDtool or Cassandra based Newts.

Requirements

• Elasticsearch release with Cluster Stats API support

• OpenNMS Horizon/Meridian

• Installed with XML data collector plugin (included in > v20)

• Provision your Elasticsearch nodes into OpenNMS for monitoring

• The config example has OpenNMS installed in /opt/opennms, on Debian/Ubuntu change the path accordingly to /usr/share/opennms

The XML data collector plugin is not installed by default and can be installed with

Installing XML collector plugin < v20

yum install opennms-plugin-protocol-xml
apt-get install opennms-plugin-protocol-xml

After installing the plugin it is required to restart OpenNMS with

systemctl restart opennms

Install data collection configuration

Checkout this repository with git or copy the files manually using the GitHub raw link to download the configuration files.

Install configuration for metrics collection and RRD graph definitions

git clone https://github.com/opennms-config-modules/elasticsearch.git
cd elasticsearch
cp snmp-graph.properties.d/elasticsearch-cluster-graph.properties /opt/opennms/etc/snmp-graph.properties.d
cp xml-datacollection/elasticsearch-cluster-stats.xml /opt/opennms/etc/xml-datacollection
cp xml-datacollection/elasticsearch-cluster-health.xml /opt/opennms/etc/xml-datacollection

Configure Collectd

Add the following lines collectd-configuration.xml in the in the default installed ``example1 package. The collector service needs to be added on the very end of the file, it tells OpenNMS to the XML collector for this type of service. Enable the JSON Handler to allow parsing the output accordingly.

Create a new collection service for the Elasticsearch Cluster metrics

<service name="Elasticsearch" interval="300000" user-defined="false" status="on">
    <parameter key="collection" value="elasticsearch-cluster-status" />
    <parameter key="handler-class" value="org.opennms.protocols.json.collector.DefaultJsonCollectionHandler"/>
</service>

<collector service="Elasticsearch" class-name="org.opennms.protocols.xml.collector.XmlCollector"/>

Edit the xml-datacollection-config.xml which defines RRD configuration and add the Elasticsearch collection.

    <xml-collection name="elasticsearch-cluster-stats">
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
        <xml-source url="http://{ipaddr}:9200/_cluster/stats">
            <import-groups>xml-datacollection/elasticsearch-cluster-stats.xml</import-groups>
        </xml-source>
        <xml-source url="http://{ipaddr}:9200/_cluster/health">
            <import-groups>xml-datacollection/elasticsearch-cluster-health.xml</import-groups>
        </xml-source>
    </xml-collection>

Test Elasticsearch Cluster Status

The Elasticsearch service can be used to test if the cluster status is green and ok with Pollerd. For this case the HTTP Monitor can be used with the following configuration:

<service name="Elasticsearch" interval="300000" user-defined="true" status="on">
  <parameter key="retry" value="1"/>
  <parameter key="timeout" value="3000"/>
  <parameter key="port" value="9200"/>
  <parameter key="url" value="/_cluster/stats"/>
  <parameter key="response" value="200-202,299"/>
  <parameter key="response-text" value="~.*status.:.green.*"/>
</service>

<monitor service="Elasticsearch" class-name="org.opennms.netmgt.poller.monitors.HttpMonitor" />

Restart OpenNMS and provision a service on your nodes IP interface named Elasticsearch.

Add threshold alarms

Some default alarm configuration can be added if needed.

Threshd config package in thresholds/threshd-configuration.xml contains a package which will assign the thresholds to all nodes with an Elasticsearch service on an interface. In thresholds/thresholds.xml are the threshold definitions.

Those configurations can be copied into a running OpenNMS instances config.

Add threshold events

Event file is required for the thresholds above so alarm raising and automatical resolving is possible.

cp ${HOME}/opennms-config-workspace/elasticsearch/events/*.xml ${OPENNMS_HOME}/etc/events

Include in eventconf.xml

<event-file>events/es-threshold.events.xml</event-file>

Author

ronny@opennms.org / mfuhrmann@opennms.org