Merge pull request #40 from guilhemmarchand/version_1025

Version 1025
guilhemmarchand · Nov 23, 2019 · 476d732 · 476d732
2 parents 3325f36 + 15fd94f
commit 476d732
Show file tree

Hide file tree

Showing 20 changed files with 5,497 additions and 1,095 deletions.
diff --git a/docs/about.rst b/docs/about.rst
@@ -1,10 +1,8 @@
 About
 =====
 
-* Author: Guilhem Marchand
+* Author: Guilhem Marchand, Splunk certified consultant and part of Splunk Professional Services
 
 * First release published in July 2019
 
-* Purposes:
-
-**The Splunk application for data sources monitoring provides a handy user interface associated with a simple and efficient workflow for discovery, management and alerting of the data sources and hosts availability**
+* License: Apache License 2.0
diff --git a/docs/configuration.rst b/docs/configuration.rst
@@ -90,12 +90,14 @@ Finally, in addition the following macro is used within the searches, and can be
 Activation of builtin alerts
 ============================
 
-**TrackMe provides out of the box alerts that be used to deliver alerting when a data source or host reaches a red alert:**
+**TrackMe provides out of the box alerts that can be used to deliver alerting when a monitored component reaches a red state:**
 
 - TrackMe - Alert on data source availability
 
 - TrackMe - Alert on data host availability
 
+- TrackMe - Alert on metric host availability
+
 **These alerts are disabled by default, and need to be manually enabled if you wish to use them in your global alerting workflow.**
 
 trackme_admin role for granular access

diff --git a/docs/deployment.rst b/docs/deployment.rst
@@ -34,3 +34,5 @@ Upgrades
 ========
 
 Upgrading the Splunk application is pretty much the same operation than the initial deployment.
+
+All of TrackMe components and configuration items are upgraded resilient, in respects with Splunk configuration good practices.
diff --git a/docs/img/configure_ui.png b/docs/img/configure_ui.png
diff --git a/docs/img/metric_host_drilldown.png b/docs/img/metric_host_drilldown.png
diff --git a/docs/img/metric_host_main.png b/docs/img/metric_host_main.png
diff --git a/docs/index.rst b/docs/index.rst
@@ -1,18 +1,20 @@
-.. dsmon documentation master file, created by
+.. trackme documentation master file, created by
    sphinx-quickstart on Tue Sep 18 23:25:46 2018.
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
 Welcome to the Splunk TrackMe application documentation
 ========================================================
 
-**The Splunk TrackMe application for data source monitoring provides a handy user interface and workflow for Splunk administrators to monitor the availability of their data sources:**
+**The Splunk TrackMe application provides automated monitoring and visibility insight of your data sources availability, with a powerful user interface and workflow for Splunk product owners to detect and alert on failures or abnormal latency:**
 
-- Discover and store data sources and hosts information and states
-- Provides a handy and easy user interface to manage states, configuration and quickly trouble shoot alerts
-- Analyse and detect lack of data and performance lagging of data sources and hosts
+- Discover and store key states information of data sources, data hosts and metric hosts availability
+- Provides a powerful user interface to manage activation states, configuration and quickly trouble availability failure detection
+- Analyse and detect lack of data and performance lagging of data sources and hosts within your Splunk deployment
 - Record and investigate historical changes of statuses, as well as administrators changes (audit flipping and changes)
-- Easy administration via graphical human interface
+- Easy administration via graphical human interface from A to Z
+- No matters the purpose of your Splunk deployment, trackMe will easily become an essential and easy piece of your deployment, and even providing efficient answers to PCI and compliance requirements
+- Never let again your team be the last to discover what empty and no results found mean!
 
 .. image:: img/screenshot.png
    :alt: screenshot.png
@@ -48,6 +50,15 @@ This tiny application provides a handy user interface associated with a simple b
 
 Made by Splunk admins for Splunk admins, the TrackMe application provides builtin powerful features to monitor and administer you data source monitoring the easy way!
 
+**Use case for TrackMe?**
+
+No matters the purpose of your Splunk deployment, trackMe will easily become an essential and positive piece of your Splunk journey:
+
+- Security Operation Centers (SOC) with or without Enterprise Security compliance: detect lack of data, abnormal latency potentially impacting your security posture
+- PCI and compliance: deliver, alert and action
+- Monitoring and insight visibility about your indexes, sourcetypes, events and metrics
+- General data activity monitoring and detection of Zombie data
+
 Overview:
 =========
 

diff --git a/docs/releasenotes.rst b/docs/releasenotes.rst
@@ -1,6 +1,14 @@
 Release notes
 #############
 
+Version 1.0.25
+==============
+
+- feature: Introducing support for metric store availability monitoring with metric hosts and granular detection of metric availability failure and latency
+- feature: Refresh button in all modal windows, improved placements for buttons, improved navigation coherence between modal windows
+- fix: data host modal embedded charts and table should honour tstats main filter, whitelists and blacklists
+- fix: Improved Mobile dashboard
+
 Version 1.0.24
 ==============
 

diff --git a/docs/support.rst b/docs/support.rst
@@ -1,7 +1,7 @@
 Support
 #######
 
-This is application is community supported.
+This application is community supported.
 
 To get support, use of one the following options:
 
@@ -15,7 +15,7 @@ Open a question in Splunk answers for the application:
 Splunk community slack
 ======================
 
-Contact me on Splunk community slack, or even better, ask the community !
+Contact me on Splunk community slack, and even better, ask the community!
 
 - https://splunk-usergroups.slack.com
 

diff --git a/docs/userguide.rst b/docs/userguide.rst
@@ -96,19 +96,58 @@ The data hosts state table exposes the information and the state of each data ho
 - **state:** the state of the data source based on the monitoring rules for this data host
 - **data_last_lag_seen:** the latest lag value in seconds seen for that data source
 - **data_max_lag_allowed:** the maximal value of lag accepted for this data source
-- **monitoring:** the monitoring state of this data source, can be enabled or disabled
+- **monitoring:** the monitoring state of this data host, can be enabled or disabled
 - **data_monitoring_wdays:** defines the week days monitoring rule for the data source, different values are possible and exposed further in this documentation
 
 **Trackers:**
 
-The update of the data source monitoring collection is driven by the execution of the data source scheduled tracker reports:
+The update of the data host monitoring collection is driven by the execution of the data host scheduled tracker reports:
 
 - TrackMe - Data hosts availability short term tracker, runs every 5 minutes over the last 4 hours
 - TrackMe - Data hosts availability short term tracker, runs every hour over the last 7 days
 
 Both tracker reports rely on extremely fast and cost less tstats queries.
 Even on very large environments, the tracker's run time and running costs are very limited.
 
+Metric hosts availability tracking
+==================================
+
+.. image:: img/metric_host_main.png
+   :alt: metrics_host_main.png
+   :align: center
+
+**Metric hosts availability tracking monitors hosts generating metrics stored into metrics indexes, it provides:**
+
+- Single form overview of the total number of metrics hosts discovered ("METRIC HOSTS")
+- Single form overview of the number of metric hosts in alert ("ANY PRIORITY METRIC HOSTS IN ALERT")
+- Single form overview of the number of metric hosts in alert with an high priority ("HIGH PRIORITY METRIC HOSTS IN ALERT")
+- Single form overview of the total number of metric hosts that are not being monitored ("METRIC HOSTS NOT MONITORED")
+- Filters for investigations
+- A dynamic and interactive table representation of the metric hosts content. (see bellow)
+
+**Metric host state table:**
+
+The metric hosts state table exposes the information and the state of each metric host:
+
+- **metric_host:** the discovered name of the host
+- **metric_index:** the name of the index(es) where resides the data
+- **metric_category:** this field represents the main category of the metrics group, being the first segment of the metric_name value
+- **metric_details_human:** A multi-value field which tracks for each metric category the individual status
+- **latest time:** The very latest time a metric was seen for the host (between all metric categories)
+- **priority:** a value that describes the priority (low / medium / high) of the metric host, to be used for granular alerting purposes
+- **state:** the state of the metric host, by default shall any of the metric categories enters in a red state so will be the host state
+- **monitoring:** the monitoring state of this host, can be enabled or disabled
+
+**Trackers:**
+
+The update of the metric host monitoring collection is driven by the execution of the metric host scheduled tracker report:
+
+- TrackMe - metric hosts availability tracker, runs every 5 minutes over the last 5 minutes
+
+The tracker uses the mstats command to retrieve the latest value and the according time on a per metric category.
+
+These information are merged with the existing (if any) information stored in the KVstore collection, to finally define a state for each metric category, and a state for each host.
+
 Interactive drilldown and administration of objects
 ===================================================
 
@@ -126,14 +165,20 @@ The main concept of the user interface resides in providing an easy and interact
    :alt: data_host_drilldown.png
    :align: center
 
-*Both tracking provides the same types of access to the administration options:*
+*Accessing a metric host overview and options:*
+
+.. image:: img/metric_host_drilldown.png
+   :alt: metric_host_drilldown.png
+   :align: center
+
+*Different options are available depending on the type of object:*
 
 .. image:: img/drilldown_mainoptions.png
    :alt: drilldown_mainoptions.png
    :align: center
 
-Modification of data sources or hosts monitoring rules
-======================================================
+Modification of objects and monitoring rules
+============================================
 
 Enabling / Deactivating monitoring
 ----------------------------------
@@ -142,7 +187,7 @@ Enabling / Deactivating monitoring
    :alt: enable_disable.png
    :align: center
 
-Each object, either a data source or a data host, has a monitoring state that will be enabled or disabled.
+Each object has a monitoring state that will be enabled or disabled.
 
 The monitoring state drives different aspects of the restitution within the UI, and as well the fact that this object will result in an alert trigger or not.
 
@@ -185,6 +230,8 @@ Modifying monitoring week days
 
 **You can modify the rules for days of week monitoring, which means specifying for which days of the week a data will be monitored actively:**
 
+*Week days monitoring rules apply to event data only (data sources and hosts)*
+
 .. image:: img/week_days1.png
    :alt: week_days1.png
    :align: center

diff --git a/trackme/bin/getidxwhitelist.py b/trackme/bin/getidxwhitelist.py
@@ -55,6 +55,12 @@ class CountMatchesCommand(StreamingCommand):
         **Description:** Name of the field that will hold the match count''',
         require=True, validate=validators.Fieldname())
 
+    outname = Option(
+        doc='''
+        **Syntax:** **outname=***<outname>*
+        **Description:** Name of the outpuf field that will hold the index name''',
+        require=True, validate=validators.Fieldname())
+
     pattern = Option(
         doc='''
         **Syntax:** **pattern=***<regular-expression>*
@@ -64,6 +70,7 @@ class CountMatchesCommand(StreamingCommand):
     def stream(self, records):
         self.logger.debug('CountMatchesCommand: %s', self)  # logs command line
         pattern = self.pattern
+        outname = self.outname
 
         count = 0
         whitelist = ""
@@ -82,7 +89,7 @@ def stream(self, records):
 
         # whitelist is empty
         if count == 0:
-            whitelist = "[('data_index', '*')]"
+            whitelist = "[('" + str(outname) + "', '*')]"
 
         yield {'_raw': str(whitelist)}
 

diff --git a/trackme/default/app.conf b/trackme/default/app.conf
@@ -16,4 +16,4 @@ label = TrackMe
 [launcher]
 author = Guilhem Marchand
 description = Easy data tracking system for Splunk admins
-version = 1.0.24
+version = 1.0.25
diff --git a/trackme/default/collections.conf b/trackme/default/collections.conf
@@ -14,6 +14,16 @@ replicate = false
 [kv_trackme_host_monitoring]
 replicate = false
 
+#
+# Metric Hosts availability monitoring
+#
+
+[kv_trackme_metric_host_monitoring]
+replicate = false
+
+[kv_trackme_metric_lagging_definition]
+replicate = false
+
 #
 # blacklisting for data source monitoring
 #
@@ -47,6 +57,11 @@ replicate = false
 [kv_trackme_data_host_monitoring_whitelist_index]
 replicate = false
 
+# whitelisting for metric host monitoring
+
+[kv_trackme_metric_host_monitoring_whitelist_index]
+replicate = false
+
 #
 # blacklisting for data host monitoring
 #
@@ -66,6 +81,13 @@ replicate = false
 [kv_trackme_data_host_monitoring_blacklist_sourcetype]
 replicate = false
 
+#
+# blacklisting for metric host monitoring
+#
+
+[kv_trackme_metric_host_monitoring_blacklist_metric_category]
+replicate = false
+
 #
 # Default lagging value
 #