Version 1.1.38 (#68)

* Version 1.1.38 - Docs for Confluent Interceptors monitoring * circli use latest Python * Switch to metric store * Work in progress * Fix Issue #64 - Missing double quotes in the flush interval telegraf config UI generator #64 * Interceptors work in progress * Fix Issue #66 - Missing dot in telegraf config for Kafka Connect #66 * Work in progress * Version 1.1.38 - Feature: Integration with Confluent Interceptor Monitoring for producers and consumers advanced lag monitoring with Confluent - Change: Kafka Smart Monitoring goes Dark theme! - Change: When no Kafka Connect source or sink connector have been detected, show the aggregated single form in black rather than yellow - Fix: Issue #64 - mMissing double quotes in the flush interval telegraf config UI generator #64 - Fix: Issue #66 - Missing dot in telegraf config for Kafka Connect #66 * gitignore, appinspect * Final release * docs update * fix in props.conf for interceptor * Final release * last fix * Fix Issue ##67 * Version 1.1.38 * fix props.conf * optimized props * fixing yaml docker-compose * minor improvement * extend troubleshooting * docs under review * docs
guilhemmarchand · Oct 22, 2020 · 7beb6dd · 7beb6dd
1 parent 98bb1b1
commit 7beb6dd
Show file tree

Hide file tree

Showing 84 changed files with 1,477 additions and 75 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -20,7 +20,7 @@ version: 2.1
 executors:
   content-executor:
     docker:
-      - image: circleci/python:3.7.2
+      - image: circleci/python:latest
     working_directory: ~/repo
 
 jobs:

diff --git a/.gitignore b/.gitignore
@@ -33,3 +33,6 @@ backups
 # Mac
 .DS_Store
 */.DS_Store
+
+# appinspect reports
+appinspect_report_*
diff --git a/build.sh b/build.sh
@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #set -x
 
+# for Mac OS X
+export COPYFILE_DISABLE=true
+
 PWD=`pwd`
 app="telegraf-kafka"
 version=`grep 'version =' telegraf-kafka/default/app.conf | awk '{print $3}' | sed 's/\.//g'`

diff --git a/docs/about.rst b/docs/about.rst
@@ -19,6 +19,8 @@ About
 
 * LinkedIn Kafka monitor to provide end to end monitoring (https://github.com/linkedin/kafka-monitor)
 
+* Confluent Interceptors Monitoring (https://docs.confluent.io/current/control-center/installation/clients.html)
+
 * Kafka Consumers lag monitoring with Burrow (https://github.com/linkedin/Burrow)
 
 **An ITSI module is also available:** https://da-itsi-telegraf-kafka.readthedocs.io

diff --git a/docs/conf.py b/docs/conf.py
@@ -130,7 +130,7 @@
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-#html_logo = None
+html_logo = "img/logo.png"
 
 # The name of an image file (relative to this directory) to use as a favicon of
 # the docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32

diff --git a/docs/docker_testing.rst b/docs/docker_testing.rst
@@ -9,15 +9,17 @@ https://github.com/guilhemmarchand/kafka-docker-splunk
 
 *Example:*
 
-- 3 x nodes Zookeeper cluster
-- 3 x nodes Apache Kafka brokers cluster
-- 3 x nodes Apache Kafka connect cluster
-- 1 x node Confluent schema-registry
-- 1 x Splunk standalone server running in docker
-- 1 x LinkedIn Kafka monitor node
-- 1 x Telegraf collector container to collect metrics from Zookeeper, Kafka brokers
-- 1 x Telegraf collector container to collect metrics from Kafka Connect (including source and sink tasks)
-- 1 x Telegraf collector container to collect metrics from LinkedIn Kafka monitor
+- Zookeeper cluster (3 nodes)
+- Kafka broker cluster (3 nodes)
+- Kafka connect cluster (1 node, can be extended up to 3 or more with additional config)
+- Confluent schema-registry
+- Confluent kafka-rest
+- Confluent ksql-server
+- Kafka Xinfra SLA monitor container
+- Telegraf container polling and sending to your Splunk metric store
+- Yahoo Kafka Manager
+- Confluent Interceptor console collector
+- Kafka Burrow Consumer lag Monitoring
 
 .. image:: img/draw.io/docker_template.png
    :alt: docker_template

diff --git a/docs/health_view.rst b/docs/health_view.rst
@@ -61,6 +61,10 @@ Zookeeper dashboard view
    :alt: dashboard_zookeeper1.png
    :align: center
 
+.. image:: img/dashboard_zookeeper2.png
+   :alt: dashboard_zookeeper2.png
+   :align: center
+
 Kafka broker dashboard view
 ###########################
 
@@ -100,6 +104,10 @@ Kafka broker dashboard view
    :alt: dashboard_kafka_broker_pic9.png
    :align: center
 
+.. image:: img/dashboard_kafka_broker_pic10.png
+   :alt: dashboard_kafka_broker_pic10.png
+   :align: center
+
 Kafka topic dashboard view
 ##########################
 
@@ -138,6 +146,14 @@ Kafka connect dashboard view
    :alt: dashboard_kafka_connect_pic5.png
    :align: center
 
+.. image:: img/dashboard_kafka_connect_pic6.png
+   :alt: dashboard_kafka_connect_pic6.png
+   :align: center
+
+.. image:: img/dashboard_kafka_connect_pic7.png
+   :alt: dashboard_kafka_connect_pic7.png
+   :align: center
+
 Kafka connect sink task dashboard view
 ######################################
 
@@ -153,6 +169,10 @@ Kafka connect sink task dashboard view
    :alt: dashboard_kafka_connect_sink_pic3.png
    :align: center
 
+.. image:: img/dashboard_kafka_connect_sink_pic4.png
+   :alt: dashboard_kafka_connect_sink_pic4.png
+   :align: center
+
 Kafka connect source task dashboard view
 ########################################
 
@@ -168,6 +188,17 @@ Kafka connect source task dashboard view
    :alt: dashboard_kafka_connect_source_pic3.png
    :align: center
 
+.. image:: img/dashboard_kafka_connect_source_pic4.png
+   :alt: dashboard_kafka_connect_source_pic4.png
+   :align: center
+
+Confluent Interceptors Monitoring
+#################################
+
+.. image:: img/dashboard_confluent_interceptors.png
+   :alt: dashboard_confluent_interceptors.png
+   :align: center
+
 Confluent schema-registry dashboard view
 ########################################
 
@@ -179,8 +210,8 @@ Confluent schema-registry dashboard view
    :alt: dashboard_schema_registry_pic2.png
    :align: center
 
-LinkedIn Kafka monitor view
-###########################
+Xinfra Kafka monitor view
+#########################
 
 .. image:: img/dashboard_kafka_monitor_pic1.png
    :alt: dashboard_kafka_monitor_pic1.png

diff --git a/docs/img/confluent_interceptor1.png b/docs/img/confluent_interceptor1.png
diff --git a/docs/img/confluent_interceptor2.png b/docs/img/confluent_interceptor2.png
diff --git a/docs/img/coremain.png b/docs/img/coremain.png
diff --git a/docs/img/dashboard_confluent_interceptors.png b/docs/img/dashboard_confluent_interceptors.png
diff --git a/docs/img/dashboard_kafka_broker_pic1.png b/docs/img/dashboard_kafka_broker_pic1.png
diff --git a/docs/img/dashboard_kafka_broker_pic10.png b/docs/img/dashboard_kafka_broker_pic10.png
diff --git a/docs/img/dashboard_kafka_broker_pic2.png b/docs/img/dashboard_kafka_broker_pic2.png
diff --git a/docs/img/dashboard_kafka_broker_pic3.png b/docs/img/dashboard_kafka_broker_pic3.png
diff --git a/docs/img/dashboard_kafka_broker_pic4.png b/docs/img/dashboard_kafka_broker_pic4.png
diff --git a/docs/img/dashboard_kafka_broker_pic5.png b/docs/img/dashboard_kafka_broker_pic5.png
diff --git a/docs/img/dashboard_kafka_broker_pic6.png b/docs/img/dashboard_kafka_broker_pic6.png
diff --git a/docs/img/dashboard_kafka_broker_pic7.png b/docs/img/dashboard_kafka_broker_pic7.png
diff --git a/docs/img/dashboard_kafka_broker_pic8.png b/docs/img/dashboard_kafka_broker_pic8.png
diff --git a/docs/img/dashboard_kafka_broker_pic9.png b/docs/img/dashboard_kafka_broker_pic9.png
diff --git a/docs/img/dashboard_kafka_connect_pic1.png b/docs/img/dashboard_kafka_connect_pic1.png
diff --git a/docs/img/dashboard_kafka_connect_pic2.png b/docs/img/dashboard_kafka_connect_pic2.png
diff --git a/docs/img/dashboard_kafka_connect_pic3.png b/docs/img/dashboard_kafka_connect_pic3.png
diff --git a/docs/img/dashboard_kafka_connect_pic4.png b/docs/img/dashboard_kafka_connect_pic4.png
diff --git a/docs/img/dashboard_kafka_connect_pic5.png b/docs/img/dashboard_kafka_connect_pic5.png
diff --git a/docs/img/dashboard_kafka_connect_pic6.png b/docs/img/dashboard_kafka_connect_pic6.png
diff --git a/docs/img/dashboard_kafka_connect_pic7.png b/docs/img/dashboard_kafka_connect_pic7.png
diff --git a/docs/img/dashboard_kafka_connect_sink_pic1.png b/docs/img/dashboard_kafka_connect_sink_pic1.png
diff --git a/docs/img/dashboard_kafka_connect_sink_pic2.png b/docs/img/dashboard_kafka_connect_sink_pic2.png
diff --git a/docs/img/dashboard_kafka_connect_sink_pic3.png b/docs/img/dashboard_kafka_connect_sink_pic3.png
diff --git a/docs/img/dashboard_kafka_connect_sink_pic4.png b/docs/img/dashboard_kafka_connect_sink_pic4.png
diff --git a/docs/img/dashboard_kafka_connect_source_pic1.png b/docs/img/dashboard_kafka_connect_source_pic1.png
diff --git a/docs/img/dashboard_kafka_connect_source_pic2.png b/docs/img/dashboard_kafka_connect_source_pic2.png
diff --git a/docs/img/dashboard_kafka_connect_source_pic3.png b/docs/img/dashboard_kafka_connect_source_pic3.png
diff --git a/docs/img/dashboard_kafka_connect_source_pic4.png b/docs/img/dashboard_kafka_connect_source_pic4.png
diff --git a/docs/img/dashboard_kafka_topic_pic1.png b/docs/img/dashboard_kafka_topic_pic1.png
diff --git a/docs/img/dashboard_kafka_topic_pic2.png b/docs/img/dashboard_kafka_topic_pic2.png
diff --git a/docs/img/dashboard_kafka_topic_pic3.png b/docs/img/dashboard_kafka_topic_pic3.png
diff --git a/docs/img/dashboard_zookeeper1.png b/docs/img/dashboard_zookeeper1.png
diff --git a/docs/img/dashboard_zookeeper2.png b/docs/img/dashboard_zookeeper2.png
diff --git a/docs/img/draw.io/Kafka ITSI module.xml b/docs/img/draw.io/Kafka ITSI module.xml
diff --git a/docs/img/draw.io/confluent_interceptor_diagram.png b/docs/img/draw.io/confluent_interceptor_diagram.png
diff --git a/docs/img/draw.io/overview_diagram.png b/docs/img/draw.io/overview_diagram.png
diff --git a/docs/img/logo.png b/docs/img/logo.png
diff --git a/docs/img/overview_brokers.png b/docs/img/overview_brokers.png
diff --git a/docs/img/overview_brokers_detailed.png b/docs/img/overview_brokers_detailed.png
diff --git a/docs/img/overview_confluent_interceptors.png b/docs/img/overview_confluent_interceptors.png
diff --git a/docs/img/overview_topics.png b/docs/img/overview_topics.png
diff --git a/docs/img/overview_topics_details.png b/docs/img/overview_topics_details.png