Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collect Apache Kafka consumer metrics #1226

Merged
merged 19 commits into from
Nov 15, 2021
Merged

Collect Apache Kafka consumer metrics #1226

merged 19 commits into from
Nov 15, 2021

Commits on Nov 4, 2021

  1. Add KafkaConsumerMetricRegistry for collecting kafka consumer metrics 

    Implements KafkaConsumerMetricsRegistry as singleton for collecting the Apache Kafka consumer metrics.
Some minor code styling refactorings.
Adjust ClientActorPropsFactory to be a singleton, since it doesn't have any state.

Signed-off-by: David Schwilk <david.schwilk@bosch.io>
    @DerSchwilk
    DerSchwilk committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    717d0fb View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2021

  1. Add Kamon orchestration for Kafka consumer metrics. 

    Adds Kamon gauges for kafka consumer metrics. The kamon gauges report the metric values as double. Thus also added double gauge API to Gauge. Remembers new consumers for metric reporting init, since the consumerControl is not ready from the get-go and produces NullPointerException if the metrics are accessed directly after instantiation.

Signed-off-by: David Schwilk <david.schwilk@bosch.io>
    @DerSchwilk
    DerSchwilk committed Nov 8, 2021
    Configuration menu
    Copy the full SHA
    20c4f7c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into feature/kafka-metrics

    @DerSchwilk
    DerSchwilk committed Nov 8, 2021
    Configuration menu
    Copy the full SHA
    c915b9d View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2021

  1. Adjust kafka consumer metric naming 

    Signed-off-by: David Schwilk <david.schwilk@bosch.io>
    @DerSchwilk
    DerSchwilk committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    a97a0d0 View commit details
    Browse the repository at this point in the history

  2. Fix concurrency issues in KafkaConsumerMetricsRegistry 

    Signed-off-by: David Schwilk <david.schwilk@bosch.io>
    @DerSchwilk
    DerSchwilk committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    2986ac9 View commit details
    Browse the repository at this point in the history

  3. Correct dependency mgmt. for kamon akka 

    Signed-off-by: David Schwilk <david.schwilk@bosch.io>
    @DerSchwilk
    DerSchwilk committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    56a34ef View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2021

  1. Remove unused dependency 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    2939d4d View commit details
    Browse the repository at this point in the history

  2. Review changes 

    * Passing the KafkaConsumerMetricsRegistry to the KafkaConsumerStreamFactory
  since it is used for all kinds of streams.
* Using a unique identifier per stream to make sure that each stream
  has its own metrics available. Before we only could distinguish between
  qos 0 and qos 1 per connection.
* Remove concept of Remember registration and just register immediately
* Handle NullpointerException which seems to be a bug in kafka streams

Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    18f4ee6 View commit details
    Browse the repository at this point in the history

  3. Use consumerId also for deregistering from kafka consumer metrics 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    3aa2869 View commit details
    Browse the repository at this point in the history

  4. De-Register automatically after the consumerControl has completed eit… 

    …her successfully or execptionally

* This makes the registration more stable/consistent because it's less likely
  that a developer forgets to de-register or makes de-register wrong
  (see previous commit)

Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 11, 2021
    Configuration menu
    Copy the full SHA
    362399b View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2021

  1. Move KafkaConsumerMetricsRegistry registering to extra method 

    In effort to fix immutability test failures:
"The 'this' reference is passed outwith the constructor."

Signed-off-by: David Schwilk <david.schwilk@bosch.io>
    @DerSchwilk
    DerSchwilk committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    71516e2 View commit details
    Browse the repository at this point in the history

  2. enforce connectionId type for parameters 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    cbda604 View commit details
    Browse the repository at this point in the history

  3. Remove ConsumerMetricsRegistry and schedule reporting in consumer act… 

    …or instead

* This avoids manual thread handling

Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    ef6eb9e View commit details
    Browse the repository at this point in the history

  4. Enforce stronger type for source suppliers in consumer streams 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    f88ca53 View commit details
    Browse the repository at this point in the history

  5. remove unnecessary blank lines 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    6d1188f View commit details
    Browse the repository at this point in the history

  6. Remove unnecessary config 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    0552838 View commit details
    Browse the repository at this point in the history

  7. Enforce stronger type for sinks in ConsumerStream constructors 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    1bdae37 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2021

  1. Merge branch 'master' into feature/kafka-metrics 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 15, 2021
    Configuration menu
    Copy the full SHA
    584b7d8 View commit details
    Browse the repository at this point in the history

  2. use camelCase for tag names 

    Signed-off-by: Yannic Klem <yannic.klem@bosch.io>
    @Yannic92
    Yannic92 committed Nov 15, 2021
    Configuration menu
    Copy the full SHA
    a2114a5 View commit details
    Browse the repository at this point in the history