prometheus metric exporter #10412

Tiaaa · 2020-09-21T06:03:33Z

Fixes #8621

Adds a new extension prometheus-emitter to expose Druid metrics for collection directly by a Prometheus server.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

Key changed/added classes in this PR

org.apache.druid.emitter.prometheus.*

This is re-open the PR here #8621

jinfwhuang · 2020-10-02T18:24:05Z

extensions-contrib/prometheus-emitter/src/main/resources/defaultMetrics.json

@@ -0,0 +1,128 @@
+{
+  "query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", "conversionFactor": 1000.0, "help":  "Seconds taken to complete a query."},


Are these conversion a good idea?

It would mean that these metrics will be slightly different from how they are described in this documentation. https://druid.apache.org/docs/latest/operations/metrics.html

https://prometheus.io/docs/practices/naming/#base-units
It's coming from prometheus common practice.

cc @michaelschiff

Okay. I guess there are tradeoffs with either choices. Maybe a good way to do is that for those converted, we can just put the unit in the prometheus names. Otherwise, if we refer to the druid metrics doc, we would find the name of the metrics to be documented in a different unit.

ArvinZheng · 2020-10-28T01:00:22Z

extensions-contrib/prometheus-emitter/src/main/resources/defaultMetrics.json

+  "query/failed/count" : { "dimensions" : [], "type" : "count", "help": "Number of failed queries"},
+  "query/interrupted/count" : { "dimensions" : [], "type" : "count", "help": "Number of queries interrupted due to cancellation or timeout"},
+
+  "query/cache/delta/numEntries" : { "dimensions" : [], "type" : "count", "help": "Number of entries in cache"},


potential bug, deltas can be negative but Prometheus counter accepts only non-negative increment.

Will change to guage. This only happens in since last emission we got more entries evicted than added right.

...theus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitterConfig.java

ArvinZheng · 2020-10-28T01:24:38Z

.../prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitter.java

+  private final Metrics metrics;
+  private final PrometheusEmitterConfig config;
+  private final PrometheusEmitterConfig.Strategy strategy;
+  private final Pattern pattern = Pattern.compile("[^a-zA-Z0-9_][^a-zA-Z0-9_]*");


Reuse the pattern in PrometheusEmitterConfig

These two is not the same regex. The one in PromtheusEmitterConfig is for namespace regex that need to start with alphabetic character.

Oh, sorry, my bad.

ArvinZheng · 2020-10-28T01:24:51Z

...ns-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/Metrics.java

+  private static final Logger log = new Logger(Metrics.class);
+  private final Map<String, DimensionsAndCollector> map = new HashMap<>();
+  private final ObjectMapper mapper = new ObjectMapper();
+  private final Pattern pattern = Pattern.compile("[^a-zA-Z_:][^a-zA-Z0-9_:]*");


Reuse the pattern in PrometheusEmitterConfig

used the one in PrometheusEmitter.java

ArvinZheng · 2020-10-28T03:53:57Z

...ns-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/Metrics.java

+    }
+  }
+
+  public Map<String, DimensionsAndCollector> getMap()


maybe we can rename the map to registeredMetrics and then we could rename this method to getRegisteredMetrics(), I feel like this will be easier to read

ArvinZheng · 2020-10-28T04:02:54Z

...ns-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/Metrics.java

+      }).readValue(is);
+    }
+    catch (IOException e) {
+      throw new ISE(e, "Failed to parse metric dimensions and types");


same as above

ArvinZheng · 2020-10-28T04:11:42Z

.../prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitter.java

+    }
+  }
+
+  void emitMetric(ServiceMetricEvent metricEvent)


ArvinZheng · 2020-10-28T04:28:54Z

.../prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitter.java

+    Map<String, DimensionsAndCollector> map = metrics.getMap();
+    try {
+      for (DimensionsAndCollector collector : map.values()) {
+        pushGateway.push(collector.getCollector(), config.getNamespace(), ImmutableMap.of(config.getNamespace(), identifier));


potential NPE? if the configured strategy is not pushgateway, then this pushGateway wouldn't have been instantiated

Also should we use a more meaningful label name for identifier instead of using the config.getNamespace()?

Will add the null check - however flush() for this emitter should only called by close() which strategy check already done.

For the identifier label name, any suggestion? The config.namespace will be set in config files for each service. So for example peon task it could be peon=taskXXX as groupingKey.

ArvinZheng · 2020-10-28T04:37:16Z

.../prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/PrometheusEmitter.java

+
+
+  @Override
+  public void start()


we should schedule a task to push updates periodically when the strategy is set to pushgateway

Added, every 5min sounds reasonable?

Sorry I missed this - I think the scheduled executor may not be necessary. Main reason we've added strategy pushgateway is for things that are potentially too short-lived to be scraped by prometheus (in druid that's really just peon tasks). Things that are living long enough to be pushing every 5 minutes are likely not "task" based, and may be better fit for normal scraping. I lean toward keeping things simple, and pushing once at close seems sufficient.

Given the only metric pushed by peon is "last pushed timestamp", I think it's valid to remove the scheduled task. Removed.

ArvinZheng · 2020-10-28T04:46:16Z

...ns-contrib/prometheus-emitter/src/main/java/org/apache/druid/emitter/prometheus/Metrics.java

+
+  public DimensionsAndCollector getByName(String name, String service)
+  {
+    if (map.containsKey(name)) {


return Optional.ofNullable(map.get(name)).orElse(map.get(service + "_" + name));

Changed the second part to a getOrDefault() for simplification. I don't see the need of changing this function return type from DimensionsAndCollector to Optional<DimensionsAndCollector>

you don't need to change the return type, but anyway this is a minor comment, feel free to pick whichever you prefer

suhassumukh · 2021-01-04T15:21:20Z

Is there a timeline for this feature?

michaelschiff · 2021-01-04T21:10:11Z

For what is worth, Ive been running production clusters with this extension for monitoring for over a year. Things are stable but there are a couple open issues around pushgateway collection of Peon task metrics that are the main reasons to delay merging.

Tiaaa · 2021-01-15T03:40:52Z

Verification of metrics on a local mini druid cluster as shown in below table.

Coordinator, historical, broker, router, middle-manager, overlord set up as prometheus monitoring target.
Peon metrics is sent to pushgateway.

Component	Metric
Coordinator	# TYPE druid_segment_size gauge druid_segment_size{dataSource="wiki_test",} 21679.0 # TYPE druid_segment_loadqueue_count gauge druid_segment_loadqueue_count{server="stats_druid_historical_2_stats_druid_historical_stats_dev_svc_cluster_local_8083",} 0.0 druid_segment_loadqueue_count{server="stats_druid_historical_1_stats_druid_historical_stats_dev_svc_cluster_local_8083",} 0.0 druid_segment_loadqueue_count{server="stats_druid_historical_0_stats_druid_historical_stats_dev_svc_cluster_local_8083",} 1.0
Historical	# TYPE druid_query_time histogram druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.1",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.25",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.5",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.75",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="1.0",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="2.5",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="5.0",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="7.5",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="10.0",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="30.0",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="60.0",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="120.0",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="300.0",} 1.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="+Inf",} 1.0 druid_query_time_count{dataSource="wiki_test",type="scan",} 1.0 druid_query_time_sum{dataSource="wiki_test",type="scan",} 0.093
Broker	# TYPE druid_query_time histogram druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.1",} 2.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.25",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.5",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="0.75",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="1.0",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="2.5",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="5.0",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="7.5",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="10.0",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="30.0",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="60.0",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="120.0",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="300.0",} 3.0 druid_query_time_bucket{dataSource="wiki_test",type="scan",le="+Inf",} 3.0 druid_query_time_count{dataSource="wiki_test",type="scan",} 3.0 druid_query_time_sum{dataSource="wiki_test",type="scan",} 0.132 # TYPE druid_query_node_time histogram druid_query_node_time_bucket{server="stats_druid_historical_1_stats_druid_historical_stats_dev_svc_cluster_local_8083",le="0.1",} 0.0
Router	# TYPE druid_jvm_mem_used gauge druid_jvm_mem_used{memKind="heap",} 4.8370752E7 druid_jvm_mem_used{memKind="nonheap",} 6.4390328E7
Middle-manager	# TYPE druid_jvm_pool_used gauge druid_jvm_pool_used{poolKind="heap",poolName="PS_Old_Gen",} 2.8118984E7 druid_jvm_pool_used{poolKind="nonheap",poolName="Code_Cache",} 1.2116608E7 druid_jvm_pool_used{poolKind="heap",poolName="PS_Survivor_Space",} 0.0 druid_jvm_pool_used{poolKind="nonheap",poolName="Metaspace",} 4.5766376E7 druid_jvm_pool_used{poolKind="heap",poolName="PS_Eden_Space",} 2.5587941728E10
Overlord	# TYPE druid_task_run_time histogram druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="0.1",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="0.25",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="0.5",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="0.75",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="1.0",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="2.5",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="5.0",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="7.5",} 0.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="10.0",} 1.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="30.0",} 1.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="60.0",} 1.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="120.0",} 1.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="300.0",} 1.0 druid_task_run_time_bucket{dataSource="wiki_test",taskType="index_parallel",le="+Inf",} 1.0 druid_task_run_time_count{dataSource="wiki_test",taskType="index_parallel",} 1.0 druid_task_run_time_sum{dataSource="wiki_test",taskType="index_parallel",} 9.553
Peon	push_time_seconds{druid="index_parallel_wiki_test_2021-01-10T23:42:35.061Z",instance="",job="druid"} 1.6103221647556024e+09

michaelschiff · 2021-01-15T05:07:40Z

@Tiaaa awesome! I see peon metrics in pushgateway as well.

One thing - the label we're using right now is the task ID. I think this it going to be too high cardinality for Prometheus

clintropolis · 2021-01-15T06:15:06Z

hmm, it looks like the commits have become sort of messed up for this PR one way or another and github is showing a lot of unrelated commits. @Tiaaa any chance you can try clean this up to only show the changes of this PR to make it easier to review?

… updated statsd json)

…are not using seconds. Correct metric dimension name in documentation

…st multiple calls to PrometheusEmitter.start()

…alues. Additional tests

…task to pushGateway

…mmit

stroeovidiu · 2021-02-26T13:44:47Z

Any update on this ? Looking forward having it.

michaelschiff · 2021-02-27T00:05:48Z

@clintropolis it looks like this is the last failing build step: https://travis-ci.com/github/apache/druid/jobs/483326128 - seems unrelated to the new emitter. Are we good to merge?

clintropolis · 2021-02-27T00:18:59Z

@clintropolis it looks like this is the last failing build step: https://travis-ci.com/github/apache/druid/jobs/483326128 - seems unrelated to the new emitter. Are we good to merge?

Sorry for the delay, I will have a look as soon as I'm able and see if we can get this merged 👍

clintropolis

A few minor comments, but overall lgtm 👍 I don't know prometheus very well, but I think the mappings of metrics looks reasonable.

Thanks for your patience and persistence!

docs/development/extensions-contrib/prometheus.md

docs/operations/metrics.md

docs/development/extensions-contrib/prometheus.md

clintropolis · 2021-03-02T07:07:01Z

website/.spelling

@@ -1223,7 +1228,7 @@ SysMonitor
 TaskCountStatsMonitor
 TaskSlotCountStatsMonitor
 bufferCapacity
-bufferpoolName
+bufferPoolName


oops, missed one, it's causing CI to fail

clintropolis

👍

michaelschiff · 2021-03-09T18:56:31Z

@clintropolis anything left we need to do before merge?

stroeovidiu · 2021-03-09T19:07:59Z

Will this be available în the next 0.21.0 release?

Thank you

clintropolis · 2021-03-09T22:36:34Z

@clintropolis anything left we need to do before merge?

Oops no, sorry got distracted and hadn't got back to this yet.

Will this be available în the next 0.21.0 release?

Unfortunately we have already cut the branch for 0.21.0, after which we only merge bug fixes, so this will go out in the release after that. 0.21.0 has been a bit delayed, so it shouldn't be too much longer before we begin the next release as well.

Tiaaa mentioned this pull request Sep 21, 2020

prometheus-emitter #8621

Closed

8 tasks

jinfwhuang reviewed Oct 2, 2020

View reviewed changes

clintropolis added Area - Extension Area - Metrics/Event Emitting labels Oct 26, 2020

ArvinZheng reviewed Oct 28, 2020

View reviewed changes

Michael Schiff added 20 commits January 15, 2021 18:27

prometheus-emitter

ccbfbb7

use existing jetty server to expose prometheus collection endpoint

bb8eb39

unused variables

ad43466

better variable names

b04eaba

removed unused dependencies

82f408b

more metric definitions

b6750cc

reorganize

2dc2ced

use prometheus HTTPServer instead of hooking into Jetty server

c2a7131

temporary empty help string

5a91cc0

temporary non-empty help. fix incorrect dimension value in JSON (also…

a7851af

… updated statsd json)

added full help text. added metric conversion factor for timers that …

a213638

…are not using seconds. Correct metric dimension name in documentation

added documentation for prometheus emitter

2ec1a9b

safety for invalid labelNames

13d3fc3

fix travis checks

cd392eb

Unit test and better sanitization of metrics names and label values

8246c3c

add precondition to check namespace against regex

8ab89f2

use precompiled regex

9b4c70d

remove static imports. fix metric types

5221655

better docs. fix possible NPE in PrometheusEmitterConfig. Guard again…

abae5c0

…st multiple calls to PrometheusEmitter.start()

Update regex for label-value replacements to allow internal numeric v…

fa4a985

…alues. Additional tests

Tianxin Zhao and others added 4 commits January 15, 2021 18:27

Update pom file for prometheus-emitter

6c248ce

code review comments. Counter to gauge for cache metrics, periodical …

7d4f765

…task to pushGateway

Syntax fix

4d90a41

Dimension label regex include numeric character back, fix previous co…

3a7a2b6

…mmit

Tiaaa force-pushed the feature/prometheus-metric-exporter branch from 4b0b414 to 3a7a2b6 Compare January 16, 2021 02:28

Tianxin Zhao added 8 commits January 15, 2021 18:33

bump prometheus-emitter pom dev version

b61f1b3

Remove scheduled task inside poen that push metrics

b1b8d61

Fix checkstyle

e7e9e2f

Merge branch 'master' into feature/prometheus-metric-exporter

bffc63d

Unit test coverage

b405b55

Merge branch 'master' into feature/prometheus-metric-exporter

a449027

Unit test coverage

8f3463f

Spelling

5cafe2f

michaelschiff approved these changes Feb 27, 2021

View reviewed changes

Merge branch 'master' into feature/prometheus-metric-exporter

c507452

clintropolis requested changes Feb 27, 2021

View reviewed changes

docs/development/extensions-contrib/prometheus.md Outdated Show resolved Hide resolved

docs/operations/metrics.md Outdated Show resolved Hide resolved

docs/development/extensions-contrib/prometheus.md Outdated Show resolved Hide resolved

Doc fix

42048a5

clintropolis reviewed Mar 2, 2021

View reviewed changes

spelling

807f8dd

clintropolis approved these changes Mar 3, 2021

View reviewed changes

clintropolis added the Release Notes label Mar 3, 2021

clintropolis merged commit a57c28e into apache:master Mar 9, 2021

clintropolis added this to the 0.22.0 milestone Aug 12, 2021

clintropolis mentioned this pull request Sep 3, 2021

[Draft] 0.22.0 Release Notes #11657

Closed

		@@ -0,0 +1,128 @@
		{
		"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer", "conversionFactor": 1000.0, "help": "Seconds taken to complete a query."},

prometheus metric exporter #10412

prometheus metric exporter #10412

Conversation

Tiaaa commented Sep 21, 2020

Key changed/added classes in this PR

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tiaaa Nov 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelschiff Jan 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhassumukh commented Jan 4, 2021

michaelschiff commented Jan 4, 2021

Tiaaa commented Jan 15, 2021

michaelschiff commented Jan 15, 2021 • edited Loading

clintropolis commented Jan 15, 2021

stroeovidiu commented Feb 26, 2021

michaelschiff commented Feb 27, 2021

clintropolis commented Feb 27, 2021

clintropolis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clintropolis left a comment

Choose a reason for hiding this comment

michaelschiff commented Mar 9, 2021

stroeovidiu commented Mar 9, 2021

clintropolis commented Mar 9, 2021

Tiaaa Nov 1, 2020 •

edited

Loading

michaelschiff Jan 15, 2021 •

edited

Loading

michaelschiff commented Jan 15, 2021 •

edited

Loading