Add support for reading prometheus endpoint #20

yaacov · 2017-05-21T16:25:15Z

Description
Add support for reading Prometheus endpoint

If prometheus endpoint exist, read metrics from prometeus

yaacov · 2017-05-21T16:26:02Z

@miq-bot add_label compute/containers, wip

cc// @simon3z @cben @moolitayer @zeari

miq-bot · 2017-05-21T16:26:32Z

@yaacov Cannot apply the following label because they are not recognized: compute/containers

simon3z · 2017-05-22T09:28:16Z

@yaacov let's split this in 3 parts (2 of which we can actually merge):

evaluate if we can replace the complex computation of cpu_usage_rate_average(based on the cpu time and the number of cores) with the machine/x/node_utilization
extract the framework about collection that you did here (to generalize hawkular/prometheus)
prometheus implementation (as WIP) leveraging the refactor/framework of point 2

yaacov · 2017-05-28T11:40:15Z

@yaacov let's split this in 3 parts (2 of which we can actually merge):

evaluate if we can replace the complex computation of cpu_usage_rate_average(based on the cpu time and the number of cores) with the machine/x/node_utilization

extract the framework about collection that you did here (to generalize hawkular/prometheus)

prometheus implementation (as WIP) leveraging the refactor/framework of point 2

@simon3z @moolitayer

extract the framework about collection that you did here (to generalize hawkular/prometheus)

Moved to #25

evaluate if we can replace the complex computation of cpu_usage_rate_average(based on the cpu time and the number of cores) with the machine/x/node_utilization

Implementation depend on #25.

simon3z · 2017-05-30T14:42:25Z

evaluate if we can replace the complex computation of cpu_usage_rate_average(based on the cpu time and the number of cores) with the machine/x/node_utilization

Implementation depend on #25.

@yaacov I'd rather not have the new metrics depending on a refactor if possible. I want to backport them in the cleanest way possible.

miq-bot · 2017-06-08T10:53:57Z

This pull request is not mergeable. Please rebase and repush.

miq-bot · 2017-07-05T09:13:59Z

This pull request is not mergeable. Please rebase and repush.

simon3z · 2017-07-11T15:40:10Z

@yaacov should we add the prometheus spec tests as well?

yaacov · 2017-07-11T15:41:03Z

should we add the prometheus spec tests as well?

@simon3z 👍 I will

miq-bot · 2017-07-12T10:39:00Z

Some comments on commit yaacov@afd4bb8

spec/models/manageiq/providers/kubernetes/container_manager/metrics_capture_spec.rb

⚠️ - 131 - Detected allow_any_instance_of. This RSpec method is highly discouraged, please only use when absolutely necessary.
⚠️ - 138 - Detected allow_any_instance_of. This RSpec method is highly discouraged, please only use when absolutely necessary.
⚠️ - 153 - Detected allow_any_instance_of. This RSpec method is highly discouraged, please only use when absolutely necessary.
⚠️ - 160 - Detected allow_any_instance_of. This RSpec method is highly discouraged, please only use when absolutely necessary.

miq-bot · 2017-07-12T10:39:06Z

Checked commit yaacov@afd4bb8 with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0
9 files checked, 0 offenses detected
Everything looks fine. ⭐

yaacov · 2017-07-12T10:42:07Z

@yaacov should we add the prometheus spec tests as well?

@simon3z 👍 specs added

a. specs found error in pod collection, fixed [ labels changed current prometheus deployment ]
b. update error handling to work with both pre 4.0.0 and post 4.0.0. versions of hawkular client [56]

moolitayer · 2017-07-12T14:22:16Z

app/models/manageiq/providers/kubernetes/container_manager/metrics_capture.rb

@@ -57,7 +58,11 @@ def perf_collect_metrics(interval_name, start_time = nil, end_time = nil)
                "[#{start_time}] [#{end_time}]")

      begin
-        context = CaptureContext.new(target, start_time, end_time, INTERVAL)
+        context = if ems && ems.connection_configurations.prometheus.try(:endpoint)


you defined a method called prometheus_endpoint. Can you use that?

no we can not use it here :-(
it's a chicken and egg problem, to use it we need a context, but if we do not have a context we can't use it :-(

So move it to the manager and the capture context

moving functionality around just for the sake of moving it is pointless, is their a reason for moving it ?

you clearly need this function in a wider scope then the capture context (for example to determine which context to create) then you won't have to copy it in different places.

(for example to determine which context to create)

you give this case as an example for wider scope ?

moolitayer

There is something which I don't get, maybe you could explain. metrics collection is supposed to be disabled unless we have an endpoint named hawkular due to https://github.com/ManageIQ/manageiq-providers-kubernetes/blob/master/app/models/manageiq/providers/kubernetes/container_manager/metrics_collector_worker.rb#L18
This change is hard to review since it adds new functionality and also renames a bunch of files. Please split it to ease reviewers work - one for refactor related stuff (moving stuff around, renaming etc) and one for new functionality

Thanks!

yaacov · 2017-07-12T14:49:59Z

Please split it to ease reviewers work - one for refactor related stuff (moving stuff around, renaming etc) and one for new functionality

We already tried that [1] I do not think we should do that again.

[1] #25

yaacov · 2017-07-12T14:51:53Z

metrics collection is supposed to be disabled unless we have an endpoint named hawkular

than we should fix the ems.supports_metrics? [1] function, we can do that in a new PR.

[1] https://github.com/ManageIQ/manageiq-providers-kubernetes/blob/master/app/models/manageiq/providers/kubernetes/container_manager_mixin.rb#L18

moolitayer · 2017-07-12T15:10:28Z

We already tried that [1] I do not think we should do that again.

[1] #25

I'm not asking that to create extra work for you.
Reviewing many changes takes a lot of time (and I have my own assignments on top of that).
If the changes are clearly separated so that I don't have to figure out what is being copied and what is new functionality that is a lot easier. I promise you that with smaller changes we will get better code quality.

yaacov · 2017-07-12T15:27:01Z

I'm not asking that to create extra work for you.

@simon3z @moolitayer

I did not think so :-) it's a matter of time usage and priorities -

we have two other PR's that wait for this one to be merged [1] [2] they are also high priority.
this PR depend on Prometheus version and the way Promethean is deployed, for example the pod labelling changed in the last month ( because of new version or relabeling [3] ) . We will have a lot of bugs on this new functionalty. we can not wait for the prometheus deployment to be finalized.

New functionality, especially of this size, will come with bugs, and we will fix them when we find them -

first will be fixing the [4] supports_metrics? bug :-)

[1] ManageIQ/manageiq-ui-classic#1677
[2] #47
[3] https://github.com/zgalor/scripts/blob/master/openshift/prometheus_template.yaml#L238
[4] https://github.com/ManageIQ/manageiq-providers-kubernetes/blob/master/app/models/manageiq/providers/kubernetes/container_manager_mixin.rb#L18

yaacov · 2017-07-13T03:29:26Z

I promise you that with smaller changes we will get better code quality.

Yes, but:

https://xkcd.com/844/

yaacov · 2017-07-13T06:53:55Z

#59 fixes the supports_metrics? bug

moolitayer · 2017-07-13T07:30:00Z

@yaacov I've asked to move a few files to a different PR so that review is more effective - I do it all the time. I will not approve this pull request as it is.

yaacov · 2017-07-13T08:47:38Z

I will not approve this pull request as it is.

Ok :-)

yaacov · 2017-07-13T14:39:23Z

@simon3z as you requested in todays meeting, this PR is closed in favour of #62, since the Prometheus part depend on #62 being merged, I will post it once #62 is merged.

miq-bot added the wip label May 21, 2017

yaacov force-pushed the read-prometheus-metrics-endpoint branch 9 times, most recently from 9e3e6d7 to 849ade5 Compare May 21, 2017 18:36

miq-bot added the unmergeable label Jun 8, 2017

yaacov force-pushed the read-prometheus-metrics-endpoint branch from 849ade5 to bc1f8a6 Compare June 11, 2017 15:13

miq-bot removed the unmergeable label Jun 11, 2017

yaacov force-pushed the read-prometheus-metrics-endpoint branch 2 times, most recently from ac2367f to b0b08f7 Compare June 11, 2017 15:26

yaacov mentioned this pull request Jun 11, 2017

[WIP][POC]Demo jun 11 #37

Closed

yaacov force-pushed the read-prometheus-metrics-endpoint branch 2 times, most recently from 8791308 to b3b8ced Compare June 15, 2017 12:10

This was referenced Jun 18, 2017

Add support for monitoring selection in UI ManageIQ/manageiq-ui-classic#1501

Merged

[WIP] Abstract the metric capture code #25

Closed

zgalor mentioned this pull request Jun 26, 2017

[WIP][POC]: Add support for reading prometheus endpoint #51

Closed

miq-bot added the unmergeable label Jul 5, 2017

yaacov force-pushed the read-prometheus-metrics-endpoint branch from b3b8ced to 470b865 Compare July 6, 2017 11:02

yaacov force-pushed the read-prometheus-metrics-endpoint branch 4 times, most recently from d2f0a86 to d9e1450 Compare July 11, 2017 14:48

yaacov mentioned this pull request Jul 12, 2017

Add prometheus view to ad hoc metrics ManageIQ/manageiq-ui-classic#1677

Merged

yaacov force-pushed the read-prometheus-metrics-endpoint branch 2 times, most recently from 8a023e9 to c17ccbc Compare July 12, 2017 10:31

Add support for reading prometheus endpoint

afd4bb8

yaacov force-pushed the read-prometheus-metrics-endpoint branch from c17ccbc to afd4bb8 Compare July 12, 2017 10:34

simon3z requested review from cben and moolitayer July 12, 2017 13:51

moolitayer reviewed Jul 12, 2017

View reviewed changes

moolitayer suggested changes Jul 12, 2017

View reviewed changes

moolitayer mentioned this pull request Jul 13, 2017

Fix support_metrics function to include prometheus #59

Merged

yaacov closed this Jul 13, 2017

moolitayer mentioned this pull request Jul 13, 2017

Enable metric collection from arbitrary sources #62

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for reading prometheus endpoint #20

Add support for reading prometheus endpoint #20

yaacov commented May 21, 2017

yaacov commented May 21, 2017

miq-bot commented May 21, 2017

simon3z commented May 22, 2017

yaacov commented May 28, 2017

simon3z commented May 30, 2017

miq-bot commented Jun 8, 2017

miq-bot commented Jul 5, 2017

simon3z commented Jul 11, 2017

yaacov commented Jul 11, 2017

miq-bot commented Jul 12, 2017

miq-bot commented Jul 12, 2017

yaacov commented Jul 12, 2017

moolitayer Jul 12, 2017

yaacov Jul 12, 2017

moolitayer Jul 12, 2017

yaacov Jul 12, 2017

moolitayer Jul 12, 2017

yaacov Jul 12, 2017

moolitayer left a comment •

edited

yaacov commented Jul 12, 2017

yaacov commented Jul 12, 2017 •

edited

moolitayer commented Jul 12, 2017

yaacov commented Jul 12, 2017

yaacov commented Jul 13, 2017 •

edited

yaacov commented Jul 13, 2017

moolitayer commented Jul 13, 2017

yaacov commented Jul 13, 2017

yaacov commented Jul 13, 2017

Add support for reading prometheus endpoint #20

Add support for reading prometheus endpoint #20

Conversation

yaacov commented May 21, 2017

yaacov commented May 21, 2017

miq-bot commented May 21, 2017

simon3z commented May 22, 2017

yaacov commented May 28, 2017

simon3z commented May 30, 2017

miq-bot commented Jun 8, 2017

miq-bot commented Jul 5, 2017

simon3z commented Jul 11, 2017

yaacov commented Jul 11, 2017

miq-bot commented Jul 12, 2017

miq-bot commented Jul 12, 2017

yaacov commented Jul 12, 2017

moolitayer Jul 12, 2017

Choose a reason for hiding this comment

yaacov Jul 12, 2017

Choose a reason for hiding this comment

moolitayer Jul 12, 2017

Choose a reason for hiding this comment

yaacov Jul 12, 2017

Choose a reason for hiding this comment

moolitayer Jul 12, 2017

Choose a reason for hiding this comment

yaacov Jul 12, 2017

Choose a reason for hiding this comment

moolitayer left a comment • edited

Choose a reason for hiding this comment

yaacov commented Jul 12, 2017

yaacov commented Jul 12, 2017 • edited

moolitayer commented Jul 12, 2017

yaacov commented Jul 12, 2017

yaacov commented Jul 13, 2017 • edited

yaacov commented Jul 13, 2017

moolitayer commented Jul 13, 2017

yaacov commented Jul 13, 2017

yaacov commented Jul 13, 2017

moolitayer left a comment •

edited

yaacov commented Jul 12, 2017 •

edited

yaacov commented Jul 13, 2017 •

edited