New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Prometheus histograms in the heatmap panel #10009

Closed
bergquist opened this Issue Nov 28, 2017 · 21 comments

Comments

Projects
None yet
8 participants
@bergquist
Contributor

bergquist commented Nov 28, 2017

There are three things that need to be solved to support Prometheus histograms in the heatmap panel.

Transform the histogram result into the format expected by the heatmap panel

Prometheus stores the values of each bucket incrementally. So we have to decrement each bucket in the Prometheus data source before sending it to the heatmap panel. I started working on that in a branch but it's out of sync with master. The transform function can be found at https://github.com/bergquist/grafana/blob/prom_heatmap/public/app/plugins/datasource/prometheus/result_transformer.ts#L21 and one test for the function can be found at https://github.com/bergquist/grafana/blob/prom_heatmap/public/app/plugins/datasource/prometheus/specs/result_transformer.ts#L17 Those can be used to get started quickly.

We need to handle Prometheus +Inf serie.

In Prometheus, each histogram contains a serie named +Inf which is used whenever a value does not find in any other predefined bucket. This serie can help you decide if you should are more buckets or not. I'm not sure how to visualize this serie, if at all. If we decide to not show it we should remove it from the response when transforming the result into heat map format.

Exponential buckets

Using the helper method in the Prometheus client libs you usually choose between Linear and exponential buckets. Would be great if the heat map panel could support this as well log base scales.

@bergquist bergquist added this to Ready For Implementation in Issue preparations Dec 18, 2017

@torkelo torkelo added this to the 5.0.0-alpha2 milestone Dec 21, 2017

@alexanderzobnin alexanderzobnin self-assigned this Dec 28, 2017

@alexanderzobnin

This comment has been minimized.

Show comment
Hide comment
@alexanderzobnin

alexanderzobnin Dec 28, 2017

Contributor

@bergquist I'm continuing work on this, rebased onto master (https://github.com/alexanderzobnin/grafana/tree/prom-heatmap)

Contributor

alexanderzobnin commented Dec 28, 2017

@bergquist I'm continuing work on this, rebased onto master (https://github.com/alexanderzobnin/grafana/tree/prom-heatmap)

@torkelo torkelo modified the milestones: 5.0.0-alpha2, 5.0 Jan 8, 2018

@torkelo torkelo removed this from the 5.0 milestone Jan 17, 2018

@discordianfish

This comment has been minimized.

Show comment
Hide comment
@discordianfish

discordianfish Jan 19, 2018

Contributor

That'd be awesome. Feel free to ping me whenever you need help/input from the prometheus side of things!

Contributor

discordianfish commented Jan 19, 2018

That'd be awesome. Feel free to ping me whenever you need help/input from the prometheus side of things!

@torkelo torkelo added this to the 5.1 milestone Feb 21, 2018

@torkelo torkelo removed this from Ready For Implementation in Issue preparations Feb 21, 2018

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Feb 28, 2018

Alexander, we're all praying for you.

neopaf commented Feb 28, 2018

Alexander, we're all praying for you.

@bergquist

This comment has been minimized.

Show comment
Hide comment
@bergquist

bergquist Mar 13, 2018

Contributor

Closed by #11087 \o/

Contributor

bergquist commented Mar 13, 2018

Closed by #11087 \o/

@bergquist bergquist closed this Mar 13, 2018

bergquist added a commit that referenced this issue Mar 13, 2018

@bergquist

This comment has been minimized.

Show comment
Hide comment
@bergquist

bergquist Mar 22, 2018

Contributor

This feature needs better docs.

Contributor

bergquist commented Mar 22, 2018

This feature needs better docs.

@alexanderzobnin

This comment has been minimized.

Show comment
Hide comment
@alexanderzobnin

alexanderzobnin Mar 22, 2018

Contributor

I agree.

Contributor

alexanderzobnin commented Mar 22, 2018

I agree.

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Mar 26, 2018

Alexander, friends, I could not figure out what I got.
Probably I am missing something simple.
But my pre-bucketed Prometheus metrics do not show up as I expect them to.
My dream is to see on Y-axis the milliseconds it took to process something.
I have pre-bucketed data and I see them in Prometheus (following a link in Graphana):
2018-03-26 15 54 42
I expect one graph with heatmap color showing number of hits (popup shows hits, but e+24 is very strange).

I see with latest night build for Centos (5.1.0-13147pre1):
2018-03-26 15 55 34
separate bands per bucket.

docs
http://docs.grafana.org/features/panels/heatmap/

hint at group by, but I couldn't find that group by in Heatmap graph.
Please give a hint?

neopaf commented Mar 26, 2018

Alexander, friends, I could not figure out what I got.
Probably I am missing something simple.
But my pre-bucketed Prometheus metrics do not show up as I expect them to.
My dream is to see on Y-axis the milliseconds it took to process something.
I have pre-bucketed data and I see them in Prometheus (following a link in Graphana):
2018-03-26 15 54 42
I expect one graph with heatmap color showing number of hits (popup shows hits, but e+24 is very strange).

I see with latest night build for Centos (5.1.0-13147pre1):
2018-03-26 15 55 34
separate bands per bucket.

docs
http://docs.grafana.org/features/panels/heatmap/

hint at group by, but I couldn't find that group by in Heatmap graph.
Please give a hint?

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Mar 26, 2018

If important, other tabs:
2018-03-26 15 55 08

2018-03-26 15 55 13

neopaf commented Mar 26, 2018

If important, other tabs:
2018-03-26 15 55 08

2018-03-26 15 55 13

@F30

This comment has been minimized.

Show comment
Hide comment
@F30

F30 Mar 26, 2018

@neopaf What version of Grafana are you using? This feature just got merged 13 days ago, afaik it is not in a released version yet.

F30 commented Mar 26, 2018

@neopaf What version of Grafana are you using? This feature just got merged 13 days ago, afaik it is not in a released version yet.

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Mar 26, 2018

@F30 5.1.0-13147pre1 (from Nightly builds).
I expected it to have the master merge (not sure if my assumptions are on solid ground)

neopaf commented Mar 26, 2018

@F30 5.1.0-13147pre1 (from Nightly builds).
I expected it to have the master merge (not sure if my assumptions are on solid ground)

@torkelo

This comment has been minimized.

Show comment
Hide comment
@torkelo

torkelo Mar 26, 2018

Member

Think you need to use the quantile prometheus function

Member

torkelo commented Mar 26, 2018

Think you need to use the quantile prometheus function

@discordianfish

This comment has been minimized.

Show comment
Hide comment
@discordianfish

discordianfish Mar 26, 2018

Contributor

Also don't confuse summary with histograms. I haven't looked closer but grafana probably only supports histograms and what you have looks like a summary: https://prometheus.io/docs/practices/histograms/

Contributor

discordianfish commented Mar 26, 2018

Also don't confuse summary with histograms. I haven't looked closer but grafana probably only supports histograms and what you have looks like a summary: https://prometheus.io/docs/practices/histograms/

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Mar 26, 2018

@torkelo @discordianfish
Friends, I feel quantile/histograms are not related to my original understanding of this task.

On client side I observe events straight forwardly:

	private val histogram = Histogram.build.exponentialBuckets(0.0001, 10, 5).name("PMF").labelNames("entry", "stage", "status").help("In seconds.").register

	def observe(entry: String, stage: String, status: String, timer: SimpleTimer): Unit = {
		histogram.labels(entry, stage, status).observe(timer.elapsedSeconds)

And I get direct pre-bucketed values that need no estimates.
I thought, current task is designed to visualise precisely that information.

Anybody feels this change is in available nightly builds?

neopaf commented Mar 26, 2018

@torkelo @discordianfish
Friends, I feel quantile/histograms are not related to my original understanding of this task.

On client side I observe events straight forwardly:

	private val histogram = Histogram.build.exponentialBuckets(0.0001, 10, 5).name("PMF").labelNames("entry", "stage", "status").help("In seconds.").register

	def observe(entry: String, stage: String, status: String, timer: SimpleTimer): Unit = {
		histogram.labels(entry, stage, status).observe(timer.elapsedSeconds)

And I get direct pre-bucketed values that need no estimates.
I thought, current task is designed to visualise precisely that information.

Anybody feels this change is in available nightly builds?

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Mar 26, 2018

But I did try this:

histogram_quantile(0.95, sum(rate(PMF_bucket{entry="POST /gc/metaProfile", stage="mergeTwoServiceCubes", job="pmf", status="success"}[1m])) by (le))

And got in Prometheus:
2018-03-26 17 01 12
(I feel that information is already lost on this step)

In Grafana:
2018-03-26 17 01 36

With strange bucket names (not exponential, not as created originally).

I truly don't understand what's going on with this function.
But my original understanding is that it is not needed.
My data is already pre-bucketed.

neopaf commented Mar 26, 2018

But I did try this:

histogram_quantile(0.95, sum(rate(PMF_bucket{entry="POST /gc/metaProfile", stage="mergeTwoServiceCubes", job="pmf", status="success"}[1m])) by (le))

And got in Prometheus:
2018-03-26 17 01 12
(I feel that information is already lost on this step)

In Grafana:
2018-03-26 17 01 36

With strange bucket names (not exponential, not as created originally).

I truly don't understand what's going on with this function.
But my original understanding is that it is not needed.
My data is already pre-bucketed.

@oseiberts11

This comment has been minimized.

Show comment
Hide comment
@oseiberts11

oseiberts11 Mar 26, 2018

@neopaf: I don't think you should use the histogram_quantile function here. That function really throws away a lot of data from the histogram; the outcome is definitely no longer a histogram so trying to show it as one would indeed show nonsense.

oseiberts11 commented Mar 26, 2018

@neopaf: I don't think you should use the histogram_quantile function here. That function really throws away a lot of data from the histogram; the outcome is definitely no longer a histogram so trying to show it as one would indeed show nonsense.

@bergquist

This comment has been minimized.

Show comment
Hide comment
@bergquist

bergquist Mar 27, 2018

Contributor

As said earlier, avoid the histrogram_quantile function for visualization. Instead use something like this:
sum(rate(PMF_bucket{entry="POST /gc/metaProfile", stage="mergeTwoServiceCubes", job="pmf", status="success"}[1m])) by (le)

and set Legend format to {{le}}

Contributor

bergquist commented Mar 27, 2018

As said earlier, avoid the histrogram_quantile function for visualization. Instead use something like this:
sum(rate(PMF_bucket{entry="POST /gc/metaProfile", stage="mergeTwoServiceCubes", job="pmf", status="success"}[1m])) by (le)

and set Legend format to {{le}}

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Mar 27, 2018

Thanks, Carl.
Thanks, Alexander for your efforts and latest commit to docs alexanderzobnin@b97cede

It works for me:

2018-03-27 10 57 52

JSON of panel, for history:
PAF heat map experiment-1522137445922.json.txt

Prometheus view:
2018-03-27 10 58 09

I'm locally-happy :)
Thanks again!

neopaf commented Mar 27, 2018

Thanks, Carl.
Thanks, Alexander for your efforts and latest commit to docs alexanderzobnin@b97cede

It works for me:

2018-03-27 10 57 52

JSON of panel, for history:
PAF heat map experiment-1522137445922.json.txt

Prometheus view:
2018-03-27 10 58 09

I'm locally-happy :)
Thanks again!

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Mar 27, 2018

I feel key change that made it all work for me was "format as Heatmap".
Maybe that should be a default when creating a Heatmap panel?

neopaf commented Mar 27, 2018

I feel key change that made it all work for me was "format as Heatmap".
Maybe that should be a default when creating a Heatmap panel?

@marefr

This comment has been minimized.

Show comment
Hide comment
@marefr

marefr Mar 27, 2018

Member

Docs added in #11383. Closing this

Member

marefr commented Mar 27, 2018

Docs added in #11383. Closing this

@marefr marefr closed this Mar 27, 2018

@alexanderzobnin

This comment has been minimized.

Show comment
Hide comment
@alexanderzobnin

alexanderzobnin Mar 28, 2018

Contributor

@neopaf it's a default format for heatmap :) not sure problem was in the format because it only converts histogram buckets from cumulative to regular. I think it's due to wrong label format, but not sure.

Contributor

alexanderzobnin commented Mar 28, 2018

@neopaf it's a default format for heatmap :) not sure problem was in the format because it only converts histogram buckets from cumulative to regular. I think it's due to wrong label format, but not sure.

@neopaf

This comment has been minimized.

Show comment
Hide comment
@neopaf

neopaf Apr 2, 2018

@alexanderzobnin could be. I'm super happy. Thanks again!
(большое спасибо!)

neopaf commented Apr 2, 2018

@alexanderzobnin could be. I'm super happy. Thanks again!
(большое спасибо!)

marefr added a commit that referenced this issue Apr 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment