Heatmap Charts

[timbunce]

See Brendan Gregg's excellent blog post on Latency Heat Maps that explains how heat maps enable highly-effective visualization of large volumes of time-series data.

See also this ACM journal article containing many examples showing the ability of heat maps to reveal patterns that would be hidden in normal graphs.

I think this would be a significant addition to Kibana for many use-cases, especially those using time-series data.

[elvarb]

Take a look at graphite for this. Check out grafana.org for a very familiar interface.

[timbunce]

Thanks @elvarb but as far as I know neither graphite (which I use often) or grafana (which we plan to use soon) offer heat-maps.

Even if they did, the data I want to display is in elasticsearch and kibana is the ideal way to present it. Supporting a heat-map chart style would be a natural addition to the existing histogram panel.

[elvarb]

I'm using grafana against influxdb and by using three queries I can get some sort of a heat map. Each query the same except one uses max, other mean and the last min.

The way smoke ping works would be a goal maybe?

[timbunce]

While I appreciate the suggestion @elvarb, what you describe does not sound at all like a heat map. For example, it wouldn't show 'hot spots'. A heat map requires the color/density of the pixels to vary according to the number of value that fall into that slot. The heat map shown here shows rich information that would be mostly hidden using a simple min/median/max visualization. And similarly for the information in these heat maps.

[elvarb]

I agree, not perfect by far but in the direction. Another way I have been using to simulate a heatmap is using a point graph.

It should be possible to modify the point settings to add opacity of the points and then displaying all points in the time series. When the points overlap a 10% opacity per point would quickly become more visible.

[timbunce]

A point graph that allows the point size and opacity to be set would be workable, yes.
I think kibana currently allows the point size to be set but not the opacity, right?

So, incremental steps towards a heat map:

• Support setting the point opacity - at this point there's something usable.
• Support allowing the y-axis and/or x-axis values to be 'bucketed' to some specified granularity. This would make the points sit on top of one another, producing a much cleaner visualisation.
• Support using squares/rectangles instead of circles for the point size.
• Automatically set the square/rectangle dimensions to the size of the bucket granularity.

Those last three could be combined into a 'heat map' style that would also enable opacity if selected.

[LukasHavemann]

Hello,
as part of a students project we have developed a Heatmap Panel for Kibana.

Below you can see the activity of users on a server:

And we implemented a zoomable and pannable 3d-View:

We are planning to publish the code next month. Maybe this could be the foundation to a more generic and better heatmap.

Regards,
Lukas

[timbunce]

Interesting. Thanks @LukasHavemann.

[LukasHavemann]

We published the Code: https://github.com/ppi-ag/kibana-heatmap

[elvarb]

Nice work. Why does it require a special timestamp field instead of using a normal timestamp field?

[LukasHavemann]

The special timestamp field is a byproduct of a other more sophisticated feature which we implemented in our stack. At that time I was not so familiar with the Elasticsearch API so I used this special field and the terms panel as foundation for the heatmap panel to get the job quickly done. I think it should be possible to write a query with the Aggregations API which will do the same thing.

[pemontto]

This look brilliant, thanks Lukas! Do you have an example of the using it with the elasticsearch script functionality?

[LukasHavemann]

@yogi183 Sorry not really. I tried some different things, but as I said I am not that familiar with the Elasticsearch API.
In my mind Elasticsearch is not able to determine the weekday or the hour of a timestamp. To solve that I basically wrapped this Java Code in a Native (Java) Script Plugin for Elasticsearch. Of course you need some more lines to write a Native Script Plugin but there is a good tutorial here. I did not use dynamic scripting because for the sake of performance and security.

With that I was able to make request like this:

curl -XPOST " http://localhost:9200/example/doc/_search" -d'
{
    "query" : {
      "match_all" : { } 
    }, 
   "script_fields": {
      "weekday": {
         "script": "timeslot-extractor", 
         "params": { 
            "field": "timestamp"
         },
         "lang": "native"
      }
   }
}'

This query returns a additional scripted field for the matched documents. But if my memories are correct the problem was that elasticsearch is not capable of aggregating on scripted fields or something like that. So I gave up and added the additional timestamp field during the import.

[LukasHavemann]

@rashidkpc Are there any plans to implement a heatmap or a scatter plot panel in Kibana? #928 #978

[rashidkpc]

Yep, its on the roadmap. We have a working prototype, but we're refactoring how things work to play better with aggregations in a more flexible and composable way

[pjcard]

Hello, is it possible to get an idea of where on the roadmap it is? (weeks/months/years). It seems an slightly odd omission for a package specialising in visualising big data.

[pemontto]

+1

[pjcard]

I think I understand the reason now - kibana 3 uses facets whereas
presumably the next kibana will use aggregations? Anyway I've found the elk
stack very useful so far so thanks all.
On 8 Aug 2014 17:03, "yogi183" notifications@github.com wrote:

+1

—
Reply to this email directly or view it on GitHub
#1265 (comment)
.

[stormpython]

A few notes as this is a work in progress. The current functionality for the 2D Heatmap is such:

1. A user will have the ability to choose an Intensity, i.e. a metric for the color scale (e.g. Count of Documents, Average, Min, Max etc).
2. A user will have the ability to choose an aggregation type (e.g. Date Histogram, Histogram, Range, Terms, etc) for data displayed in Columns.
3. A user will have the ability to choose an aggregation type (e.g. Date Histogram, Histogram, Range, Terms, etc) for data displayed in Rows.
4. The default display is a count of _all documents, which renders 1 block.
   
5. As a user creates aggregations for either a single column or row, each block or rect fills the space of the containing DOM element. This means at times, you can have rather elongated rectangles instead of nice squares.
   
6. However, as the final dimension is put in, the squares more or less even out depending on the query and the data.
   
7. A user will also have the option to choose an aggregation type to split the charts by, allowing anyone to create multiple Heatmaps in 1 visualization (as is the default behavior for other charts in Kibana).
   

Major things left to accomplish:

1. Getting Labels Right!
   • Each Heatmap visualization should have a Columns title (e.g. Histogram: bytes) and a Rows title (e.g. Terms: geo.src).
   • Labels for the Columns and Rows blocks or rects should work for small datasets as well as large datasets. However with large datasets, the problem is that there is not always enough space to render a label.
     For example:
     
   • Two solutions have been proposed:
     • Filter the labels depending on the number of rows and columns
     • Remove labels when there are too many rows and/or columns and allow the user to use the
       Columns title, Rows title, and tooltip to get their bearings.
   • Labels need to also work in the case when multiple charts are created in 1 visualization
2. Adding the Legend
   • A legend that breaks down the color range into n intensity buckets.

@jthomassie feel free to add to this list.

[sahilthapar]

👍

[pjcard]

Very nice, looking good.

On 20 November 2014 04:42, Sahil Thapar notifications@github.com wrote:

[image: 👍]

—
Reply to this email directly or view it on GitHub
#1265 (comment)
.

[timbunce]

Thanks for the update @stormpython. It looks great!

[jthomassie]

• We should change sort for some agg results, like date histogram (below). ES returns data sorted by bucket count.
• Should add user config to select quantile scale or quantize scale and number of colors in scale range.

quantile:

quantize:

[ferranmunoz]

+1

[naisanza]

+1

[yehosef]

@rashidkpc http://bl.ocks.org/mbostock/3074470 is an example using d3 and canvas. Perhaps this approach would work here. Even though in general you're using SVG, it's really just because you're using d3 and it uses SVG by default, no? But even this approach is only needed for very dense heat maps - most traditional heat maps should work fine with SVG it seems (eg https://github.com/rstudio/d3heatmap, http://bl.ocks.org/tjdecke/5558084)

I think for most of the applications for which people need heatmaps, svg would be fine. If needed, you could put a limit on the number of data points returned like there are in other visualizations.

[markwalkom]

Just wanted to comment that we've had a few users we've talked to ask for scatter plot graphs :)

[Pigueiras]

@stormpython @jthomassie Do you have your prototype available somewhere? If not, could you publish the patch so we can take a look at it, please?

[rashidkpc]

Currently there's no prototype, you'd need to start from scratch.

[sebastianstolzenberg]

+1

[tarunsapra]

+1

[Pigueiras]

We are trying to make a prototype of this for our specific usecase. What we want is to show a square with a color associated the most/least common value. In Kibana 3, we installed the Heatmap plugin that allows us to fulfill our requirements. Something like:

As this was removed from the roadmap, we are trying to work on this, as this is a blocker requirement for us in order to move from K3 to K4. I have to tell that I didn't find easy at all how to add new visualizations, as you explicitly ask us not to develop any custom plugin and it's not documented because at some point you may change this (#1440 ?)

My original idea was to have something like this in left panel to select metrics to build the heatmap.

For documents like:

{
  "service": "service1",
  "timestamp" "22H",
  "field": "value1"
}

There would be a "metrics" section, where you can select what field will be represented as a square and another dropdown where you can select if you want to show the most/least repeated one. The rows/columns buckets, will allow the user to select what values should appear in the X/Y axis and what type of aggregation should be performed. This would just be perfect for us, but to make it more generic, if the metric is a number I thought to have something like:

Where you can select the aggregation metric for the field (min, max, mean...) and define some thresholds. So depending on the value of each square, it will show a different color if it's in one range or another.

What we developed until now is this fork of Kibana 4.1.1 (branch: heatmap). It's something that it's still on development (not very usable at the moment). Here a screenshot of what we have right now:

Some things that need improvement:

• The left panel is not as I planned at the beginning, because I didn't find any easy way to modify it. Instead of that, I copied it from the PieChart and use it in the Heatmap. The query that the PieChart is doing is not exactly the query that I would like to do, but for a prototype does the job.
• No tooltips on the squares.
• Legend and heatmap are not synchronized.
• Rendering problems where there is not enough space in the axis

We were wondering if you would be interested in giving some feedback/ideas to us in case you might want to include this as a visualization in K4.

[stormpython]

@Pigueiras
Yes, the process for creating visualizations is currently not straightforward, please see my comments in #3456. We are working on changing that process currently, so while the heatmap is off the roadmap, we will be bringing it back in the future once our new visualization library is complete.

I can't offer specific feedback on how to do this since:

1. It would be a really involved discussion
2. The process for creating visualizations will change

I can say however that the pie chart query is not ideal for heatmaps. It would be better to use our point_series (bar, area, line) chart queries instead. To customize the left panel, you'd need to add new angular directives to get the functionality that you are looking for. Tooltip and Legend are separate modules that need to be hooked in to a visualization to have them synchronized. For examples on how this is done, you can look at some the of chart visualization code and the handler.js file (for the legend). Finally, overlapping labels is an unavoidable problem. One solution would be to filter the labels when there is not enough space to show all of them. Or you could try constraint relaxing or using a force layout to prevent labels from overlapping.

[sumitchawla]

+1

[marcinkubica]

+1

[acb122]

+1

[mjpowersjr]

+1

[stormpython]

I have broken this out into a plugin.

Please note: This plugin is still in alpha. There is a list of items that need to be completed before this is a fully functioning plugin. You are free to test it out in its current state, but be warned that several things are still broken. I plan to check these items off the list within the next several weeks so that I can get this out for beta testing.

Once the plugin is in beta, I plan on closing this issue and redirecting all issues and feature request to the plugin repo.

[Crazyigor1987]

+1

@stormpython
You sir, made my day!
Thank you so much for your effort!

[tereka114]

+1