Heatmap no longer displays empty cell for missing values

[Nezz]

Kibana version:
8.2.2 (Elastic Cloud)

The bug was not present in 8.1.3 that we upgraded from. This is a regression introduced in 8.2.

Elasticsearch version:
8.2.2 (Elastic Cloud)

Server OS version:
N/A

Browser version:
Firefox 101.0, Chrome 102.2

Browser OS version:
macOS 12.3.1

Original install method (e.g. download page, yum, from source, etc.):
Elastic Cloud

Describe the bug:
Heat map used to display an empty white cell when a value was missing. Now it displays a color corresponding to the minimum value.
This bug accidentally fulfills this feature request:
#42661

Sadly I did not find anything relevant in the changelog.

Steps to reproduce:

1. Create a heat map
2. Set the cell value to the average of some number

Expected behavior:
The cell without a value should not be colored.

Screenshots (if relevant):
Only cells with values should be colored, such as this one:

Notice that the tooltip shows "0%".

Now cells without values are colored too:

Notice that the tooltip does not show any value.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[flash1293]

@markov00 Can you look into this? I'm not sure whether this the expected behavior of heatmap - it seems something has changed and now with a band which includes 0 it will also color null values in the same way (which wasn't the case in 8.1)

@Nezz as a workaround you can turn off "Show empty rows" which should show the gaps again:

Take this vis which is similar to the one you are showing:

By flipping off the "show empty rows" for x and y axis:

You get empty cells with no data called out again with the background color:

[markov00]

I can confirm that this is a bug of the current heatmap behavior when mapping nulls values. It maps them as 0, which is actually an unwanted behavior.

The only difference that I see is that this behavior was the default one since, at least, elastic-charts v39.0.0 (I haven't checked earlier because we used a different scale. But the fact is that:
Kibana 8.0 is using 40.3.0 and Kibana 8.1 is using 43.1.1

and both these versions behave the same (you can double check by changing the version in the code sandbox example: https://codesandbox.io/s/focused-cloud-7ozfme?file=/src/App.tsx the null cell will be always colored with blue.

[flash1293]

@markov00 Ah, you are correct, we changed something on the Kibana side - previously we were filtering out data rows with null before passing them to the chart, but we stopped that behavior (to be able to signal to the heatmap chart the "full matrix" of values so it can render appropriately). Do you think it makes sense to change this behavior on elastic-charts side?

[flash1293]

I thought a bit about it and I don't think we can cleanly fix it on the Kibana side - if we starting filtering out the null rows again, then the order of the y axis won't be stable anymore again (that's why we did that change in the first place) - the null rows contain relevant information.

[markov00]

@flash1293 we can definitely fix it on our side, but we should decide what we wanna do with those null cells? do you prefer us to filter them completely from the view (no text, no tooltip) or do you prefer to leave this choice to the user: show a visual clue like - to display a missing value or hide completely any text/tooltip for missing ones?

[flash1293]

show a visual clue like - to display a missing value or hide completely any text/tooltip for missing ones?

Hiding it completely seems like the right approach to me, otherwise there are three cases - "no row", null and 0 which doesn't seem helpful to me.

[markov00]

Ok, no problem in hiding it by default.

Just a clarification: actually, the cases are:

1. NaN if the metric, on that coordinate, doesn't produce a valid value (for example: when the intersection between the x coordinate and the y coordinate doesn't have documents, or the number of documents is not sufficient to compute the metric correctly - see windowed functions-, or the metric calculates the log of 0 or a zero denominator in a fraction) I don't see a real difference between "no-documents/row" and "null", both of them if you apply the metric, produce the same results.
2. a finite number 99.9 percent of the other cases (0 is within that case, is not a special one)

If we can distinguish 1) between cases where we don't have data to compute the metric and a non-finite result of the metric calculation then we could also consider marking the non-finite one with something like -, if not we should just hide those cells.

[markov00]

@flash1293 how do your field formatters threats NaN values? does they return an empty string for non finite numbers? because if not we should add a guard internally to avoid calling the formatted with an non finite value

[Nezz]

@Nezz as a workaround you can turn off "Show empty rows" which should show the gaps again:

@flash1293 Thank you, I have confirmed the workaround :)

[flash1293]

Blocked on elastic/elastic-charts#1708

[flash1293]

Fixed via #134450