Histogram2d's default colors with transparent background considered harmful

[yha]

histogram2d renders empty bins as transparent, which contrasts strongly with nearly-empty bins in colormaps where small is dark, including the default colormap:

x = Float64[]
let y = 0.0
    for _=1:500
        push!(x, y += randn() - 0.1y)
    end
end
histogram2d([x[2:end]], [x[1:end-1]])

I would say this is a misleading representation of the data. This is much better:

histogram2d([x[2:end]], [x[1:end-1]], background="black")

It's worse when the distribution has spikes surrounded by zeros:

using Distributions
x = let n = 100_000
    rand(Poisson(1),n) .* exp.(rand(n).*(2π*im)) .+ 0.1randn(n)
end
histogram2d(x)

The spike in the middle is barely visible. The distribution really looks like this:

I think the best solution is to simply render empty bins as zeros rather than transparent. This is what matplotlib does. The distinction between 0 and 1 should be usually negligible, having to do with noise and sample size.
Alternately, the default colormap for 2d histograms should have white at 0 (I don't think there's currently any such colormap, though some are close), or the default background for it should be black.

[mkborregaard]

I disagree with this. For most uses the absence or presence of something is meaningful, and shouldn't be obscured by being the same black color. You can easily use a color scale that is light at the lowest values, if you need this - e.g. color = :Blues_r.

[yha]

For most uses the absence or presence of something is meaningful

Can you give an example of such a use case?
In any case, I think the bigger problem is that the scale is reversed relative to the default white background, so the higher values are almost invisible next to zero.
Even if the distinction between 0 and 1 is meaningful, surely it's never as meaningful as the distinction between 0 and a lot?

[mkborregaard]

I think actually your example above is such a use case... It also makes it much clearer when used for spotting intercorrelations among variables.
I sympathise with your second point but - I'm not sure about changing the default color scale for this particular seriestype. @daschw what do you think?

[daschw]

I totally agree with @mkborregaard.
I get your point @yha, but I think we (in Plots) should not obscure the distinction between 0 and 1. If the users want to do so, they could simply add something like bg_inside = cgrad()[1]. Not sure if this would justify a new attribute to achieve this more easily ...
I would not change the color scale for a specific seriestype either.

[yha]

I think actually your example above is such a use case...

It might be, but it's not really a use case, just an artificial example (it's based on something I actually encountered when plotting a histogram, but in the actual use case the "ripples" were stronger, and I probably wouldn't have cared about any that only had 1's in them). In most cases, a histogram is for estimating a distribution based on samples from that distribution, and the 0-1 distinction is meaningless noise, but is given the strongest contrast by default. I also don't understand what you mean about "spotting intercorrelations".
I think if the distinction between 0 and 1 matters, it probably makes more sense to do a scatter plot anyway. But if we want to keep accentuating this distinction in histograms, a reversed colormap is still a great improvement, I think. The same example as above with :grays_r:

If the users want to do so, they could simply add something like bg_inside = cgrad()[1].

That's not very obvious, and also brittle. For colormaps other than the default it would have to be something like c=:viridis, bg_inside=cgrad(:viridis)[1], which is pretty cumbersome. And it doesn't work for StatsPlots.marginalhist, because it colors the marginals as well.
So I believe this does justify an attribute, or a change to the default colormap for histograms. If there is no objection, I'll open a PR, but otherwise, I guess I'll just write my own histogram recipe.

[daschw]

So I believe this does justify an attribute, or a change to the default colormap for histograms. If there is no objection, I'll open a PR, but otherwise, I guess I'll just write my own histogram recipe.

I guess there would be no harm introducing an attribute like color_zeros::Bool = false, or some better keyword name, to simplify this for the user. From my side there's no objection against this PR.