Integration with Plots.jl

[bkamins]

I am making an issue here, because what I discuss is not released yet.

When we deprecate CategoricalString we should coordinate with Plots.jl so that it is aware of CategoricalValue{String} and properly unwrap it (or maybe even properly unwrap any CategoricalValue), as otherwise a lot of plotting code that was working in the past will be broken.

[nalimilan]

You mean that the fact that values no longer inherit from AbstractString is going to break Plots.jl? If so, then we should get in touch with them right now, as they can start fixing this using CategoricalValue (or only CategoricalValue{String}), which already exists. But they will probably be reluctant to take a dependency on CategoricalArrays.

[bkamins]

Yes - it is going to break (actually I think not break but produce deprecation warnings) a lot of code that used to work when plotting data frames with categorical columns that are strings.

But I guess you are right that CategoricalArrays.jl as a dependency for Plots.jl is problematic. However, maybe I was not right and it is not a problem. CategoricalString is AbstractString so I guess then Plots.jl assumes it is a string and does not touch it. CategoricalValue is not AbstractString so maybe Plots.jl in such a case will convert it to string, which will automatically work. Then we have a problem only in "transition period" when CategoricalString is still used by CategoricalArrays.jl but is deprecated (then plotting will produce deprecation warnings as Plots.jl does index into strings). So maybe a "hard" switch to CategoricalValue and dropping CategoricalString immediately would be better.

@daschw - do you have any opinion on this?

[nalimilan]

I don't think it should print deprecation warnings, as AFAICT Plots.jl doesn't use CategoricalString at all. But if it relies indirectly on the fact that CategoricalString <: AbstractString, then user code will break without a deprecation.

[bkamins]

Yes it relies on CategoricalString <: AbstractString so for example it does things like:

julia> x = categorical(["abc"])
1-element CategoricalArray{String,1,UInt32}:
 "abc"

julia> first(x[1])
┌ Warning: `iterate(x::CategoricalString)` is deprecated, use `iterate(String(x))` instead.

and if x was passed to plot then such things happen. When you do:

julia> using Plots

julia> plot(1, x)

you get like 10 warnings now as:

julia> x[1]
CategoricalString{UInt32} "abc"

is this clear now where the problem is? (still maybe nothing needs to be done if
x = categorical(["abc"]) will produce CategoricalValue{String} elements)

[daschw]

Thanks a lot for thinking of Plots and informing me! I don't have time right now, but I will check and report back (or ask some stupid questions) soon.

[daschw]

So, from what I've tried now with CategoricalArrays master, it won't just work with CategoricalValue. Without CategoricalValue{<:AbstractString} <: AbstractString and CategoricalValue{<:Real} <: Real or something similar, Plots won't know to extract/convert the data.
A very simple solution is using a type recipe.

@recipe f(::Type{T}, ::T) where {T <: CategoricalValue} = (get, string)

It returns two functions. The first tells Plots how to convert CategoricalValues to something it can handle (String, Number) and the second defines how to format this converted values in the axes tick labels. I tested the recipe above and it works for both CategoricalValue{<:String} and CategoricalValue{<:Real}.
I think that would be the easiest way forward. Now it just has to be decided, whether

• Plots takes a CategoricalArrays dependency and defines above recipe or
• CategoricalArrays takes a RecipesBase dependency to add this.

Of course our preferred option would be the latter. We are currently working on supporting RecipesBase also with Makie. This will probably take a little longer, but when we have that you'd also have automatic Makie support for CategoricalArrays, if you include the recipe here.

[nalimilan]

Ah that's too bad. I guess we'll have to depend on RecipesBase then (we already depend on JSON for similar reasons, which is absurd). We really need a mechanism for optional dependencies.

Something I don't really understand: why doesn't Plots simply call string on the values of unknown types to get a string representation? That would work with CategoricalValue.

[bkamins]

why doesn't Plots simply call string on the values of unknown types to get a string representation?

This is what also assumed

[daschw]

Something I don't really understand: why doesn't Plots simply call string on the values of unknown types to get a string representation? That would work with CategoricalValue.

Consider cvs = categorical([1, 2, 1000]). If we convert this to string (as a type unknown to Plots) it would have no numerical representation, so the values would be next to each other on the axis with equal distance, which is surely not what we want:

plot(string.(cvs), 1:3)

[bkamins]

This I understand, but I guess recipes could allow to opt-out from a default behaviour which would be to convert to string (as currently I understand it errors).

[daschw]

Yeah, maybe it would make sense to have string conversion as default. Honestly, I have never thought about this. Then it would error if no string method is defined for the type or lead to unexpected results like in my example above. Still, I guess you would not want that behavior for CategoricalValue{<:Real}. So the issue of depending on RecipesBase or Plots depending on CategoricalArrays remains. I'm not sure how bad a CategoricalArrays dependency would be for Plots. For sure we want to support plotting Dataframes and similar Tables.
@mkborregaard What's your opinion on this, or do you see a different solution without introducing dependencies in Plots and CategoricalArrays?
In my opinion this is a perfect showcase of what RecipesBase was created for.

[bkamins]

I think CategoricalArrays.jl is not a good dependency for Plots.jl as of today.

The reason is that CategoricalArrays.jl invalidates a lot of compiled code from Base currently:

julia> @time using Plots
  6.199417 seconds (8.32 M allocations: 469.658 MiB, 3.66% gc time)

julia> @time plot(1:10)
  4.805318 seconds (9.98 M allocations: 497.441 MiB, 2.38% gc time)

vs

julia> @time using CategoricalArrays, Plots
 10.360881 seconds (23.50 M allocations: 1.183 GiB, 5.03% gc time)

julia> @time plot(1:10)
  5.236466 seconds (10.96 M allocations: 562.959 MiB, 3.25% gc time)

so reduction of time-to-first-plot would be more problematic again.

[daschw]

OK, thanks, good to know!

[nalimilan]

Consider cvs = categorical([1, 2, 1000]). If we convert this to string (as a type unknown to Plots) it would have no numerical representation, so the values would be next to each other on the axis with equal distance, which is surely not what we want:

categorical([1, 2, 1000]) should definitely not be treated as numerical data. The point of CategoricalValue is to indicate that whatever value it wraps is categorical, i.e. such values should be considered as qualitative instead of quantitative. So it would be completely fine to call string on it.

Actually, the recipe would probably have to be:

@recipe f(::Type{T}, ::T) where {T <: CategoricalValue} = (string, string)

to ensure that numbers wrapped in categorical values are not treated as numeric.

(In practice, CategoricalValue{<:Real} is probably not that useful nor common. CategoricalValue{String} is by far the most useful case.)

[mkborregaard]

Doesn't that break ordinal? That's a fairly common use case, using categorical values to order categories (e.g. groups)