Data format interface

[piever]

At the moment the data format is slightly inconsistent. The following two options are supported.

• A dataset is passed, eg data(df) * mapping(:col1, :col2) (long format), or data(df) * mapping(:col1, [:col2, :col3]) (wide format)
• No dataset is passed, eg mapping([rand(10), for _ in 1:3], [rand(10), for _ in 1:3], color=["a", "b", "c"]) (pregrouped format)

Naturally, this is inconsistent: whether data is in a dataset or in a set of variables should not affect whether it is considered "pregrouped" (each entry is a trace in the plot) or not.

However, the above approach has a key advantage: long and wide format can be supported simultaneously, AlgebraOfGraphics can figure it out based on whether a mapping has a unique symbol or a list of symbols. The two approaches can already happen simultaneously, with e.g. mapping(:col1, [:col2, :col3], color = :col4). This format (which we can call columnwise) is also useful when data is in separate variables, and is the defaut in Plots.jl (where one can do scatter(rand(10), rand(10, 5)) to plot 5 traces).

Proposal

Distinguish between three possible formats (going from data to plot):

1. Raw: the whole data is passed as-is to the plotting function, eg visual(Heatmap) * mapping(1:10, 1:10, rand(10, 10))
2. Columnwise: each column of the data correspond to a trace, eg visual(Scatter) * mapping(rand(10), rand(10, 5)) gives 5 traces
3. Pregrouped: each entry of the data correspond to a trace, eg visual(Scatter) * mapping([rand(10), for _ in 1:5], [rand(10), for _ in 1:5], color=["a", "b", "c", "d", "e"]) gives 3 traces

Conceptually, the difference is how one slices the data to get a trace. The option raw slices along all directions (the whole data is a unique trace), the option columnwise slices along the first direction (each column is a trace), the option pregrouped slices along no directions (each entry is a trace). So a possible API could be

# pass the info together with data
data([df=nothing]; [dims=1]) # similar to eg `sum(v, dims=1)`,  or `mapslices(plot, v, dims=1)`

where possible values of dims are

• All() for raw (using and reexporting DataAPI.All)
• 1 for columnwise (IMO, should be the default)
• () for pregrouped

⚠️ This could be confusing, as it seems mapslices and eachslice follow different conventions in base.

In this scenario, having variables in a dataset or separated has no effect: the columns are simply extracted, then the slicing context is applied.

Most things seem possible with a reasonably easy syntax with the column-wise default, other than plots where it's not possible / convenient to pass data as vectors (eg surface), in which case one should explicitly declare the raw context.

Remark

The raw context may not be necessary, it can be achieved via the pregrouped context by wrapping each entry in fill. In that case, a simpler API may be data([df=nothing]; [pregrouped=false]).

Naming and styling "slices"

At the moment, this is also inconsistent, as for the pregrouped context, the user simply passes the list of names, eg

mapping([rand(10) for _ in 1:3], [rand(10) for _ in 1:3], color=["a", "b", "c"])

whereas for the columnwise context at the moment one uses a dims helper function, eg

data(df) * mapping(:col1, [:col2, :col3], color=dims(1) => renamer([:col2, :col3]))

This feels a bit inconsistent and is annoying in practice. Unfortunately, passing color=[:col2, :col3] wouldn't work, as AoG would try to apply the data extraction / slicing mechanism to it.

A possibility to uniform both approaches would be to have an escape or just function to signal "ignore this item when applying the data extraction / slicing pipeline. So the two examples above, with the new proposal, would be

data(dims=()) * mapping([rand(10) for _ in 1:3], [rand(10) for _ in 1:3], color=escape(["a", "b", "c"]))
data(df) * mapping(:col1, [:col2, :col3], color=escape([:col2, :col3]))

Note that in the first case escape is not really necessary, as data(dims=()) does not in practice cause any data extraction / slicing.

[knuesel]

Thanks for publishing this. I must say I already find the current API rather complex and confusing, and some of the proposed changes would make it even more overwhelming... I've been thinking about how to simplify the API towards a few general rules so here's my two cents:

• Naming: mapping([rand(10) for _ in 1:3], [rand(10) for _ in 1:3], color=["a", "b", "c"]) is the outlier. It's too magical, consistency would require something like dims(1).

• To make dims(1) more acceptable, renamer could be implicit when passing a list of symbols/strings with =>. Two ideas:

  • Recognize vectors on the right-hand side of => and allow the following:

    mapping(..., color=dims(1) => ["A", "B"])
    
    # With a legend title
    mapping(..., color=dims(1) => ["A", "B"] => "Groups")

  • Do something smart with broadcasting, like DataFrames does:

    mapping(..., color=dims(1) .=> ["A", "B"])
    
    # With a legend title
    mapping(..., color=dims(1) .=> ["A", "B"] => "Groups")     # or maybe require parentheses here

• Currently, data declaration is not consistent: data only works for tabular data, while mapping is used for pre-grouped data. It's a bit confusing... it would be clearer if mapping was only used for mappings. It's also restrictive: it works for pre-grouped data but not for simple data: mapping(1:5, 1:5) doesn't work, it clashes with mapping of column indices (e.g. in transformations like mapping(1:5 => f) and wide format like mapping(:a, [1,2,3,4,5])).

  I think it would be better to have a clear separation: data() is only for data, mapping() only for mappings. Then provide mapdata() as a shortcut to declare and map data in one step. This removes the ambiguity of 1:5 and provides a simpler "direct plotting of data" that is often requested by users:

  data(df) * mapping(:a, 1:5)   # Wide format with 5 series
  
  mapdata(1:5, 1:5)             # Two simple positional parameters (one series)

• This way, pre-grouped and wide data can be handled consistently: for any parameter, an "outer vector" is used to specify groups:

  • For table data this gives mapping(:a, [:b, :c]).

  • For pre-grouped variables outside of a table, mapdata(1:5, [rand(5), rand(5)]) does the same thing.

• Matrices don't need special treatment, "raw" is the default (except for pre-processing of categorical types). Indeed matrices are valid Makie parameters so it's best to keep it like this. The pre-grouped API already gives a reasonable solution for the "five traces with one matrix" case (see below).

Examples

It seems to me the following examples cover the desired functionality in a simple way (but maybe I don't have the full picture):

Simple data

df = (a=1:3, b=rand(3), c=["x", "y", "z"])
data(df) * mapping(:a, :b, color=:c)

Short form:

mapdata(1:3, rand(3), color=["x", "y", "z"])

Pre-grouped

df = (a=1:5, b=1:5, c=rand(5), d=rand(5))
data(df) * mapping([:a, :b], [:c, :d], color=dims(1))

Short form:

mapdata([1:5, 1:5], [rand(5), rand(5)], color=dims(1))

With group names

data(df) * mapping([:a, :b], [:c, :d], color=dims(1) => ["A", "B"])

Short form:

mapdata([1:5, 1:5], [rand(5), rand(5)], color=dims(1) => ["A, "B"])

With more dimensions

df = (a=-2:2, b=1:5, c=rand(5), d=rand(5), e=rand(5), f=rand(5))
data(df) * mapping([:a :b; :a :b], [:c :d, :e :f], marker=dims(1), color=dims(2))

Short form:

mapdata([[-2:2] [1:5]; [-2:2] [1:5]], fill(rand(5), 2, 2), marker=dims(1), color=dims(2))

Wide data and mixed data

This is just a special case of pre-grouped data where some parameters have no groups:

df = (a=1:5, b=rand(5), c=rand(5), d=["a", "b", "c", "d", "e"])
data(df) * mapping(:a, [:b, :c], color=:d)

Short form:

mapdata(1:5, [rand(5), rand(5)], color=["a", "b", "c", "d", "e"])

More complex

df = (a=1:5, b=rand(5), c=rand(5), d=["a", "b", "c", "d", "e"], e=rand(1:5))
data(df) * mapping(:a, [:b, :c], color=[:d, :e])

Short form:

mapdata(1:5, [rand(5), rand(5)], color=[["a", "b", "c", "d", "e"], rand(1:5)])

Matrix data

Here it probably makes sense to only support the short form, mapping directly the data without going through a "table":

visual(Heatmap) * mapdata(1:10, 1:10, rand(10, 10))

Matrix columns

No specific support for this, it can be done easy enough as a case of pre-grouped data:

A = rand(10, 10)
visual(Scatter) * mapdata(1:10, [eachcol(A)...])

[piever]

Thanks for the thoughtful comment! If I understand correctly, conceptually the main difference is that, instead of having the user "explicitly declare" whether data is pregrouped or not, AlgebraOfGraphics would understand it based on the type of the positional arguments.

My understanding of the proposal

Case 1. All arguments are vectors or collections of vectors. This is both pregrouped or wide case (or long case, if there are no "collections of vectors" (can be general collections, as eachcol(mat)). There is ambiguity in

v1, v2, v3 = rand(3), rand(3), rand(3)
mapdata(1:3, [v1, v2, v3], color=["a", "b", "c"])

as to whether this is

scatter!(1:3, v1, color=["a", "b", "c"])
scatter!(1:3, v2, color=["a", "b", "c"])
scatter!(1:3, v3, color=["a", "b", "c"])

or rather

scatter!(1:3, v1, color="a")
scatter!(1:3, v2, color="b")
scatter!(1:3, v3, color="c")

If I understand your proposal correctly, it is the former. The latter would be supported via color=dims(1) => ["a", "b", "c"]. (I find it reasonable to make renamer less needed and instead allow the user to pass directly arrays or dictionaries (maybe lists of pairs).

How exactly to use broadcasting to pass the transforming function and name definitely deserves some thought, but I imagine a reasonable solution can be found.

Case 2. Some arguments are neither vectors nor collections of vectors (eg, heatmap(1:3, 1:3, rand(3, 3)). In this case, one wouldn't really do much (other than converting categorical variables, possibly only in positional arguments).

Potential ambiguities

As far as I see it, there are a couple of possible ambiguities this introduces.

1. The Makie plot itself takes as argument a collection of vectors, see eg scatter(::Vector{Point2f}) or poly(::Vector{Vector{Point2f}}). This would confuse the machinery. OTOH, Point2f (or, more generally, StaticVector) can be special-cased as to not count as vector. It is a bit awkward, but IMO it makes sense to distinguish between a geometrical object and a list of things. poly is a bit more problematic, but one could recommend users to go for poly(::Vector{Polygon}) instead. In general, we could also push for Makie and GeometryBasics to rely on custom types rather than vectors for geometric objects.

2. This latter thing is less likely to be an issue, but things would be slightly problematic if a user had v::Vector{<:AbstractVector} where the entries of v were actually values of a categorical variable (that for some reason is valued in vectors). This is probably an odd scenario anyway, and I imagine it would be handled by something like nonnumeric (or an explicit marker that the variable is categorical).

Mapdata versus mapping

This is the part of the proposal that I understand the least. If mapdata has the role of mapping when no data is present, then why do we need it at all? Could mapping do all the job, with the following simple rule: In the presence of data, symbols are replaced by columns, lists of symbols by lists of columns, and default names are added. This way, there is no difference between data(df) * mapping(:x, :y) and mapping(df.x => "x", df.y => "y")?

If that is the case, than it may not be completely necessary to have a whole new function for that, mapping could probably work for both.

One related thought, however, is #328. There, the problem is that one wants to pass a vector markersize as mapping (it is a vector with the same length of the data, so for things to work, it should be passed to mapping, passing it to visual screws up with grouping). The slightly awkward thing is that here it is a bit hard to pass it (one would need to modify the dataframe). Possible solutions:

data(df; newvar=[20,30]) * mapping(:x, :y, color=:s, markersize=:newvar) # add syntax to add things on the fly
data(df) * mapping(:x, :y, color=:s, markersize=escape([20,30])) # explicitly declare that `[20, 30]` is not part of the "table context"
data(df) * mapping(:x, :y, color=:s) * mapdata(markersize=[20,30]) # mapdata means "ignore existing dataset, use `[20, 30]` as is

I'd be slightly unhappy to have to use 3 exported functions for this, not sure about the best way forward.

[knuesel]

Regarding case 1 and 2: yes that's right. In case 1, ["a", "b", "c"] is a simple data vector like 1:3 and v1 so it's treated the same. And dims(1) can be understood as the "simple data vector" that gives the indices of the first group dimension. This way it seems quite consistent.

Note that in my mind the cases were divided differently:

1. "arrays of scalars": for example 1:3, v1, ["a", "b", "c"] and rand(3, 3). These are used as-is except for categorical conversion.
2. "arrays of arrays": for example [rand(5), rand(5)]. Such a parameter always defines groups (corresponding to different traces).

Column identifiers represent vectors, so in this sense it's consistent: the [:b, :c] in data(df)*mapping(:a, [:b, :c]) represents an array of arrays so it defines groups.

Ambiguities:

1. You're quite right with Point and poly (I couldn't find an example of poly(::Vector{Vector{Point2f}}), but this signature is definitely a problem). Note that it's only a problem with the short form (direct values without data(df)). The user could always fall back to the long form:

   df = (; p=[[Point2f(0,0), ...]])  # df.p isa Vector{Vector{Point2f}}
   data(df) * mapping(:p)

2. Another good point. nonnumeric should do the job here as you say (or here also the user can fall back on the long form).

Mapdata vs mapping: indeed! I hadn't thought of using the "no data" information, I think that would work fine. Much nicer than introducing mapdata.

Regarding #328, another option would be to allow dims(0) to refer to the dimension along the data series, allowing something like

data(df) * mapping(:x, :y, color=:s, markersize=dims(0) => [20,30])

I guess this could prove useful in other contexts too... And maybe more useful than escape as it would allow transformations based on the index in the series:

# Marker size proportional to element index:
data(df) * mapping(:x, :y, color=:s, markersize=dims(0))

data(df) * mapping(:x, :y, color=:s, markersize=dims(0) => sqrt)

Edit: maybe this dims(0) could also serve in the ambiguous cases with poly, etc. (though of course it's annoying for the user when things don't just work)...