Add support for All, Between and Not broadcating

[bkamins]

For select and combine we should in the future add option for All, Between and Not broadcating (when DataAPI.jl and InvertedIndices.jl is updated).

Unfortunately it is not possible to broadcast regex and colon as they are in Base.

@nalimilan An alternative to Not(:a) .=> fun could be to add yet another wrapper, e.g. Spread that would read Spread(Not(:a)) .=> fun and we would not have to change anything anywhere. What do you think? The benefit is that then Spread(:) .=> fun and Spread(r"x") .=> fun could also work.

[nalimilan]

My current thinking is that Spread would be equivalent to All if we merge JuliaData/DataAPI.jl#10. So the question is whether using All for this is OK, or whether it's worth introducing yet another Spread object (or any other name).

[bkamins]

No - it is not worth to add Spread as discussed.

I would leave a discussion what to do with this for later. For now we will add names(df, selector) option that will give the same for now.

I keep this open I we might come back to this decision after 1.0 release.

[pdeffebach]

Thanks for this. I understand that names(df, selector) is useful.

However I would really like to work on this for the post-1.0 period. It's a nice feature that would match Stata's convenient syntax.

[bkamins]

I would add this to DataFramesMeta.jl as we can intercept things like Between(:x1, :x3) .=> fun at macro processing step.

[CameronBieganek]

I'm afraid I'm a bit late to the select discussion. I have a proposal that avoids the need for broadcasting altogether. I apologize for proposing a syntax change after you've already merged the select PR.

Quoting from the Julia docs:

Tuples are an abstraction of the arguments of a function

With that in mind, here's my proposal:

• To apply a multivariate function to multiple columns, use the following syntax:
  • select(df, (:a, :b) => dot)
• To apply a univariate function to multiple columns, use the following syntax:
  • select(df, [:a, :b] => sum)

Though the parsing of : => sum presents one small difficulty:

julia> : => sum
ERROR: syntax: space not allowed after ":" used for quoting
Stacktrace:
 [1] top-level scope at REPL[193]:0

julia> (:) => sum
Colon() => sum

julia> [:] => sum
[Colon()] => sum

It seems like the Julia parser could be modified to parse : => var as Colon() => var. I don't see any ambiguity there. But in the meantime, (:) => var and [:] => var are parsed correctly and would work.

Advantages of this approach:

• There would no longer be a need to broadcast the pair operator over column selectors.
• Not(:y) => fun works, without needing to wrap with Cols and without needing to make Not broadcastable.
• [r"x", :] => fun works, without needing to wrap with Cols.
• [startswith("hello"), Between(:x1, :x3)] => fun works, without needing to wrap with Cols and without needing to make Between broadcastable.
• (:a, :b) => dot looks very similar to the corresponding closure syntax.
• Cols / All wouldn't actually be needed at all, since the arguments to select are effectively wrapped in a Cols.

Perhaps Cols would still be needed for getindex, but maybe that could be solved by allowing getindex to accept multiple column selector arguments, like select.

To clarify, under this proposal, Not(...) and Between(...) would expand to arrays of columns, [...].

Disadvantages:

There is one use case I can think of that works now on master but would be more difficult under the new proposal. This currently works:

julia> df = DataFrame(a=1, b=2, c=3, d=4);

julia> select(df, Between(:b, :d) => ByRow(max))
1×1 DataFrame
│ Row │ b_c_max │
│     │ Int64   │
├─────┼─────────┤
│ 1   │ 4       │

I'm not sure how that would be done under the new proposal. On the other hand, it doesn't seem like a very common use case to me, whereas [:a, :b] => sum and [:] => sum are quite common.

EDIT:
I think the above could be achieved in the new proposal with

select(df, AsTable(Between(:b, :d)) => ByRow(maximum))

[bkamins]

Thank you for the comment. Essentially you propose to introduce "automatic broadcasting" of vectors so if someone passes:

vector1 => sum => vector2

it gets parsed as currently would

vector1 .=> sum .=> vector2

This is doable without a problem and then special casing Between, Not, All, :, Cols is not a problem.

Also then you do not need tuple syntax to pass multiple arguments to a function, you would just write (using your problematic example):

select(df, [Between(:b, :d)] => ByRow(max))

or using a simpler example:

select(df, [[:a, :b]] => cor)

In summary:

1. what you propose is possible to be done
2. I think both options (adding broadcasting to Between, All etc.) and what you propose are comparable (both have some cases in which they are more convinient)

The issue is that:

1. adding broadcasting to Between etc. is not a pressing issue (we can do it later as it would be non-breaking) and currently you can write names(df, cols) that already is broadcastable;
2. your proposal is breaking so if we went for it we should decide on doing it now

@nalimilan - what do you think? As this decision is blocking form 0.21 release.