Rename @linq macro?

[CameronBieganek]

I'm not sure if the current plan is to keep @linq or not, but if it is kept, I think a different name would make more sense. @linq only makes sense if you're a former C# programmer. Perhaps @query instead?

Also, I'd like to argue in favor of keeping @linq, or rather @query. Here's a query from the README that uses Pipe.jl (except I've changed :a to a, etc).

using Pipe
out = @pipe df |>
    @transform(_, y = 10 * x) |>
    @where(_, a .> 2) |>
    @by(_, b, meanX = mean(x), meanY = mean(y)) |>
    @orderby(_, meanX) |>
    @select(_, meanX, meanY, var = b)

What I would prefer is a begin block syntax like this:

using Pipe
out = @query df begin
    transform(y = 10 * x)
    where(a .> 2)
    by(b, meanX = mean(x), meanY = mean(y))
    orderby(meanX)
    select(meanX, meanY, var = b)
end

I think the begin-block syntax is much easier to read. The second example avoids the noise from all the @, _, and |> symbols.

Actually, I already have an open issue for begin-block syntax for the @linq macro: #136

[xiaodaigh]

I actually want to separate it out #173

[CameronBieganek]

That seems a little excessive. Then we would have DataFramesMetaMeta.jl. 😄

[xiaodaigh]

That seems a little excessive. Then we would have DataFramesMetaMeta.jl.

To me the linq macros is just a distraction. There are too many things to maintain anyway.

[CameronBieganek]

On the contrary, I think a begin block form for a @query (or @linq) macro should be a pretty fundamental component of a macro package for DataFrames.jl, since it's the easiest to read and the closest to a SQL statement.

[xiaodaigh]

think a begin block form for a

using Lazy: @> # or using DataConvenience

@> df begin
  groupby(:grp)
  combine(:col1 => mean)
end

[CameronBieganek]

It's true that @> gets us most of the way there. But it has two downsides:

1. You have to introduce a dependency on Lazy.jl just for a pipe. Most of Lazy.jl is code unrelated to piping.

2. When combined with the macros from DataFramesMeta.jl, you have to write this:

@> df begin
    @transform(c = a + b)
    @groupby(g)
    @combine(c_mean = mean(c))
end

But I'd rather write this:

@query df begin
    transform(c = a + b)
    groupby(g)
    combine(c_mean = mean(c))
end

The second form would require a custom macro in DataFramesMeta.jl.

[xiaodaigh]

@query df begin
    transform(c = a + b)
    groupby(g)
    combine(c_mean = mean(c))
end

That doesn't compose well with other functions. Also, it's an extra burden on dev which is already stretching very thin. I vote for deprecating @linq. Interested author can revive it in another package.

Also, what's stopping DataFramesMeta.jl from just re-exporting @> or just stripping out the @> code and putting just that into DataFramesMeta.jl?

[CameronBieganek]

That doesn't compose well with other functions.

It composes just as well as @>. The semantics I have in mind is that it behaves the same as @>, except it makes the following replacements:

Input	Replaced With
transform	@transform
select	@select
where	@where
groupby	@groupby
combine	@combine

I'm willing to contribute a @query macro like this to DataFramesMeta.jl.

[xiaodaigh]

I'm willing to contribute a @query macro like this to DataFramesMeta.jl.

I see. That sounds like a good idea.

I also support making the @query macro in another package and DataFramesMeta can just re-export it

[nalimilan]

Maybe instead of special-casing transform, select, etc., the macro could just apply the same transformations to all functions? I.e. m = mean(x) would become :x => (x -> mean(x)) => :m and so on. Then any function would be supported, and custom functions supporting the pair syntax from packages or defined by users would benefit from the DataFramesMeta syntax.

@pdeffebach latest PR has come quite close to this already AFAICT.

[pdeffebach]

Maybe instead of special-casing transform, select, etc., the macro could just apply the same transformations to all functions? I.e. m = mean(x) would become :x => (x -> mean(x)) => :m and so on. Then any function would be supported, and custom functions supporting the pair syntax from packages or defined by users would benefit from the DataFramesMeta syntax.

yes this is basically exactly what we do [here](yes, this is

DataFramesMeta.jl/src/DataFramesMeta.jl

Line 100 in e0bfa26

function fun_to_vec(kw::Expr; nolhs = false)

). The current implementation throws an error if an input is not a Symbol or an expression of the form y = f(:x). If we simply uncomment that error, then we will fully support any expression that could be passed to DataFrames.transform.

With regards to re-naming. I would prefer @chain to @query because "query" already has a strong meaning in the data ecosystem. However I am hesitant to simply re-name as the only motivation for a breaking change.

That said, one feature I would like to implement for @linq is an optional underscore currying that would override the default first-argument parsing. I frequently want to use do blocks in pipes and that is currently not possible with @linq.

@linq df |>
    transform(y = f(:x)) |>
   groupby(:a) |>
   combine(_) do sdf
       # complicated stuff
   end |> 
   select(...)

@CameronBieganek I don't think we need a @groupby macro, since there is no special behavior this would have. One of the benefits of @linq currently is that you don't have awkward inconsistencies with which expressions are macros and which are not macros.

An important conflict that as not come up yet here is the tension between x and :x for variable naming. If we decide on x then presumably

@select(df, Between(a, z), y = f(x))

would work. But then, we would likely need some way for groupby(df, g) to work where g is parsed as :g. One method is as @nalimilan descibed where we search through all expressions in the pipe and replace all literals with Symbols. This is possible but would cause lots of headaches and bugs when we mess with expressions we don't "own", so to speak. I don't want to see

@linq df |>
    select(x, y, z = f(x))
    groupby(:g)
    custom_fun(:x, :y)

The mix of :x and x is confusing.

[nalimilan]

Sorry, I wrote m = mean(x), but that could as well have been m = mean(:x). The debate between x and :x is mostly orthogonal to the question of having @linq (or its new name) apply its transformations to any function AFAICT.

The idea of allowing optional underscores to pipe into an argument other than the first one is interesting.

[pdeffebach]

I think it's relevant because the main benefit of @linq is consistency and ease of reading, so how we mix :x and x matters in that respect.

As you know, we want to emulate dplyr's ..

[CameronBieganek]

I agree that @chain (or @pipe, which is already taken) is a better name for this than @query.

---

Maybe instead of special-casing transform, select, etc., the macro could just apply the same transformations to all functions? I.e. m = mean(x) would become :x => (x -> mean(x)) => :m and so on.

That would work fairly well, except then we wouldn't be able to add other arbitrary functions to the chain/pipe. For instance, I'd like to be able to write the following:

@chain df begin
    transform(c = a + b)
    where(a < 100)
    tail(30)
    repeat(outer = 3)
end

If we implemented your proposal, then repeat(outer = 3) would be turned into

repeat(..., [] => (() -> 3) => :outer) 

(or something like that), which would cause an error.

---

Personally, my main desire for a @chain macro is to be able to write something that looks as similar to a SQL query as possible, and thus avoids all the noise from @, _, and |> symbols. However, I do want to be able to chain other functions, so it shouldn't be strictly limited to SQL keywords (i.e. DataFramesMeta macros).

---

I don't think we need a @groupby macro, since there is no special behavior this would have.

Well, we would need a macro if we want to allow groupby(a) instead of groupby(:a). 🙂

---

an optional underscore currying that would override the default first-argument parsing

I like this idea.

---

@linq df |>
   select(x, y, z = f(x))
   groupby(:g)
   custom_fun(:x, :y)

To ameliorate this situation, perhaps @chain could allow DataFramesMeta.jl macros to be written either with or without the @ symbol. Thus you could disambiguate the above like this:

@chain df begin
    @select(x, y, z = f(x))
    @groupby(g)
    custom_fun(:x, :y)
end

[CameronBieganek]

The new PipelessPipes.jl seems to provide almost everything I want. The only thing it doesn't let me do is write transform instead of @transform. But maybe it's better to be explicit about when you are using a DataFramesMeta macro rather than a DataFrames function. I suppose some people might want to write a pipe that, for instance, uses both DataFramesMeta.@transform and DataFrames.transform in the same pipe.