add map for AbstractDataFrame

[bkamins]

Following #1952 (comment) a split discussion placaholder for map on AbstractDataFrame functionality.

The difference of map(something, df) from map(something, eachrow(df)) is that the former would produce a DataFrame and it would allow naming columns and using shorthand transformation specifications like map for GroupedDataFrame, so something like:

map(:c => abs => :d, df)

and

map((d=:c => abs, d2=:c=abs2), df)

would be allowed

[nalimilan]

I was actually thinking the same, but I realized there could be an issue with the way map works on GroupedDataFrame currently. Indeed since map(f, gd) operates over SubDataFrame and map(:x => f, gd) operates over column vectors, it would seem natural to have map(:x => f, df) also operate on column vectors. But that goes against the view of data frames as collections of rows.

So we have to either:

• live with that semi-inconsistency
• have map operate on columns (introducing another inconsistency with data frames as collections of rows)
• change how map works on GroupedDataFrame (introducing another inconsistency with grouped data frames as collections of groups)
• just drop map completely and use other functions instead

[bkamins]

Today I was thinking about how select should work on AbstractDataFrame. I had changed my mind several times.

Initially I wanted it to take whole columns, and in such a case it is natural to have map that works row-wise (as in this proposal).

However, later I came to a conclusion that for select to work on AbstractDataFrame it is almost useless to have it work on whole columns (I could not think of any really strong use case). So if we decide to make select work row-wise, then map for AbstractDataFrame is not needed and we can just drop this issue (as it would do essentially the same what select does).

The issue is that GroupedDataFrame a whole-column in a subgroup approach is natural both for combine and for map as it is a collection of data frames. And the difference is only what is the return value - DataFrame for combine and GroupedDataFrame for map.

For AbstractDataFrame we do not have this dual need - it is natural to want a DataFrame as a result of select or map on it. On the other hand we treat a data frame as a collection of rows, so this is what I think we should stick to.

Just to sum up, e.g. the use case to standardize the column of a data frame is not so appealing as it is very easy to achieve via setproperty! or setindex!, so select does not have to cover it, or it can cover it like:

select(df, :oldcol => x -> (x - mean(df.oldcol)) / std(df.oldcol) => :newcol)

This is a bit verbose but not super bad, as opposed to eg having to write:

select(df, :oldcol => x -> abs.(x) => :newcol)

when you want to take abs of a column, while in row-wise approach it is a clean:

select(df, :oldcol => abs => :newcol)

(I think row-wise operations on data frame - like in JuliaDB.jl will be needed more often by users than whole column operations)

[bkamins]

CC @juliohm

[bkamins]

OK - I would leave this decision for post 1.0 release.