usefulness of a colwise! method?

[mkborregaard]

DataTables implement a colwise method, but no colwise!. There aren't a massive number of use cases for this, but I can think of e.g. data centering and normalization. If there are no major technical obstacles with making such a function I think it'd make a nice addition.

[cjprybol]

Are you imagining that colwise! would work like an in-place aggregate?

example:

julia> dt
10×2 DataTables.DataTable
│ Row │ A  │ B  │
├─────┼────┼────┤
│ 1   │ 1  │ 1  │
│ 2   │ 2  │ 2  │
│ 3   │ 3  │ 3  │
│ 4   │ 4  │ 4  │
│ 5   │ 5  │ 5  │
│ 6   │ 6  │ 6  │
│ 7   │ 7  │ 7  │
│ 8   │ 8  │ 8  │
│ 9   │ 9  │ 9  │
│ 10  │ 10 │ 10 │

julia> colwise!(normalize, dt)
10×2 DataTables.DataTable
│ Row │ A         │ B         │
├─────┼───────────┼───────────┤
│ 1   │ 0.0509647 │ 0.0509647 │
│ 2   │ 0.101929  │ 0.101929  │
│ 3   │ 0.152894  │ 0.152894  │
│ 4   │ 0.203859  │ 0.203859  │
│ 5   │ 0.254824  │ 0.254824  │
│ 6   │ 0.305788  │ 0.305788  │
│ 7   │ 0.356753  │ 0.356753  │
│ 8   │ 0.407718  │ 0.407718  │
│ 9   │ 0.458682  │ 0.458682  │
│ 10  │ 0.509647  │ 0.509647  │

# compared to

julia> aggregate(dt, normalize)
10×2 DataTables.DataTable
│ Row │ A_normalize │ B_normalize │
├─────┼─────────────┼─────────────┤
│ 1   │ 0.0509647   │ 0.0509647   │
│ 2   │ 0.101929    │ 0.101929    │
│ 3   │ 0.152894    │ 0.152894    │
│ 4   │ 0.203859    │ 0.203859    │
│ 5   │ 0.254824    │ 0.254824    │
│ 6   │ 0.305788    │ 0.305788    │
│ 7   │ 0.356753    │ 0.356753    │
│ 8   │ 0.407718    │ 0.407718    │
│ 9   │ 0.458682    │ 0.458682    │
│ 10  │ 0.509647    │ 0.509647    │

That should be pretty straight forward to do. What would be the preferable behavior for functions that reduce to scalars like sum/length or extrema? Fail on a dimension mismatch or resize the DataTable to be a single row? (I'm not sure if resize!(column, 1) will work, in which case it'd have to throw an error because it couldn't modify in place).

example

julia> colwise!(length, dt)
1×2 DataTables.DataTable
│ Row │ A  │ B  │
├─────┼────┼────┤
│ 1   │ 10 │ 10 │

# or
Error: DimensionMismatch: columns cannot be resized in place, use aggregate(function, dt) instead

[nalimilan]

aggregate is made to be used with a grouping variable. colwise! would just be an in-place equivalent of colwise, so something very simple. Cf. https://discourse.julialang.org/t/announcement-dataframes-future-plans/266/25.

Though thinking about it more, I'm not sure how useful such a function would be. colwise is mainly useful for summary functions which return one or two values for each column. OTC, colwise! would only work for functions which return the vectors or the same length as the input. So probably better give this function a different name.

The other issue with such a function is that colwise!(col -> col .- mean(col), df) it would encourage an inefficient pattern which allocates a copy of col before replacing the old column. A manual loop with x0 = mean(col); map!(x -> x - x0, col) is much more efficient in that regard. So if we offer such a helper function, it should be smarter than that. We could have it call the function for each element of a column, with additional column-wise arguments à la broadcast. For example: colwise!((x, x0) -> x - x0, df, colwise(mean, df).

[mkborregaard]

Reading your concerns here and thinking about it more, I am also not sure this functionality would be broadly useful. It came out of a natural impulse to work with a DataTable inplace because this is so ideomatic in julia.
Maybe if map! could simply take an eachcol iterator as argument?

[nalimilan]

Yes, that would make sense. Maybe a bit surprising that you would be able to modify an iterator, but doesn't sound like a big issue.

[nalimilan]

Leaving a breadcrumb to https://stackoverflow.com/questions/44235201/shortcut-to-transforming-a-dataframe, which provides an example of a use case for this.