public API for accessing row number of `DataFrameRow`

[ExpandingMan]

Internally this is called row but this symbol is probably far too common to export. rownumber seems like one possibility. I feel like it should probably be something from Base, but I'm not sure what, there isn't already an indexof or anything quite like that I think.

[pdeffebach]

One corner case I just thought of for this

julia> df = DataFrame(a = [1, 1, 2,2], b = rand(4));

julia> gd = groupby(df, "a");

julia> t = gd[2][1,:];

julia> getfield(t, :row)
3

[bkamins]

The corner case is that currently row always points at the parent DataFrame not at the row of SubDataFrame. This is due to performance reasons (as no intermediate index calculation is required). DataFrameRow is even not aware of existence of SubDataFrame. Since we have not exposed row in the past it was not a problem. However, if we were to add this functionality maybe this should be changed.

[bkamins]

Note that the behavior of unrolling to the true parent is the same as in Base:

julia> x = rand(3,4)
3×4 Array{Float64,2}:
 0.50024   0.728962   0.0994949  0.989262
 0.644985  0.185798   0.695228   0.260861
julia> x = reshape(1:16, 4, 4)
4×4 reshape(::UnitRange{Int64}, 4, 4) with eltype Int64:
 1  5   9  13
 2  6  10  14
 3  7  11  15
 4  8  12  16

julia> y = view(x, 2:3, 2:3)
2×2 view(reshape(::UnitRange{Int64}, 4, 4), 2:3, 2:3) with eltype Int64:
 6  10
 7  11

julia> parent(y)
4×4 reshape(::UnitRange{Int64}, 4, 4) with eltype Int64:
 1  5   9  13
 2  6  10  14
 3  7  11  15
 4  8  12  16

julia> parentindices(y)
(2:3, 2:3)

julia> z = view(y, 2:2, 2:2)
1×1 view(reshape(::UnitRange{Int64}, 4, 4), 3:3, 3:3) with eltype Int64:
 11

julia> parent(z)
4×4 reshape(::UnitRange{Int64}, 4, 4) with eltype Int64:
 1  5   9  13
 2  6  10  14
 3  7  11  15
 4  8  12  16

julia> parentindices(z)
(3:3, 3:3)

[bkamins]

This also reminded me that you can use parentindices to get row and column indices from the parent:

julia> df = DataFrame(x)
4×4 DataFrame
│ Row │ x1    │ x2    │ x3    │ x4    │
│     │ Int64 │ Int64 │ Int64 │ Int64 │
├─────┼───────┼───────┼───────┼───────┤
│ 1   │ 1     │ 5     │ 9     │ 13    │
│ 2   │ 2     │ 6     │ 10    │ 14    │
│ 3   │ 3     │ 7     │ 11    │ 15    │
│ 4   │ 4     │ 8     │ 12    │ 16    │

julia> dfy = view(df, 2:3, 2:3)
2×2 SubDataFrame
│ Row │ x2    │ x3    │
│     │ Int64 │ Int64 │
├─────┼───────┼───────┤
│ 1   │ 6     │ 10    │
│ 2   │ 7     │ 11    │

julia> parent(dfy)
4×4 DataFrame
│ Row │ x1    │ x2    │ x3    │ x4    │
│     │ Int64 │ Int64 │ Int64 │ Int64 │
├─────┼───────┼───────┼───────┼───────┤
│ 1   │ 1     │ 5     │ 9     │ 13    │
│ 2   │ 2     │ 6     │ 10    │ 14    │
│ 3   │ 3     │ 7     │ 11    │ 15    │
│ 4   │ 4     │ 8     │ 12    │ 16    │

julia> parentindices(dfy)
(2:3, 2:3)

julia> r = dfy[2, :]
DataFrameRow
│ Row │ x2    │ x3    │
│     │ Int64 │ Int64 │
├─────┼───────┼───────┤
│ 3   │ 7     │ 11    │

julia> parent(r)
4×4 DataFrame
│ Row │ x1    │ x2    │ x3    │ x4    │
│     │ Int64 │ Int64 │ Int64 │ Int64 │
├─────┼───────┼───────┼───────┼───────┤
│ 1   │ 1     │ 5     │ 9     │ 13    │
│ 2   │ 2     │ 6     │ 10    │ 14    │
│ 3   │ 3     │ 7     │ 11    │ 15    │
│ 4   │ 4     │ 8     │ 12    │ 16    │

julia> parentindices(r)
(3, 2:3)

So in summary - the answer to your question is to use parentindices and get a first dimension from it.

However, as in Base, you get a row index in the source DataFrame which is not probably what you wanted - is this correct?

(if yes - then please comment and we can think of changing the design)

[pdeffebach]

For concreteness, this is the current behavior filtering on a subdataframe using rows.

julia> df = DataFrame(a = [1, 1, 2, 2],b = rand(4));

julia> gd = groupby(df, :a);

julia> combine(gd) do sdf
           filter(r -> getfield(r, :row) == 1, sdf)
       end
1×2 DataFrame
│ Row │ a     │ b       │
│     │ Int64 │ Float64 │
├─────┼───────┼─────────┤
│ 1   │ 1     │ 0.45063 │

[bkamins]

This is expected now (apart from the fact that field access is not part of the public API and rather parentindices(r)[1] should be used). My question is - do we want to change it?

[matthieugomez]

row_number is used in SQL and dplyr

[bkamins]

So the approach would be to add another field in DataFrameRow that would remember this value and add an exported method rownumber that would return it. Do you think we should also store the reference to the direct parent of DataFrameRow (rather than the true parent - as we currently do)?

[nalimilan]

As long as we don't provide any public function to access the direct parent, storing it doesn't make sense. So the question is really: should parent(row) return the direct parent? That sounds logical, though it adds yet another field. But maybe it doesn't really matter for performance since DataFrameRow is quite slow already due to type instability.

[bkamins]

parent(row) return the direct parent

It should not, as it is not consistent with Base. The design that we have now with parent and parentindices is made to match what Base does.

it adds yet another field. But maybe it doesn't really matter for performance since DataFrameRow is quite slow already due to type instability.

This was my thinking and that is why I have asked as we would have to add another field to return rownumber anyway.
Though we would need another function name for this, e.g. rowsource?

[bkamins]

Just to add parent in Base guarantees not to return view and this is the invariant we keep.

[bkamins]

also I would add rowsource only if we feel it is needed. I can see the need for rownumber, but maybe when someone works with DataFrameRow it is never needed to know rowsource as it is either obvious or not required?
The reason is that rownumber is Int and rowsource will add a parameter to DataFrameRow (leading to additional compilation passes in some casees and we already have 2 parameters there). Of course this is a minor issue, and if you feel rowsource is needed I will add it.

[pdeffebach]

This wouldn't work in transform(df, AsTable(:) => ByRow(row) => "rownum"), right? Since that is a Tables.namedtupleiterator rather than a DataFrameRow

w.r.t rowsource if you are taking a row from a SubDataFrame I can't imagine a use-case where you need to know the row number of the original data frame.