Grouped DataFrame with array elements fails to combine

[huangyxi]

When performing a combine operation on a DataFrame using the combine function with groupby, the output format is inconsistent based on the type of aggregation being applied. Specifically, when aggregating with the :v=>sum operation, the expected output should retain the array format for the column being aggregated. However, the current behavior results in a scalar value for the aggregated column.

julia> a = DataFrame(k=[1,1,2,2], v=[[1,2],[3,4],[5,6],[7,8]], t=[1,2,3,4])
4×3 DataFrame
 Row │ k      v       t
     │ Int64  Array…  Int64
─────┼──────────────────────
   1 │     1  [1, 2]      1
   2 │     1  [3, 4]      2
   3 │     2  [5, 6]      3
   4 │     2  [7, 8]      4

julia> combine(groupby(a, :k), :v=>sum)
4×2 DataFrame
 Row │ k      v_sum
     │ Int64  Int64
─────┼──────────────
   1 │     1      4
   2 │     1      6
   3 │     2     12
   4 │     2     14

julia> combine(groupby(a, :k), :t=>sum)
2×2 DataFrame
 Row │ k      t_sum
     │ Int64  Int64
─────┼──────────────
   1 │     1      3
   2 │     2      7

Expected:

julia> combine(groupby(a, :k), :v=>sum)
2×2 DataFrame
 Row │ k        v_sum
     │ Int64    Array…
─────┼────────────────
   1 │     1    [4, 6]
   2 │     2  [12, 14]

[huangyxi]

Another example:

julia> combine(groupby(a, :k), :t=>stack)
4×2 DataFrame
 Row │ k      t_stack
     │ Int64  Int64
─────┼────────────────
   1 │     1        1
   2 │     1        2
   3 │     2        3
   4 │     2        4

Expected behavior:

julia> combine(groupby(a, :k), :t=>stack)
4×2 DataFrame
 Row │ k      t_stack
     │ Int64   Array…
─────┼────────────────
   1 │     1   [1, 2]
   2 │     2   [3, 4]

I'm uncertain about the internal mechanism, but it appears that the DataFrame might undergo flattening after the combination of grouped DataFrames.

[huangyxi]

Alternative using Base.vect and Ref.

https://discourse.julialang.org/t/groupby-and-aggregate-a-dataframe-with-custom-function-that-return-a-vector/69976

[bkamins]

This is a design feature. The rule is that if function returns a vector it gets expanded. The reason is that in a vast majority of cases this is what users expect, and requiring them to flatten the result every time in this case would be inconvenient.
Note that even the simplest :a => identity requires flattening to produce a correct result.

It is important to understand that aggregation functions decide about how to handle the results on transformations based on the VALUE returned, not based on a function called. Relying on a function called would produce many special cases that would be even harder to learn.

Your case is rare (applying sum over vector of vectors) therefore the decision was that it should be handled by a special rule. As you have found, and as is written in the docstring:

In all of these cases, function can return either a single row or multiple rows. As a particular rule, values wrapped in a Ref or a 0-dimensional AbstractArray are unwrapped and then treated as a single row.

So you can write e.g. one of these (whichever is easier to remember for you):

julia> combine(groupby(a, :k), :v=>Ref∘sum)
2×2 DataFrame
 Row │ k      v_Ref_sum
     │ Int64  Array…
─────┼──────────────────
   1 │     1  [4, 6]
   2 │     2  [12, 14]

julia> combine(groupby(a, :k), :v=>fill∘sum)
2×2 DataFrame
 Row │ k      v_fill_sum
     │ Int64  Array…
─────┼───────────────────
   1 │     1  [4, 6]
   2 │     2  [12, 14]

To get what you want.

[huangyxi]

Thank you for your response. I have updated the document to ensure proper dissemination.