Bug with complex groupby on copied columns

[Garra1980]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux
• Modin version (modin.__version__):
• Python version:
• Code we can use to reproduce:

import modin.pandas as mpd
import numpy as np

d = {'col1': [1, 2], 'col2': [3, 4], 'col3': [10,9], 'col4': [11,12], 'col5': [13,15]}

mdf = mpd.DataFrame(data=d)
def col5(x):
    return np.sum(x)

def min5_agg(x):
    return np.sum(np.abs(x))

def col6(x):
    return np.min(x)

def col7(x):
    return np.min(x)

agg_func = {
    'col5': [col5, min5_agg], 
    'col6': col6, 
    'col7': col7
}

mdf['col6'] = mdf['col2']
mdf['col7'] = mdf['col3']

res_m=mdf.groupby(['col1']).agg(agg_func)

res_m.columns.droplevel(0)   <--- Value Error: Cannot remove 1 levels from an index with 1 levels: at least one level must be left.

Describe the problem

A bit weird bug found in one of use cases of TPC-AI benchmark
Modin dataframe after groupby is incorrect:

      (col5, col5)  (col5, min5_agg)  col6  col7
col1
1               13                13     3    10
2               15                15     4     9

Pandas gives

     col5          col6 col7
     col5 min5_agg col6 col7
col1
1      13       13    3   10
2      15       15    4    9

Weirdness is that bug is reproduced only in case

mdf['col6'] = mdf['col2']
mdf['col7'] = mdf['col3']

persists in the code. Without copied columns and aggregating on them no error occurs.

Source code / logs

[Garra1980]

Seems it is also important that aggregation functions are specified as it is done in the test

[pyrito]

@Garra1980 thanks for opening the issue! I tried running the snippet and was able to reproduce this behavior locally. Like you said, the strange coupling of the columns seems to happen if we set the columns the way you mentioned. If we do something like this things seem to be working:

import modin.pandas as mpd
import numpy as np

# Initialize col6 and col7 up here instead statically
d = {'col1': [1, 2], 'col2': [3, 4], 'col3': [10,9], 'col4': [11,12], 'col5': [13,15], 'col6': [17,38], 'col7': [71,32]}

mdf = mpd.DataFrame(data=d)
def col5(x):
    return np.sum(x)

def min5_agg(x):
    return np.sum(np.abs(x))

def col6(x):
    return np.min(x)

def col7(x):
    return np.min(x)

agg_func = {
    'col5': [col5, min5_agg], 
    'col6': col6, 
    'col7': col7
}

res_m=mdf.groupby(['col1']).agg(agg_func)

res_m.columns.droplevel(0)

The Modin team will take a look and find a solution to the issue! cc: @modin-project/modin-contributors @modin-project/modin-core

[Garra1980]

yeah, something like that

[anmyachev]

Problem: when used to aggregate a list of functions, the end result must be a multicolumn. When a Modin dataframe contains several partitions, a situation may arise when a multicolumn occurs in one partition and not in another. The current implementation does not handle this case.

Solution: we can find out if the end result should be a multicolumn before executing the aggregation functions, simply by looking at the type of the variable. If this is the case, then it is possible for columns to which only one function is applied, to wrap it in list type, which will lead to the creation of a multicolumn in all partitions.

Code example:

            func_dict = {col: try_get_str_func(fn) for col, fn in func_dict.items()}
            if any((isinstance(value, list) for value in func_dict.values())):
                # multicolumn case
                new_func_dict = {col: fn if isinstance(fn, list) else [fn] for col, fn in func_dict.items()}
                func_dict = new_func_dict