Copy and grouby-sum changes column-major-ordering to row-major-ordering with huge impact on performance

[TannhauserGate42]

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd

print("pandas version: ", pd.__version__)

array = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
array.flags
print("Numpy array is C-contiguous: ", data.flags.c_contiguous)

dataframe = pd.DataFrame(array, index = pd.MultiIndex.from_tuples([('A', 'U'), ('A', 'V'), ('B', 'W')], names=['dim_one', 'dim_two']))
print("DataFrame is C-contiguous: ", dataframe.values.flags.c_contiguous)

dataframe_copy = dataframe.copy()
print("Copy of DataFrame is C-contiguous: ", dataframe_copy.values.flags.c_contiguous)

aggregated_dataframe = dataframe.groupby('dim_one').sum()
print("Aggregated DataFrame is C-contiguous: ", aggregated_dataframe.values.flags.c_contiguous)

## Output in Jupyter Notebook
# pandas version:  0.23.4
# Numpy array is C-contiguous:  True
# DataFrame is C-contiguous:  True
# Copy of DataFrame is C-contiguous:  False
# Aggregated DataFrame is C-contiguous:  False

Problem description

pandas.DataFrame.copy return a copy with underlying numpy array with the major-ordering changed from C to F if initially was C order

pandas.DataFrame.groupby.sum return a a DataFrame with underlying numpy array with the major-ordering changed from C to F if initially was C order on the DataFrames values.

This impacts performance on aggregation substantially! I observer a decrease from 17 seconds to 5 min 46 seconds when aggregating on the index on a 45023 times 100000 scenarios dataset.

Copy and Aggregation operations need to preserve the major-order on the underlying data!

Expected Output

# Output in Jupyter Notebook
 pandas version:  0.23.4
 Numpy array is C-contiguous:  True
 DataFrame is C-contiguous:  True
 Copy of DataFrame is C-contiguous:  True
 Aggregated DataFrame is C-contiguous:  True

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-862.14.4.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 3.2.3
pip: 19.0.2
setuptools: 40.8.0
Cython: 0.22
numpy: 1.15.4
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.6.3
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: 3.5.1
numexpr: 2.6.9
feather: None
matplotlib: 2.2.2
openpyxl: 2.4.7
xlrd: 1.1.0
xlwt: None
xlsxwriter: 1.0.2
lxml: None
bs4: None
html5lib: None
sqlalchemy: 1.3.3
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[jreback]

dataframes are stored as f-contiguous, if you are passing in a numpy array that is not and you have a single dtyped backing (meaning you passed a 2-d array in the first place) it will be preserved to some extent. The current design of pandas does not lead to any kind of guarantees about the back store. You can investigate if you want, but not sure anything could actually be done without breaking the world.

in any event your example doesn't show any performance difference, please edit to show the 2 cases that you claim have a difference.

[TannhauserGate42]

Hello Jeff,
Many thanks for replying.

I start reading a DataFrame from an pandas.HDFStore. It is read in C-contiguous. I wrote it into that store, out from another fully Python based application.

In the following an example with larger datasets. It shows nicely the performance drawdown. I run it in a Jupyther notebook on a 128GByte Jupyther Hub hosted on one of our machines.

The data delivered in usual use cases has 100000 scenarios, the columns axis in our case. In the example below I use 10000 only.

Here the example:

import numpy as np
import pandas as pd

names = ['super', 'natural', 'multi', 'index', 'construction']
index = pd.MultiIndex.from_product([list(name) for name in names], names=names)
columns = pd.RangeIndex(10000)
values = np.random.randn(len(index), len(columns))

def major_order(values):
    '''
    Return the major ordering as string 'C' or 'F'.

    :param values: `numpy.ndarray` or `pandas.DataFrame`
    :return: `str`
    '''
    if isinstance(values, pd.DataFrame):
        values = values.values
    return {True: 'C', False: 'F'}[values.flags.c_contiguous]

print('Initial arrays` major-order: ', major_order(values))
data = pd.DataFrame(values, index=index, columns=columns)
print('Initial dataframes` major-order: ', major_order(values))
print('Shape of data: ', data.shape)

#%%timeit
aggregated = data.groupby(['natural', 'multi', 'index']).sum()
# 228 ms ± 9.87 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

print('Shape of aggregated dataframe: ', aggregated.shape)
print('Aggregated datas` major-order: ', major_order(aggregated))
data_copy = data.copy()
print('Data copy major-order: ', major_order(data_copy))

# %%timeit
aggregated_of_copy = data_copy.groupby(['natural', 'multi', 'index']).sum()
# 3.86 s ± 92.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

aggregated_of_copy = data_copy.groupby(['natural', 'multi', 'index']).sum()
print('Data copy major-order: ', major_order(aggregated_of_copy))

[jreback]

@TannhauserGate42 you are welcome to have a look

[TannhauserGate42]

@TannhauserGate42 you are welcome to have a look

Thanks a lot! Does this mean, I shall fix it?

[jreback]

it means that if you can do something in a reasonable way w/o breaking anything else then it could be considered

[TannhauserGate42]

Thanks, will give it a try next week when I'm back in the office ...

I think, a behaviour aligned with the one expected from numpy should be envisaged.

Help on built-in function array in module numpy:
array(...)
    array(object, dtype=None, copy=True, order='K', subok=False, ndmin=0)
    
    Create an array.
    
    Parameters
    ----------
    object : array_like
        An array, any object exposing the array interface, an object whose
        __array__ method returns an array, or any (nested) sequence.
    dtype : data-type, optional
        The desired data-type for the array.  If not given, then the type will
        be determined as the minimum type required to hold the objects in the
        sequence.  This argument can only be used to 'upcast' the array.  For
        downcasting, use the .astype(t) method.
    copy : bool, optional
        If true (default), then the object is copied.  Otherwise, a copy will
        only be made if __array__ returns a copy, if obj is a nested sequence,
        or if a copy is needed to satisfy any of the other requirements
        (`dtype`, `order`, etc.).
    order : {'K', 'A', 'C', 'F'}, optional
        Specify the memory layout of the array. If object is not an array, the
        newly created array will be in C order (row major) unless 'F' is
        specified, in which case it will be in Fortran order (column major).
        If object is an array the following holds.
    
        ===== ========= ===================================================
        order  no copy                     copy=True
        ===== ========= ===================================================
        'K'   unchanged F & C order preserved, otherwise most similar order
        'A'   unchanged F order if input is F and not C, otherwise C order
        'C'   C order   C order
        'F'   F order   F order
        ===== ========= ===================================================
    
        When ``copy=False`` and a copy is made for other reasons, the result is
        the same as if ``copy=True``, with some exceptions for `A`, see the
        Notes section. The default order is 'K'.

[TannhauserGate42]

it means that if you can do something in a reasonable way w/o breaking anything else then it could be considered

Ok, so I did. Now I'd like to make a pull request but didn't succeed with pushing my branch upstream. Do I have the appropriate user rights. How shall I proceed. Sorry, for that kind of questions.

[TomAugspurger]

http://pandas.pydata.org/pandas-docs/stable/development/contributing.html

…

On May 26, 2019, at 06:39, Thomas Siller ***@***.***> wrote: > it means that if you can do something in a reasonable way w/o breaking anything else then it could be considered Ok, so I did. Now I'd like to make a pull request but didn't succeed with pushing my branch upstream. Do I have the appropriate user rights. How shall I proceed. Sorry, for that kind of questions. -- You are receiving this because you are subscr