Redesign DataFrame structure

[akharche]

Extension for #801

Implementation of new DataFrame structure based on lists instead of tuples
Improved df.count() codegen for testing
Example:

df = pd.DataFrame({'A': [1,2,3], 'B': [.5, .6, .7], 'C': [4, 5, 6], 'D': ['a', 'b', 'c']})

(['A', 'B', 'C', 'D'],)
([array([1, 2, 3], dtype=int64), array([4, 5, 6], dtype=int64)], [array([0.5, 0.6, 0.7])], [array(['a', 'b', 'c'], dtype=object)])

Reproduce:

@njit
def run_df():
    df = pd.DataFrame({'A': [1,2,3], 'B': [.5, .6, .7], 'C': [4, 5, 6], 'D': ['a', 'b', 'c']})

    print(df._columns)
    print(df._data)

    return df.count()

[pep8speaks]

Hello @akharche! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-04-30 18:36:58 UTC

[densmirn]

You mixed CamelCase and snake_case in one palce. I would propose to use CamelCase only for named tuples.
Also the definition of named tuple for me looks better in the following representation:

from typing import NamedTuple

class ColumnId(NamedTuple):
    type_id: int
    col_type_id: int

But it's up to you!

[densmirn]

I would unpack tuple something like this:

type_id, col_indices = data_typs_map[col_typ]

col_idx_list = len(col_indices)
df_structure[col_name] = ColumnId(type_id, col_idx_list)
col_indices.append(col_id)

[densmirn]

Suggested change

	for col_id, col_type in enumerate(fe_type.data):
	for col_type in fe_type.data:

[densmirn]

Are you sure that the order of types in the dict will always match the order of columns in DF?

[densmirn]

Maybe more reliable to create extra list of types as it was done in the model. Then use the list for ordered iterating.

[densmirn]

type_id looks like a counter. Is it ok that we rewrite it here? Maybe use another name for type_id in else-block and increment type_id only inside of if-block.

[densmirn]

Is there no problem with order of lists in the tuple?

[akharche]

Performance results of DataFrame with mixed types of columns

Columns	Current compile time, s	New compile time, s	New/Current
16	11.092775	9.713415	0.875652
32	28.611168	20.057998	0.701055
64	128.307209	67.369364	0.525062
128	803.671405	310.529887	0.386389

[densmirn]

Performance results of DataFrame with mixed types of columns

Columns Current compile time, s New compile time, s New/Current
4 11.092775 9.713415 0.875652
8 28.611168 20.057998 0.701055
16 128.307209 67.369364 0.525062
32 803.671405 310.529887 0.386389

The results look very good.

[AlexanderKalistratov]

@akharche examples failed