Implement equivalent of Pandas qcut()

[navdeep-G]

• Describe what feature you would like to see implemented.

Pandas equivalent of qcut(): https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html

• If possible, give an example of how it may look in the code and what result
  will be produced.

Similar to Pandas qcut(). Maybe some improvements involved API wise but unsure of those at the moment.

[st-pasha]

In pandas, qcut produces a categorical column.
We don't have categoricals just yet, but we can produce integer codes. Would this be acceptable? Also, once we add support for categoricals, the return type of this function would probably have to be changed.

In terms of API, I'm thinking of simple method that applies to a column expression and produces a quantile encoding:

DT[:, f.A.qcut(q=4)]

[navdeep-G]

@st-pasha Yes, DT[:, f.A.qcut(q=4)] seems like a good API to start with.

However, I am not sure how integer codes would work in this case. Can you give an example of what you expcet the output to look like given the current datatable framework?

[st-pasha]

qcut() is supposed to convert any int/real variable into quantiles. So for example range(12) cut into 4 quartiles would convert into [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3].

[oleksiyskononenko]

What about using Aggregator for this purpose?

>>> import datatable as dt
>>> from datatable.models import aggregate
>>> df = dt.Frame(range(12))
>>> [df_agg, df_map] = aggregate(df, min_rows = 0, n_bins = 4)
>>> df_map
     exemplar_id
---  -----------
 0             0
 1             0
 2             0
 3             1
 4             1
 5             1
 6             2
 7             2
 8             2
 9             3
10             3
11             3

[st-pasha]

@oleksiyskononenko Aggregator doesn't have such a precise semantics. For example,

>>> df = dt.Frame([0.0, 0.1, 0.1, 0.1, 0.4, 0.8, 0.9, 0.99, 0.95, 0.91])
>>> [df_agg, df_map] = aggregate(df, min_rows=0, n_bins=4)
>>> df_map
    exemplar_id
--  -----------
 0            0
 1            0
 2            0
 3            0
 4            1
 5            2
 6            2
 7            2
 8            2
 9            2

[10 rows x 1 column]

Note that in this case the fourth quartile isn't even produced.

[st-pasha]

@navdeep-G Will integer codes work for you, or should we wait with this issue until we have categorical column type?

[navdeep-G]

@st-pasha I am not sure if integer codes would work for me. I think waiting for categorical types would be best. Is there an issue for this? We can tag it here as a dependency.

[st-pasha]

Prerequisite: #1691

[oleksiyskononenko]

@st-pasha Yes, that’s because I compress it at the end. Could be easily disabled or enabled depending on an input parameter.

[mdymczyk]

@navdeep-G @st-pasha actually pandas.qcut creates either categoricals or integers but that's not important :-)

I think we (or at least MLI) should be ok with anything that describes to which group a given value belongs to so integers should be fine.

Currently, we're using the group boundaries i.e. [[0, 1), [1,2], (2,3]] etc. as extra training features, but if we instead use [0,1,2] it should give us the same result.