no "index" column in aggtarget output

[jeromedockes]

ATM the AggJoiner and AggTarget when using value counts or histogram include an unwanted "index" column in their output.

After performing a groupby, the grouping column is in the index of the pandas result. to have it as a column and be able to join on it, we need to do a reset_index. skrub._dataframe._pandas.aggregate used to do it for histogram and value counts, but not for other aggregation functions. This went unnoticed/was not a problem because the aggjoiner and aggtarget used pandas.merge which then performed the merge on the index rather than a column, because it allows "right_on" to be either a column index or an index level index.

in #945 a reset_index was added, but in both for mean() and for value_counts(). So before the reset_index was done 0 times for mean(), 1 times for value_counts(); after it was done 1 time for mean() and 2 times for value_counts() -- the second one resulting in the "index" column.

what we want is to do reset_index once in all cases.

At least that's what I understand from a quick look, it would be great if @TheooJ and @Vincent-Maladiere can confirm

[Vincent-Maladiere]

Hey @jeromedockes, actually, I don't remember why we initially needed to use reset_index. If my main table has some custom indices (e.g., due to a split), I might want to keep these. For example, this could create silent errors in data wrangling use-cases out of a pipeline. WDYT?

import pandas as pd
from skrub import AggJoiner

main = pd.DataFrame({
    "airportId": [1, 2],
    "airportName": ["Paris CDG", "NY JFK"],
}, index=[2, 3])

aux = pd.DataFrame({
    "flightId": range(1, 7),
    "from_airport": [1, 1, 1, 2, 2, 2],
    "total_passengers": [90, 120, 100, 70, 80, 90],
    "company": ["DL", "AF", "AF", "DL", "DL", "TR"],
}, index=[10, 11, 12, 13, 14, 15])

AggJoiner(
    aux_table=aux,
    main_key="airportId",
    aux_key="from_airport",
    cols=["total_passengers", "company"],
    operations=["hist(4)", "mode"],
).fit_transform(main).index.tolist()

# >>> [0, 1], instead of [2, 3]

[jeromedockes]

As discussed IRL with @Vincent-Maladiere his comment above is correct but we'll open a separate issue about it: this PR is about the index of the aux table, whereas the comment is about preserving the index of the main table

[Vincent-Maladiere]

By the way, do we need to reset the index of the aux table? As we join on a key, the index shouldn't matter, should it?

[jeromedockes]

By the way, do we need to reset the index of the aux table? As we join on a key, the index shouldn't matter, should it?

after the groupby the grouping column ends up as the index, so we reset it to put it back as a column

[Vincent-Maladiere]

Right, LGTM then

[jeromedockes]

thanks @Vincent-Maladiere