ENH Improves efficiency of ColumnTransformer for string keys

[thomasjpfan]

Reference Issues/PRs

Fixes #16327

What does this implement/fix? Explain your changes.

Uses the pandas index to find column indices

Any other comments?

Is a benchmark for this PR

from sklearn.preprocessing import FunctionTransformer
from sklearn.compose import make_column_transformer
import numpy as np
import pandas as pd

n_features = 10_000
selected_features = 7_000
columns = [f'col_{i}' for i in range(n_features)]
df = pd.DataFrame(np.ones((1, n_features)), columns=columns)

ct = make_column_transformer(*[(FunctionTransformer(), [col])
                               for col in columns[:selected_features]])

This PR

%%timeit
ct.fit_transform(df)
# 8.13 s ± 76 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

master

%%timeit
ct.fit_transform(df)
# 16.4 s ± 62 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[rth]

LGTM, aside for one comment below. Thanks @thomasjpfan !

[rth]

Do we have some test, to gracefully fail with duplicate columns names, where get_loc wouldn't behave as expected?

[thomasjpfan]

We do not have a test for this behavior. The original behavior uses index, which returns the first appearance of the column name.

I would consider this a bug and error.

[thomasjpfan]

Updated PR with a check on the return of get_loc.

[jnothman]

Please add an Efficiency entry to the change log

[glemaitre]

Just some nitpicks otherwise LGTM

[glemaitre]

Suggested change

	if not isinstance(col_idx, int):
	if not isinstance(col_idx, numbers.Integer):

[thomasjpfan]

numbers.Integer does not exist :(

[glemaitre]

oh Integral sorry

[glemaitre]

Thanks @thomasjpfan