Normalize encoders

[Vincent-Maladiere]

Adresses #1253

[Vincent-Maladiere]

This PR is now ready for review. Items to review/discuss:

• The name BlockNormalizerL2 is not great; how should we name this class, @GaelVaroquaux? This norm looks close to the population standard deviation.
• The BlockNormalizerL2 class is public in our API, but we may not want to bother users with it. If we keep this class public, should we illustrate it with an example? Like the plot from Gael above.
• The TextEncoder, StringEncoder and GapEncoder now enable this normalization by default.
• The formula of the norm could be brought to the User Guide instead of the class docstring.
• BlockNormalizerL2 takes dataframes or numpy 2D array in; and numpy array out. The norm computation is done in numpy. This flow has two benefits:
  1. Avoid unnecessary data conversions when performing normalization within encoders.
  2. Avoid dispatching the norm computation logic between Polars, Pandas, or Numpy. Pandas and Polars have distinct ways of handling Nan values, for instance.

[Vincent-Maladiere]

Updates after IRL discussion with @GaelVaroquaux

• We don't want to introduce a normalizer object, even a private one
• We don't want to add new hyperparameters for encoders; normalization should be enforced
• We also don't want to replace fit with fit_transform in the GapEncoder. The issue is that we don't have access to the vectors at the end of fit, so we have to find some proxy, like the average sample length.

[GaelVaroquaux]

Could you benchmark/profile the relative cost of fit and transform please. If transform is costly, we'll find a solution

…

On May 22, 2025, 18:31, at 18:31, Vincent M ***@***.***> wrote: @Vincent-Maladiere commented on this pull request. > """ - Fit the GapEncoder on `X`. + _ = self.fit_transform(X) We need the vectors outputted by `.fit_transform` to compute the scaling factor. Since we don't have them during `.fit`, we have to recompute them (I also suspect the training part of the GapEncoder to be a significantly larger bottleneck than the transform part, but this could be benchmarked). -- Reply to this email directly or view it on GitHub: #1274 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[Vincent-Maladiere]

Here are the benchmarks (on the main branch); the GapEncoder is amazingly slow during transform. Therefore, we have to find a solution.

reproducer on the main branch

# %%
# Benchmarking GapEncoder.fit vs transform on the main branch. We want to make sure that
# transform << fit so that using fit_transform for fit doesn't add a major computation
# bottleneck.
#
# We define a synthetic text data generate. We can vary:
# - the number of samples
# - the average sequence length
# - the number of unique sample
#
import pandas as pd
import numpy as np
from tqdm import tqdm
import skrub
from time import perf_counter
from collections import defaultdict
from matplotlib import pyplot as plt
from functools import cache
from itertools import product


@cache
def make_synthetic_text(n_samples=1000, avg_ch_size=20, unique_ratio=0.7):
    n_samples_new = int(n_samples * unique_ratio)
    max_seqlen = 500

    ch_ids = np.random.randint(ord("A"), ord("z"), size=(n_samples_new, max_seqlen))
    seq_size = np.random.normal(
        avg_ch_size, scale=avg_ch_size / 4, size=n_samples_new
    ).astype("int32")

    samples = []
    for idx in range(n_samples_new):
        samples.append("".join(map(chr, ch_ids[idx, : seq_size[idx]])))
    samples.extend(np.random.choice(samples, size=n_samples - n_samples_new))

    return pd.Series(samples, name="synthetic_text")


def run_bench(results, n_samples, encoder, avg_ch_size, unique_ratio):

    encoder = getattr(skrub, encoder)()

    samples = make_synthetic_text(n_samples, avg_ch_size, unique_ratio)

    tic = perf_counter()
    encoder.fit(samples)
    results["fit"].append(perf_counter() - tic)

    tic = perf_counter()
    encoder.transform(samples)
    results["transform"].append(perf_counter() - tic)


params = dict(
    encoders=["StringEncoder", "GapEncoder"],
    avg_ch_size=[20],
    unique_ratio=[.7],
)
all_params = list(product(*params.values()))
all_results = []
all_n_samples = 10 ** np.array([2, 3, 4, 5])
for param_set in all_params:
    results = defaultdict(list)
    for n_samples in tqdm(all_n_samples):
        print(param_set)
        run_bench(results, n_samples, *param_set)
    all_results.append(results)

# %%

log_n_samples = np.log10(all_n_samples).astype("int32")
for idx, (encoder, _, _) in enumerate(all_params):
    fig, ax = plt.subplots()
    for method in "fit", "transform":
        ax.plot(log_n_samples, all_results[idx][method], label=method)
    ax.set_xticks(
        ticks=log_n_samples, labels=[f"1e{p}" for p in log_n_samples]
    )
    ax.legend()
    ax.grid()
    ax.set_xlabel("samples")
    ax.set_ylabel("seconds")
    ax.set_title(encoder)
    plt.show()

# %%

[Vincent-Maladiere]

I plotted the sequence length as produced by different analyzers against the row-wise L2 norm of the GapEncoder. Scaling these vectors by the average length seen during training could make sense.

reproducer

# Compute the text length for some columns.
# Plot the norm of the raw vectors vs the text length. 

import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns
from sklearn.feature_extraction.text import CountVectorizer

from skrub.datasets import fetch_employee_salaries
from skrub import GapEncoder


X = fetch_employee_salaries().X
categ_cols = ['employee_position_title', 'division', 'department_name']

for i, analyzer in enumerate(["char", "char_wb", "word"]):
    fig, axes = plt.subplots(ncols=3, figsize=(6, 3))
    for j, col in enumerate(categ_cols):
        gap = GapEncoder(analyzer=analyzer)
        Xt = gap.fit_transform(X[col].head(100))
        l2_norm = np.linalg.norm(Xt, axis=1)
        n_tokens = np.array(
            CountVectorizer(analyzer=analyzer)
            .fit_transform(X[col].head(100))
            .sum(axis=1)
        ).ravel()

        ax = axes[j]
        sns.scatterplot(x=n_tokens, y=l2_norm, ax=ax, color=["blue", "red", "green"][i])
        sns.lineplot(x=n_tokens, y=n_tokens, ax=ax, linestyle="--", color="grey", alpha=.4)

        ax.set_title(col)
        if i == 0 and j == 0:
            ax.set_ylabel('L2 Norm of Encoded Vectors')
            ax.set_xlabel('Text Length')
    fig.suptitle(analyzer)
    fig.tight_layout()

[GaelVaroquaux]

Scaling these vectors by the average length seen during training could make sense

Great! It seems like a very pragmatic solution and I really like what I see.

Thanks for finding this!!

[Vincent-Maladiere]

These experiments are invariant with the number of components!

When n_components=20

When n_components=100

[GaelVaroquaux]

Nice! Let's see what this gives on the histograms

[Vincent-Maladiere]

I ended-up with this heuristics:

scaling_factor = unq_V.sum(axis=1).mean() / 2

which works reasonably well:

employee_position_title column

toxicity (longer strings):

[GaelVaroquaux]

Yes, the histograms show that it works well. Let's go with that!

[GaelVaroquaux]

A few tiny comments and we are good to go

[GaelVaroquaux]

It's always good to have a top-level docstring for a file that describes briefly what the file does. It helps people reading the codebase

[GaelVaroquaux]

Ping me when the comments are addressed, and I merge

[Vincent-Maladiere]

They are! Could you look at the file-level docstring of _scaling_factor.py?

[GaelVaroquaux]

LGTM.

Thank you very much!!

[GaelVaroquaux]

Merged!!

[Vincent-Maladiere]

Wuhuuu!!