FEA Add TextEncoder

[Vincent-Maladiere]

closes #1047

Reference
#1047

What does this PR propose?

This PR wraps the SentenceTransformers library, in the same fashion as ragger-duck and embetter does.

Following the good empirical baselines of arxiv.org/abs/2312.09634, this PR also proposes to:

1. Add a PCA within the estimator, controlled by a n_components parameter. This also allows the class to stay consistent with the other skrub encoders and perform HP tuning more easily. Following the paper, we set n_components=30 by default.
2. Add a normalization option, with a default "l2". This simplifies cosine distance to dot products and brings embeddings as comparable features to distance-based estimators (e.g., NearestNeighbors). This is useful when other normalized features are used (fuzzy joining).

What the SentenceEncoder does not:

• perform weights or embedding quantization (kept for later research)
• set torch threading, which doesn't interact well with joblib and raises an error if called twice.
• choose a different embedding Pooling scheme (only mean pooling is implemented in SentenceTransformer)

See the implementation of the paper for more details.

Todo in another PR
Enriching an existing example from skrub with the SentenceEncoder.

[Vincent-Maladiere]

Pixi is extremely annoying. It has negative returns on productivity as far as I am concerned.

[Vincent-Maladiere]

I have an intriguing segfault due to pytorch, that I can't reproduce locally with pixi or mamba. I'd be happy to get feedback on this. I suspect pixi doesn't download torch along with sentence-transformers from Pypi.

[Vincent-Maladiere]

Ping @jovan-stojanovic :)

[jovan-stojanovic]

Great!

[Vincent-Maladiere]

@GaelVaroquaux can we merge this one?

[GaelVaroquaux]

I'm having a look right now. I played with the example on my computer. It's really cool

I've created a small PR to your branch here: Vincent-Maladiere#1
If you're happy with it, can you merge it please?

Also, there is a conflict with main, which means that we cannot merge

[GaelVaroquaux]

LGTM, but please merge my doc PR Vincent-Maladiere#1 :)

[jeromedockes]

Nice!!! thanks @Vincent-Maladiere ! very exciting -- let's try to release it soon :) 🚀

[GaelVaroquaux]

Very exciting indeed! Bravo!

[GaelVaroquaux]

Darn, we have a build failure on circleCI on main:

I'm restarting the build.

[jeromedockes]

Darn, we have a build failure on circleCI on main:

Yeah I restarted it once already :/ it doesn't seem related to a PR but example 7 getting killed later (at least the first failure I saw)

[GaelVaroquaux]

Yes indeed. It fails reliably. I fear that it might be related to the PR: some nasty interaction between examples.

We'll need to bisect to know whether it's related to the dependencies that are being installed, or the code that is being run. I've created an issue to track this: #1143

[glemaitre]

Memory issues?

[Vincent-Maladiere]

I can try to bisect on this tomorrow!