Fast Clustering

[KennethEnevoldsen]

This creates an alternative fast clustering for MTEB. It is notably faster. It gains this performance by downsampling and only performing the embed step once. To then regain the robustness of creating clusters multiple times it samples from the embeddings to create clusters.

[isaac-chung]

Noticed that this PR includes some efforts for a consolidated multilingual MTEBScores class. Could that be a separate PR?

[isaac-chung]

Looking at the file names, does this mean that normal was 4x faster than the "fast" class? 🤔

[KennethEnevoldsen]

Yep in the case of Swedn it is about the same (it is the same amount of documents being embedded), the extra cost comes from the bootstrapping and fitting clusters.

[KennethEnevoldsen]

Noticed that this PR includes some efforts for a consolidated multilingual MTEBScores class. Could that be a separate PR?

Yeah, just had to consider it when designing the task, but I think you are right that it is indeed a separate PR.

[KennethEnevoldsen]

Thanks for the review @isaac-chung and @Muennighoff.

I have now:

• removed the MTEBScore (for another PR)
• Added the Fast clustering task, but keeping the old implementation
• Added two new datasets using the old implementation, adding a "superseeded_by" attribute to the old datasets, which will raise a warning.
• I have split swednClustering into P2P and S2S

I believe this is close to ready to merge, let me know what you think

edit: @imenelydiaker adding you here so that you see the "superseeded_by" implementation as well

[isaac-chung]

Most of my comments were on the MTEBScore class. I did not review the cluster datasets. The rest looks good to me 💪

[KennethEnevoldsen]

I will enable auto-merge on this PR

[Muennighoff]

Amazing work! Just noting that new models are now using this, e.g. in https://huggingface.co/spaces/mteb/leaderboard?task=clustering&language=french the voyage-law-2 model was run with fast clustering while other models were not. Is this okay?