Add Vaani Multilingual langauge detection and gender classification task

[anime-sh]

Code Quality

• Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

• Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

• New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
• Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

Adding datasets checklist

Reason for dataset addition: Add multilingual speech task

• I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
  • "facebook/wav2vec2-xls-r-300m"
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

{
  "dataset_revision": "b133cc2e158905798a723f29685e483517e61275",
  "task_name": "IISc-Vaani-Language-Detection",
  "mteb_version": "1.36.20",
  "scores": {
    "train": [
      {
        "accuracy": 0.88094,
        "f1": 0.11324,
        "lrap": 0.973541,
        "scores_per_experiment": [
          {
            "accuracy": 0.88094,
            "f1": 0.11324,
            "lrap": 0.973541
          },
          {
            "accuracy": 0.88094,
            "f1": 0.11324,
            "lrap": 0.973541
          }
        ],
        "main_score": 0.88094,
        "hf_subset": "WestBengal_Purulia",
        "languages": [
          "ben-Beng",
          "hin-Deva",
          "rjs-Beng",
          "sat-Olck"
        ]
      }
    ]
  },
  "evaluation_time": 285.6733286380768,
  "kg_co2_emissions": null
}

{
  "dataset_revision": "b133cc2e158905798a723f29685e483517e61275",
  "task_name": "IISc-Vaani-Gender-Classification",
  "mteb_version": "1.36.20",
  "scores": {
    "train": [
      {
        "accuracy": 0.620996,
        "f1": 0.618563,
        "f1_weighted": 0.618696,
        "ap": 0.572232,
        "ap_weighted": 0.572232,
        "scores_per_experiment": [
          {
            "accuracy": 0.639648,
            "f1": 0.638888,
            "f1_weighted": 0.63905,
            "ap": 0.586072,
            "ap_weighted": 0.586072
          },
          {
            "accuracy": 0.627441,
            "f1": 0.627416,
            "f1_weighted": 0.627446,
            "ap": 0.575327,
            "ap_weighted": 0.575327
          },
          {
            "accuracy": 0.602051,
            "f1": 0.601364,
            "f1_weighted": 0.601203,
            "ap": 0.556079,
            "ap_weighted": 0.556079
          },
          {
            "accuracy": 0.631836,
            "f1": 0.621147,
            "f1_weighted": 0.621769,
            "ap": 0.585662,
            "ap_weighted": 0.585662
          },
          {
            "accuracy": 0.604004,
            "f1": 0.603999,
            "f1_weighted": 0.604012,
            "ap": 0.558021,
            "ap_weighted": 0.558021
          }
        ],
        "main_score": 0.620996,
        "hf_subset": "WestBengal_Purulia",
        "languages": [
          "ben-Beng",
          "hin-Deva",
          "rjs-Beng",
          "sat-Olck"
        ]
      }
    ]
  },
  "evaluation_time": 824.7135016918182,
  "kg_co2_emissions": null
}

Adding a model checklist

• I have filled out the ModelMeta object to the extent possible
• I have ensured that my model can be loaded using
  • mteb.get_model(model_name, revision) and
  • mteb.get_model_meta(model_name, revision)
• I have tested the implementation works on a representative set of tasks.

[Samoed]

Suggested change

	self.label_column_name: literal_eval(x[self.label_column_name])
	self.label_column_name: json.loads(x[self.label_column_name])

[Samoed]

Since this dataset have only train I think this should implement cross_validation for Multilabel datasets similarly as for classification

[anime-sh]

the each subset has a train set which has 100k samples, crossfold with normal Ks (3,5,7) will be a quite expensive
One idea i had was to generate a test split from the train split in dataset transform and then proceed as a normal task

[KennethEnevoldsen]

@anime-sh will close this PR as it seems to have gotten stale