For scripts/setfit/run_fewshot.py, add warning for class imbalance w. accuracy

[tomaarsen]

Hello!

Pull request overview

• For scripts/setfit/run_fewshot.py, add a warning when the accuracy metric is used on an imbalanced test set.

Details

This PR is in response to #203, a PR on which AmazonCF was accidentally evaluated using accuracy rather than MCC. If that script was run again in the same way after this PR, then the following warning would pop up:

...
Test set: 5000
[sic]\setfit\scripts\setfit\run_fewshot.py:112: UserWarning: The test set has a class imbalance (label 1 w. 503 samples, label 0 w. 4497 samples), but is evaluated using `accuracy`, which may lead to an evaluation that does not correspond with true model performance.
  warnings.warn(
...

This would help people like myself from not realising that accuracy is used by default in scripts/setfit/run_fewshot.py, and that not all SetFit testsets are balanced.

Note that this is only for scripts/setfit/run_fewshot.py, i.e. only for development and testing purposes. I don't believe that we should bother actual users with this warning.

The warning is shown when the largest class is at least 50% larger than the smallest class.

cc: @blakechi

• Tom Aarsen

[blakechi]

Do you think we throw a warning whenever there is an unbalanced dataset is a good idea?

For me I would like to know any unbalanced dataset and the corresponding metric. Specifically, maybe we can take off this if statement to allow throwing warnings on other datasets that are unbalanced but with different metrics than "accuracy"?

[blakechi]

Same as above, maybe we can print out no matter what metric the dataset is using?
Like:

Suggested change

	", but is evaluated using `accuracy`, which may lead to an evaluation that does not correspond with true model performance.",
	f", but is evaluated using `{metric}`, which may lead to an evaluation that does not correspond with true model performance.",

[tomaarsen]

I see, I think those comments are good ideas. It would be more like a logging.info call than a warnings.warn, as it would just be informative at all times.

[blakechi]

Maybe it could be a warning since it pointing out there is an unbalanced dataset?
Just sharing thoughts :)

[tomaarsen]

I love the thoughts! My bad, I thought you proposed to always print the test set distribution, even when balanced.

[blakechi]

Sorry, forgive my limited language skills 😂

[blakechi]

Hi @tomaarsen,

Nice work! I left some comments. Would like to know your comments :)

[tomaarsen]

After implementing the changes proposed by @blakechi, the output is now as follows:

> python .\scripts\setfit\run_fewshot.py --datasets amazon_counterfactual_en


============== amazon_counterfactual_en ============
Using custom data configuration SetFit--amazon_counterfactual_en-83c4b3502f70d59a
Reusing dataset json ([sic])
Using custom data configuration SetFit--amazon_counterfactual_en-83c4b3502f70d59a
Reusing dataset json ([sic])
Test set: 5000
Evaluating amazon_counterfactual_en using 'accuracy'.
[sic]\scripts\setfit\run_fewshot.py:112: UserWarning: The test set has a class imbalance (label 1 w. 503 samples, label 0 w. 4497 samples).
  warnings.warn(


======== [sic]\scripts\setfit\results\paraphrase-mpnet-base-v2-CosineSimilarityLoss-logistic_regression-iterations_20-batch_16\amazon_counterfactual_en\train-2-0 =======
config.json not found in HuggingFace Hub
model_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.
...

Would love to hear your thoughts.

• Tom Aarsen

[blakechi]

LGTM! :)

[lewtun]

Thanks a lot for adding this warning @tomaarsen 🔥 !

We originally wrote the few-shot scripts for our paper's experiments, but adding this warning is indeed helpful for the general use - thanks!