Predictions scores for the ZeroShotGPTClassifier

[Puturbold]

I am trying to use the ZeroShotGPTClassifier and evaluate the results based on the prediction score. Is there a way to get the prediction score?

The HuggingFace ZeroShotClassifier returns a dictionary with labels and scores as lists. I am looking for a way to score the labels. Otherwise, there is no way to evaluate the labels from the ZeroShotClassifier. Especially if it returns a random label when no label matches.

[OKUA1]

@Puturbold there is no way to obtain the prediction scores. HF ZeroShotClassifier is a true classifier, whereas we are using gen models to act as if they were classifiers, but in fact they are just generating a bunch of tokens. Of course we could "ask" the model to return the prediction scores, but that will unlikely represent the actual scores.

[Puturbold]

I appreciate your quick response. Do you have any suggestions for evaluating labels without a pre-labeled reference?

[OKUA1]

Probably manual evaluation would still be the best bet if no labeled data is available. Also having prediction scores would not necessarily help, as the model can be incorrect in a very confident way (a high score for a wrong label).