Unbabel/wmt22-comet-da model not working as part of Huggingface evaluate

[alexandremuzio]

🐛 Bug

I tried using Huggingface evaluate but it is not working with the newest Unbabel/wmt22-comet-da as part of unbabel-comet 2.0.0 release. I'm not sure if this is an issue with Huggingface evaluate lib or COMET.

To Reproduce

Please check this collab notebook.

Screenshots

The issue:

.

Environment

OS: Linux
Version 2.0.0

[ricardorei]

It seems like Hugging face evaluate has to be updated to support recent comet versions. This is a bug on evaluate and not on COMET.

I'll try to find some time to investigate whats wrong with the current implementation of COMET in evaluate.

[ricardorei]

I created a pull request to evaluate to solve this: huggingface/evaluate#443

Once it is merged, it should work.

Basically it has to do with an API change. You can see the difference here