[QUESTION] About HTER models in download list.

[ElliottYan]

❓ Questions and Help

Before asking:

1. Search for similar issues.
2. Search the docs.

What is your question?

Hi, I found that the HTER models are off the download list of the current codes.
https://github.com/Unbabel/COMET/blob/master/comet/models/__init__.py
I wonder whether they are still supported in the current version.

I used version 1.0.0rc9, and it report this.
"Exception: wmt-large-hter-estimator is not in the availale_metrics or is a valid checkpoint folder."
Is that normal or should I use the previous version?
Thanks.

Code

What have you tried?

What's your environment?

• OS: Linux
• Packaging pip
• Version 1.0.0rc9

[ricardorei]

Hey @ElliottYan
I was planning to remove those as they were outperformed by the DA models but I can add them back!

For now, you will have to resort to the previous version... Those checkpoints still depend on fairseq but I have a script to convert the models into the new codebase (with transformers)

[ElliottYan]

I see. Thanks for your reply!

As I need a bounded score, I will try the previous version for now, but I would really appreciate it if you add them back to the current one.

In addition, I see you explain about HTER models are not as good as DA ones. Can I ask how bad are we talking about? Is there any result reported that I missed?

Thanks

[ricardorei]

we are not talking about a huge performance decrease but its usually a few points behind. You can look for example at the WMT20 Metrics task findins where we submitted several COMET versions. I also know that the authors of the To Ship or Not to Ship compared the DA model with the HTER model. The HTER model was even outperformed by the reference-less DA model (wmt20-comet-qe-da)

[ricardorei]

Btw the wmt20-comet-qe-da is also bounded with a sigmoid. If you only care about having a bounded score this one might be useful.

I'll add the HTER models meanwhile.

[ElliottYan]

Oh, great! The "wmt20-comet-qe-da" model also performs very well in the paper, and I will try that.
Thanks!