Why is COMET_DA default model?

[kocmitom]

Hi Unbabel,

I wanted to ask, why is the wmt20-comet-da default model when using COMET? I am a bit worried that people using it off the shelf won't understand the underlying difference and will mistakingly report COMET scores on the QE metric.

Why not set the reference-based COMET as the default, since it seems to be outperforming comet-da (also I fear that comet-da will have more biases and potential problems than reference-based).

Thank you for the answer,
Tom

[ricardorei]

Hi Tom, the default model is the reference-based model (wmt20-comet-da same as wmt-large-da-estimator-1719 from previous versions). The reference-free models are "tagged" with the qe on the name, e.g: wmt20-comet-qe-da.

The naming comet-da only has to do with the fact that the model was trained to predict DA z-scores.

[ricardorei]

This is the exact same model you used on your paper Ship not to ship:

What you used as COMET in your tables is the same as wmt20-comet-da and the COMET-src is the wmt20-comet-qe-da.

[kocmitom]

Hi Ricardo,

Oh, I really apologize, I should not be working nights. Thank you for your clarification.

Have a lovely end of the year!