TypeError: comet.models.utils.Prediction is not a dataclasss. This is a subclass of ModelOutput and so must use the @dataclass decorator.

[devrimcavusoglu]

🐛 Bug

unbabel-comet==2.1.0
Related CI: https://github.com/obss/jury/actions/runs/6409370286/job/17400400518

Traceback

..\..\..\jury\core.py:86: in __call__
    score = self._compute_single_score(inputs)
..\..\..\jury\core.py:156: in _compute_single_score
    score = metric.compute(predictions=predictions, references=references, reduce_fn=reduce_fn, **kwargs)
E:\devrim\miniconda3\envs\jury\lib\site-packages\evaluate\module.py:444: in compute
    output = self._compute(**inputs, **compute_kwargs)
..\..\..\jury\metrics\_core\base.py:551: in _compute
    result = self.evaluate(
..\..\..\jury\metrics\_core\base.py:276: in evaluate
    return eval_fn(predictions=predictions, references=references, **kwargs)
..\..\..\jury\metrics\comet\comet_for_language_generation.py:172: in _compute_single_pred_single_ref
    comet_scores = self.scorer.predict(
E:\devrim\miniconda3\envs\jury\lib\site-packages\comet\models\base.py:627: in predict
    predictions = trainer.predict(
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\trainer\trainer.py:852: in predict
    return call._call_and_handle_interrupt(
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\trainer\call.py:43: in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\trainer\trainer.py:894: in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path)
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\trainer\trainer.py:980: in _run
    results = self._run_stage()
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\trainer\trainer.py:1018: in _run_stage
    return self.predict_loop.run()
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\loops\utilities.py:181: in _decorator
    return loop_run(self, *args, **kwargs)
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\loops\prediction_loop.py:112: in run
    self._predict_step(batch, batch_idx, dataloader_idx)
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\loops\prediction_loop.py:229: in _predict_step
    predictions = call._call_strategy_hook(trainer, "predict_step", *step_kwargs.values())
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\trainer\call.py:294: in _call_strategy_hook
    output = fn(*args, **kwargs)
E:\devrim\miniconda3\envs\jury\lib\site-packages\pytorch_lightning\strategies\strategy.py:413: in predict_step
    return self.model.predict_step(*args, **kwargs)
E:\devrim\miniconda3\envs\jury\lib\site-packages\comet\models\base.py:430: in predict_step
    model_outputs = Prediction(scores=self(**batch).score)
E:\devrim\miniconda3\envs\jury\lib\site-packages\torch\nn\modules\module.py:1518: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
E:\devrim\miniconda3\envs\jury\lib\site-packages\torch\nn\modules\module.py:1527: in _call_impl
    return forward_call(*args, **kwargs)
E:\devrim\miniconda3\envs\jury\lib\site-packages\comet\models\regression\regression_metric.py:273: in forward
    return self.estimate(src_sentemb, mt_sentemb, ref_sentemb)
E:\devrim\miniconda3\envs\jury\lib\site-packages\comet\models\regression\regression_metric.py:245: in estimate
    return Prediction(score=self.estimator(embedded_sequences).view(-1))

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Prediction([('score', tensor([0.4872, 0.4201]))]), args = ()
kwargs = {'score': tensor([0.4872, 0.4201])}, is_modeloutput_subclass = True

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
    
        # Subclasses of ModelOutput must use the @dataclass decorator
        # This check is done in __init__ because the @dataclass decorator operates after __init_subclass__
        # issubclass() would return True for issubclass(ModelOutput, ModelOutput) when False is needed
        # Just need to check that the current class is not ModelOutput
        is_modeloutput_subclass = self.__class__ != ModelOutput
    
        if is_modeloutput_subclass and not is_dataclass(self):
>           raise TypeError(
                f"{self.__module__}.{self.__class__.__name__} is not a dataclasss."
                " This is a subclass of ModelOutput and so must use the @dataclass decorator."
            )
E           TypeError: comet.models.utils.Prediction is not a dataclasss. This is a subclass of ModelOutput and so must use the @dataclass decorator.

E:\devrim\miniconda3\envs\jury\lib\site-packages\transformers\utils\generic.py:327: TypeError

[dmar1n]

Hi @devrimcavusoglu, I encountered the same problem and tentatively applied the same solution as in your PR (i.e. adding the @dataclass decorator to the classes of models.utils inheriting from Transformers' ModelOutput, as indicated in the Transformers doc). However, in my case, this produces another error later, when training a unified metric model, because the Target class is passed in the scores, and as a dataclass, it does not accept any non-initialized argument (not declared in the Target class definition). I'm not sure if this is an issue with Transformers or COMET, but any hints would be appreciated.

Traceback

TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/mt/anaconda3/envs/comet/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/mt/anaconda3/envs/comet/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/opt/mt/anaconda3/envs/comet/lib/python3.10/site-packages/comet/models/base.py", line 511, in <lambda>
    collate_fn=lambda s: self.prepare_sample(s, stage="validate"),
  File "/opt/mt/anaconda3/envs/comet/lib/python3.10/site-packages/comet/models/multitask/unified_metric.py", line 362, in prepare_sample
    targets = Target(score=torch.tensor(scores, dtype=torch.float))
TypeError: Target.__init__() got an unexpected keyword argument 'score'

[lx0126z]

try this:
pip install unbabel-comet==2.0.1 --force-reinstall --no-cache-dir

[aRyBernAlTEglOTRO]

Hi @dmar1n, I think you can try adding the @dataclass(init=False) decorator to the classes of models.utils inheriting from Transformers' ModelOutput.

[dmar1n]

Thanks a lot, @aRyBernAlTEglOTRO, for the hint! I have run a quick test, and I can confirm this is actually solving the issue I explained above.

When possible, I will propose a PR with this change; otherwise, the changes implemented in 2.1 won't work for training a UniTE model, and maybe other models.

[ricardorei]

I'll try to bump the package version in the next few weeks.

[geknow]

@ricardorei @devrimcavusoglu The change is overrided.

[ricardorei]

I copied the old ModelOutput class from hugging face to utils. This error should not happen anymore