[BUG] Faithfulness metric fails because the SummaCZS model is instantiated but never assigned

[sahilds1]

Describe the bug

A clear and concise description of what the bug is.

Faithfulness metric fails because the SummaCZS model is instantiated but never assigned

lighteval/src/lighteval/metrics/metrics_sample.py

Lines 698 to 699 in 53df859

	if self.summac is None:
	SummaCZS(granularity="sentence", model_name="vitc", imager_load_cache=False) # , device=device)

To Reproduce

Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

from lighteval.tasks.requests import Doc
from lighteval.metrics.metrics_sample import Faithfulness

document = "Scientists are studying Mars to learn about the Red Planet and find landing sites for future  missions. One possible site, known as Arcadia Planitia, is covered in strange sinuous features. The shapes could be signs that the area is actually made of glaciers, which are large masses of slow-moving ice. Arcadia Planitia is in Mars’ northern lowlands."

summary = "There are strange shape patterns on Arcadia Planitia. The shapes could indicate the area might be made of glaciers. This makes Arcadia Planitia ideal for future missions."

doc = Doc(query="", choices=[], gold_index=0, specific={"text": document})
Faithfulness().compute(formatted_doc=doc, predictions=[summary])

File lighteval/metrics/metrics_sample.py:706, in Faithfulness.compute(self, predictions, formatted_doc, **kwargs)
    704 if self.normalize_pred:
    705     prediction = self.normalize_pred(prediction)
--> 706 return self.summac.score_one(inp, prediction)["score"]

AttributeError: 'NoneType' object has no attribute 'score_one'

Expected behavior

A clear and concise description of what you expected to happen.

Faithfulness.compute(predictions, formatted_doc) to return a dict[str, float] of the faithfulness scores

Version info

Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

• Operating System: macOS Monterey Version 12.4
• Python Version: Python 3.11.1
• lighteval version: lighteval==0.10.0

[sahilds1]

I opened a PR to fix this issue: #795