Skip to content

Conversation

@WeiKangda
Copy link
Contributor

Description

In lighteval/src/lighteval/metrics/metrics_sample.py line 1258, the MajAtN.compute method calls:

return self.compute_score(new_model_response, new_doc)

However, compute_score (e.g., MultilingualExtractiveMatchMetric.compute) expects the signature:

def compute(self, doc: Doc, model_response: ModelResponse) -> float

As a result, the arguments are swapped, which cause runtime errors.

To Reproduce

Steps to reproduce the behavior:

  1. Install lighteval version 0.12.2.
  2. Define a metric using MajAtN with a subclass of MultilingualExtractiveMatchMetric.
  3. Call compute on a sample Doc and ModelResponse.
  4. Observe a TypeError or incorrect metric values due to the argument order mismatch.

Expected Behavior

MajAtN.compute should pass arguments in the correct order:

return self.compute_score(new_doc, new_model_response)

so that compute_score(doc, model_response) executes correctly.

@HuggingFaceDocBuilderDev
Copy link
Collaborator

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@NathanHB NathanHB merged commit 391d5b4 into huggingface:main Nov 14, 2025
4 checks passed
@NathanHB NathanHB added the bug label Nov 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants