Skip to content

Nvidia Accuracy calculation code is not consistent with the docs #2000

New issue
New issue
Open
#2007
Open
Nvidia Accuracy calculation code is not consistent with the docs#2000
#2007
Labels
bugSomething isn't workingmodule-metricsthis is part of metrics module
@vitorsabbagh

Description

@vitorsabbagh
vitorsabbagh
opened on Apr 8, 2025

[ x ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
In the Nvidia Accuracy metric calculation, the User Answer and Reference Answer were supposed to be swapped (according to docs) in steps 1 and 2. However, it's not really swapping, as both titles and values are being swapped.

Ragas version: 0.2.14

Activity

vitorsabbagh
added
bugSomething isn't working
on Apr 8, 2025
dosubot
added
module-metricsthis is part of metrics module
on Apr 8, 2025
sahusiddharth

sahusiddharth commented on Apr 13, 2025

@sahusiddharth
sahusiddharth
on Apr 13, 2025
Contributor

Hi @vitorsabbagh, are you just referring to the example in the docs?

the example calculation is not the best example because both the response and reference is the same.

If we look under the hood, it swapping is happening.

  1. https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_nv_metrics.py#L109
formatted_prompt = StringPromptValue(
                    text=self.template_accuracy1.format(
                        query=sample.user_input,
                        answer0="User Answer",
                        answer1="Reference Answer",
                        sentence_inference=sample.response,
                        sentence_true=sample.reference,
                    )
                )
  1. https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_nv_metrics.py#L132
                formatted_prompt = StringPromptValue(
                    text=self.template_accuracy2.format(
                        query=sample.user_input,
                        answer0="Reference Answer",
                        answer1="User Answer",
                        sentence_inference=sample.reference,
                        sentence_true=sample.response,
                    )
                )

notice answer0 and answer1 they are getting swapped.

vitorsabbagh

vitorsabbagh commented on Apr 13, 2025

@vitorsabbagh
vitorsabbagh
on Apr 13, 2025
Author

Hi @sahusiddharth, you're right! However, notice that sentence_inference and sentence_true are also being swapped. It's a double swap.

sahusiddharth

sahusiddharth commented on Apr 13, 2025

@sahusiddharth
sahusiddharth
on Apr 13, 2025
Contributor

Yes @vitorsabbagh, you are correct thank-you letting us know. I will raise a PR to fix this.

DmitrySirakov
linked a pull request that will close this issue on Apr 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmodule-metricsthis is part of metrics module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @vitorsabbagh@sahusiddharth

      Issue actions
        Nvidia Accuracy calculation code is not consistent with the docs · Issue #2000 · explodinggradients/ragas