getting error in _faithfulness metric

[bhaskarturkar]

Getting below error for faithfulness metric after executing the code
from ragas import evaluate

result = evaluate(
    fiqa_eval["baseline"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
        harmfulness,
    ],
)
result

[shahules786]

Hey @bhaskarturkar can you please share the ragas version?

[bhaskarturkar]

Hi @shahules786,
I am getting this error for ragas version 0.1.2.

[daveyko]

i am also getting this error

here are the distributions

distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}

interestingly this doesn't happen for non multi_context

[shahules786]

Hey @daveyko this is a different issue, your issue will be resolved once you update to the latest version.

[daveyko]

@shahules786 its happening to me on the latest version 0.1.2

[shahules786]

@daveyko are you facing the same issue as described above?

[daveyko]

yup same issue im getting
assert isinstance(statements, dict), "Invalid JSON response"

in
File "/Users/davidko/.pyenv/versions/3.10.13/lib/python3.10/site-packages/ragas/metrics/_faithfulness.py", line 190, in _ascore

[daveyko]

this is when i run evaluate