add LlamaIndex faithfulness metric #971
Conversation
src/unitxt/metrics.py
Outdated
| @@ -2254,6 +2251,53 @@ def compute( | |||
| } | |||
|
|
|||
|
|
|||
| class LlamaIndexFaithfulness(LlamaIndexCorrectness): | |||
There was a problem hiding this comment.
There is a lot of duplicated code between the metrics. Can you please create a common base class (e.g. LlamaIndexMetric), and more the code there?
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
src/unitxt/metrics.py
Outdated
| """LlamaIndex based metric class for evaluating faithfulness.""" | ||
|
|
||
| metric_name_prefix = "faithfulness" | ||
| reference_field = "contexts" # metric doesn't require reference answers |
There was a problem hiding this comment.
Is this needed? I don't see it being used.
There was a problem hiding this comment.
As this is a referenceless metric, this is there to prevent failures of InstanceMetric where no "references" field was provided
There was a problem hiding this comment.
(
Line 877 in 1242b02
There was a problem hiding this comment.
Thanks for the quick replies on the CR comments. I see. But a "references" are always passed in unitxt - even for reference-less metrics. How did you get a failure?
There was a problem hiding this comment.
Right, that is a different discussion that is outside the scope of this pull request... So if you want I can remove this
There was a problem hiding this comment.
Yes. For now, please do (it will avoid confusion).
There was a problem hiding this comment.
Done. I will say on a more general note that as unitxt provides the possibility for users to use just specific parts of the library (in this case, maybe I want to use a unitxt metric without using the full unitxt pipeline), these sorts of questions might need more thought (cc @elronbandel)
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
|
I approved - waiting for test to pass and auto merge. |
* add LlamaIndex faithfulness metric Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> * share code between LlamaIndex metrics Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> * use existing 'score_prefix' field Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> * remove unused field Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> --------- Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
No description provided.