instance level metric outputs

[cabreraalex]

This is fantastic work!

I was wondering if you all could release the instance-level outputs from the analysis. We'd love to visualize the results using Zeno

[simonhughes22]

Can you expand on what you mean by this? You can interact with the model right now on Huggingface if that helps https://huggingface.co/vectara/hallucination_evaluation_model

[cabreraalex]

Ah sorry I meant the outputs of the hallucination evaluation model for each instance, e.g. a new column with the model's output in this file: https://github.com/vectara/hallucination-leaderboard/blob/main/leaderboard_summaries.csv

Would love to dive in and compare the summarization models more in depth, similar to reports we've published recently on the HF leaderboard and Whisper transcription models:

https://twitter.com/gneubig/status/1724872160144171104
https://twitter.com/a_a_cabrera/status/1722698009094529118

[amin2718]

Hello, I'm sorry I didn't see this earlier. Tragically, Simon passed away over Thanksgiving, and other members of the team are picking this up. We'll try to get the new column added soon.

[cabreraalex]

Oh no, I'm so sorry! Best wishes to the family and team. Of course, no rush at all!