Add Vectara Hallucination Detection Model

[Josephrp]

Added vectara hallucination detection model to the huggingface class

adding an exposition/model example using huggingface models end to end for demo

this is a draft PR , just need to add some text to explain the examples we have chosen

---

	🚀 This PR description was created by Ellipsis for commit 4f3353e.

Summary:

The PR adds a new method hallucination_evaluator to the Huggingface class in hugs.py for evaluating the hallucination score of a combined input of two statements using the Huggingface hallucination evaluation model.

Key points:

• Added hallucination_evaluator method to Huggingface class in hugs.py.
• The method uses the HUGS_HALLUCINATION_API_URL endpoint for the hallucination evaluation model from Huggingface.
• The method takes two arguments: model_output and retrieved_text_chunks, combines them, and sends a POST request to the API.
• The response is parsed to extract the hallucination score.

---

Generated with ❤️ by ellipsis.dev

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ellipsis-dev]

❌ Changes requested.

• Reviewed the entire pull request up to 4f3353e
• Looked at 61 lines of code in 1 files
• Took 1 minute and 3 seconds to review

More info

• Skipped 4 files when reviewing.
• Skipped posting 1 additional comments because they didn't meet confidence threshold of 50%.

1. trulens_eval/trulens_eval/feedback/provider/hugs.py:485:

• Assessed confidence : 100%
• Grade: 40%
• Comment:
  The method hallucination_evaluator does not handle the case when the response from the API is not a list and not a proper HTTP response. This could lead to unexpected behavior. Consider adding an else clause to handle this case.
• Reasoning:
  The new method hallucination_evaluator is not handling the case when the response from the API is not a list and not a proper HTTP response. This could lead to unexpected behavior.

Workflow ID: wflow_I5ako30SCD4tU8DH

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. We'll respond in a few minutes. Learn more here.

[joshreini1]

Vectra -> Vectara?

Please remove the stray "or."

Reply via ReviewNB

[joshreini1]

Can you remove the .ipynb checkpoints from this change?

Also can you elaborate on why there are two separate notebooks to show this capability?

[joshreini1]

since this notebook is focused on the Vectara HHEM evaluator, can you rename the notebook to reflect that?

Suggestion: vectara_hallucination_evaluator.ipynb

Reply via ReviewNB

[joshreini1]

It'd be useful here to show usage of this evaluator as part of a recorded app, e.g. as shown in https://www.trulens.org/trulens_eval/langchain_quickstart/

Reply via ReviewNB

[joshreini1]

This notebook seems similar to the one under /models - what is the distinction?

Reply via ReviewNB

[joshreini1]

Thanks for all the work on this @MN-Noor @Josephrp - this is close! Just added a few small comments before it can go in

[Josephrp]

hi @MN-Noor would you like to take the last reviews above ?

@joshreini1 , thanks for the comments, should we keep both notebooks renamed or remove one or the other?

sorry to bother you but i'll be able to wrap this up :-)

[joshreini1]

@Josephrp keep the one in /models and remove the other. Thanks!

[Josephrp]

@joshreini1 thanks ! @MN-Noor wrapped it up nicely , hope that's us done - until next time !

[joshreini1]

@Josephrp @MN-Noor do you have a Twitter/x account? I’ll shoutout from the TruLens handle if so!

[Josephrp]

ha! i sure do , here's me .
Noor's doesnt work well where she is , so that's why she prefers linked-in

hope that's okay :-)