truthness is a ultra-flexible tool designed to evaluate the answers generated by Large Language Models by comparing them to ground-truth answers. The library supports any langchain-compatible models as arbiter and multilingual config-based prompts.
- Intersaction over Cartesian: measure the relevance between extracted facts from the ground-truth and LLM-generated data.
- F-precision: (not implemented) precision over the extracted facts.
- ToV: (not implemented) similarity between answers with respect to Ton-of-Voice
To install Truthness, you need to have Poetry installed. If you don't have Poetry installed, you can install it by following the instructions here.
Once you have Poetry installed, you can install Truthness by running the following command:
git clone https://github.com/me0em/truthness.git
cd truthness
poetry installthis chapter is in progress rn