DAHL

This repository provides the benchmark dataset and automated pipeline for hallucintion evaluation of long-form text generated by Large Language Models.
Link to Paper

Dataset Construction

We generate possible examination questions based on research papers crawled from PMC through gpt-4-1106-preview and manually filter them out, leaving only high-quality questions.

Evaluation Pipeline

The automtaed evaluation pipeline includes two stages: Splitting the response into atomic units, and checking the factuality of each atomic unit.

Install

git clone https://github.com/seemdog/DAHL.git
cd DAHL

Response Generation

Huggingface Models

python generate_response_hf.py --model meta-llama/Meta-Llama-3-8B-Instruct \
--temperature 0.6 \
--max_new_tokens 256

OpenAI Models

python generate_response_gpt.py --model gpt-4o \
--api_key YOUR_API_KEY \
--temperature 0.6

Evaluation

model_to_evaluate: The target model for evaluation. It should be the huggingface directory after the slash('/')
e.g. Meta-Llama-3-8B-Instruct (O) meta-llama/Meta-Llama-3-8B-Instruct (X)
openAI_API_key: Your openAI API key.
perplexityAI_API_key: Your perplexityAI API key.
model_to_use_perplexityAI: The model which will be used in perplexity AI.
e.g. llama-3-8b-instruct

cd evaluate
sh run.sh model_to_evaluate openAI_API_key perplexityAI_API_key model_to_use_perplexityAI

Saving Results

The final DAHL score will be saved in a .txt file.

Citation

@misc{seo2024dahldomainspecificautomatedhallucination,
      title={DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine}, 
      author={Jean Seo and Jongwon Lim and Dongjun Jang and Hyopil Shin},
      year={2024},
      eprint={2411.09255},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.09255}, 
}