LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Study

Code

Make sure to use --recurse-submodules flag to clone the submodules as well.

git clone --recurse-submodules

We use the Mamba package manager.

mamba env create -f cfg.yml
conda activate cfg

The generated CFs can be found under ./llms/. CFs from GPT3.5/4 will be published upon approval from OpenAI (Credits for the experiments were obtained through OpenAI API Researcher Access Program).
GPT4 evaluation scores can be found under ./llm-eval-gpt4/
To run the data augmentation results use ./src/eval_augmentation.py (e.g., python eval_augmentation.py --data_origin llms/gpt3.5-20240313 --task sentiment --training_split combined --gpu 0 --model bert-base-uncased --seed 0)
To get the predictions using a finetuned classifier use ./src/add_preds.py (e.g., python add_preds.py --data_origin llms/llama2-20231209/ --task sentiment --training_split combined --gpu 0 --model textattack/bert-base-uncased-imdb)
To add perplexity use ./notebooks/add_ppl.ipynb
To add distance use the scripts under ./notebooks/ that start with dist
The CFs generation process can be found under ./src/gen_cf

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
cls-eval-gpt3.5		cls-eval-gpt3.5
cls-eval-gpt4/NLI/crowd		cls-eval-gpt4/NLI/crowd
llm-eval-gpt3.5		llm-eval-gpt3.5
llm-eval-gpt4		llm-eval-gpt4
llms-ppl-preds		llms-ppl-preds
llms-raw		llms-raw
llms		llms
notebooks		notebooks
src		src
README.md		README.md
cfg.yml		cfg.yml
main_results.csv		main_results.csv
main_results_nli.csv		main_results_nli.csv
main_results_no_training.csv		main_results_no_training.csv
main_results_sa.csv		main_results_sa.csv
main_results_speech.csv		main_results_speech.csv