LM-TOAST

Code and data of the Findings of ACL 2023 [paper] "Making Pre-trained Language Models both Task-solvers and Self-calibrators".

Installation

pip install -r requirements.txt
mkdir data

Data Preparation

We use the same datasets and preprocessing scripts as in this [paper]. You can download the datasets from Google Drive [download], and upload the folder (TextClassification) to the data directory. Then, all the datasets used in the paper can be find in ./data/TextClassification.

Experiments

export PYTHONPATH='pwd':$PYTHONPATH
python src/scripts/run.py --model_name t5 --scale base --dataset_name amazon_food --save_path amazon_t5base.ckpt

You can change the --model_name, --scale to run different experiments with different models and scales. The --dataset_name can be chosen from sst2, mnli, yahoo_answers_topics, amazon_food, civil_comments, dynasent.

The hyper-parameters, such as the repeat number, consistent_factor, data augmentation approach, are configured as default values in the file. You may run the following code to check the hyper-parameters configuration:

python src/scripts/run.py --help

Process Results

To compute the metrics for calibration. Run:

export PYTHONPATH='pwd':$PYTHONPATH
python src/scripts/metric.py --setting_list SETTING_LIST --model_list MODEL_LIST --dataset_list DATASET_LIST

By passing SETTING_LIST, MODEL_LIST and DATASET_LIST, you can find the final metrics for all the experiments in the directory ./metrics.

Citation

Please kindly cite our paper:

@inproceedings{chen-etal-2023-making,
    title = "Making Pre-trained Language Models both Task-solvers and Self-calibrators",
    author = "Chen, Yangyi  and
      Wang, Xingyao  and
      Ji, Heng",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    year = "2023",
    publisher = "Association for Computational Linguistics",    
}