Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold
This repository is the official implementation of Revisit Overconfidence for OOD Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Thresholde (NAACL2022) by Yanan Wu, Keqing He, Yuanmeng Yan, Qixiang Gao, Zhiyuan Zeng, Fujia Zheng, Lulu Zhao, Huixing Jiang, Wei Wu, Weiran Xu.
An OOD detection model based on reassigned contrastive learning with adaptive class-dependent threshold.
The architecture of the proposed model:
An example of confused label pair:
We use anaconda to create python environment:
conda create --name python=3.6
Install all required libraries:
pip install -r requirements.txt
-
Baseline : Cross-entropy Loss
sh run.sh 0
-
Baseline : Supervised contrastive loss
sh run.sh 1
-
Reassigned Contrastive Learning
sh run.sh 2
The parameters that must be specified:
dataset
, required, The dataset to use, CLINC_OOD_full, ATIS, SNIPS...mode
, optional, specify running mode, options:train
: only train modeltest
: only predict the model with trained modelboth
: train model, and use the trained model to predict
detect_method
, required when the mode is 'test' or 'both', specify the settings to detect ood samples, options:energylocal
: using energy for predicting with adaptive class-dependent threshold
model_dir
, required when the mode is 'test', specify the directory contains model fileunseen_classes
, oosconfused_pre_epoches
, default=5global_pre_epoches
, default=25ce_pre_epoches
, default=30
The parameters that have default values (In general, it can stay fixed):
gpu_device
, default = 1output_dir
, default = ./outputsembedding_file
, default = /glove/glove.6B.300d.txtembedding_dim
, default = 300max_seq_len
, default = None,max_num_words
, default = 10000max_epoches
, default = 200patience
, default = 20batch_size
, default = 200seed
, default = 2022T
, default = 1train_class_num
, default = n
- BiLSTM_LMCL main model
- LMCL Loss
- SCL Loss
- RCL Loss
Model main experiment, including:
- Process Data
- Select OOD type randomly or specify OOD type
- Train model(BiLSTM + LSTM)
- CE
- SCL
- RCL
- Predict model
- energylocal
- get_score, get f1, f1_seen, f1_unseen result according to the confusion matrix
- confidence, calculate mahalanobis or euclidean distance based on the confidence of each category
- get_test_info, get predicting results including text,label,softmax probability, softmax prediction,softmax confidence,(if use lof) lof prediction result,(if use gda) gda mahalanobis distance, (if use gda) the gda confidence
- log_pred_results, Read'. / output_dir... /result.txt' file, print the result in the specified format, and convert it to the 'results.json' file.
CLINC-Full | Snips | ||||||||||||
Global Threshold | Local Threshold(ours) | Global Threshold | Local Threshold(ours) | ||||||||||
OOD | IND | OOD | IND | OOD | IND | OOD | IND | ||||||
F1 | Recall | F1 | Recall | F1 | Recall | F1 | Recall | F1 | Recall | F1 | Recall | ||
MSP | CE | 54.70 | 44.50 | 86.28 | 64.35 | 62.00 | 87.06 | 74.39 | 80.19 | 88.37 | 78.03 | 82.46 | 90.21 |
MSP | SCL | 56.98 | 46.34 | 87.94 | 65.88 | 64.31 | 88.00 | 80.57 | 89.60 | 79.25 | 78.03 | 82.57 | 91.30 |
MSP | RCL(ours) | 61.71 | 53.90 | 88.45 | 67.43 | 64.92 | 88.76 | 81.00 | 81.52 | 91.71 | 83.53 | 82.94 | 93.28 |
GDA | CE | 65.79 | 64.14 | 87.90 | 68.06 | 73.50 | 87.95 | 77.33 | 79.23 | 90.08 | 81.96 | 84.52 | 90.11 |
GDA | SCL | 68.04 | 66.92 | 88.60 | 70.85 | 73.40 | 88.63 | 80.27 | 82.46 | 91.19 | 83.45 | 87.20 | 92.58 |
GDA | RCL(ours) | 72.61 | 70.00 | 88.98 | 73.88 | 75.10 | 89.03 | 85.24 | 86.95 | 93.89 | 87.91 | 88.57 | 94.65 |
Energy | CE | 68.87 | 66.30 | 88.02 | 71.67 | 72.50 | 88.78 | 78.75 | 79.27 | 91.00 | 82.65 | 84.70 | 92.58 |
Energy | SCL | 71.12 | 71.01 | 88.59 | 73.15 | 73.20 | 88.98 | 81.72 | 81.99 | 91.27 | 85.04 | 85.83 | 95.42 |
Energy | RCL | 74.30 | 72.03 | 89.56 | 75.32 | 78.60 | 89.67 | 86.41 | 87.16 | 94.40 | 89.21 | 89.45 | 95.42 |
@inproceedings{wu-etal-2022-revisit,
title = "Revisit Overconfidence for {OOD} Detection: Reassigned Contrastive Learning with Adaptive Class-dependent Threshold",
author = "Wu, Yanan and
He, Keqing and
Yan, Yuanmeng and
Gao, QiXiang and
Zeng, Zhiyuan and
Zheng, Fujia and
Zhao, Lulu and
Jiang, Huixing and
Wu, Wei and
Xu, Weiran",
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.naacl-main.307",
doi = "10.18653/v1/2022.naacl-main.307",
pages = "4165--4179",
abstract = "Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a task-oriented dialog system. A key challenge of OOD detection is the overconfidence of neural models. In this paper, we comprehensively analyze overconfidence and classify it into two perspectives: over-confident OOD and in-domain (IND). Then according to intrinsic reasons, we respectively propose a novel reassigned contrastive learning (RCL) to discriminate IND intents for over-confident OOD and an adaptive class-dependent local threshold mechanism to separate similar IND and OOD intents for over-confident IND. Experiments and analyses show the effectiveness of our proposed method for both aspects of overconfidence issues.",
}