Skip to content
/ IS-CSE Public

This repositoy contains the official implementation of AAAI 2023 paper: Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

Notifications You must be signed in to change notification settings

dll-wu/IS-CSE

Repository files navigation

IS-CSE

This code can be used to reproduce the results of our paper Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding(Our paper has been accepted to AAAI2023.)

Dependencies

We run our code on NVIDIA A100 with CUDA version over 11.0.

You may use the command below

conda create -n iscse python=3.8
conda activate iscse

pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html

pip install -r requirements.txt

Dataset

We use the same training data and evaluation data as in SimCSE. To download the data, please use the following instructions:

Training data:

cd data
bash download_wiki.sh

Evaluation data(STS tasks):

cd SentEval/data/downstream
bash download_dataset.sh

Training

Our code for IS-CSE follows the released code of SimCSE, from which we can get training data and evaluation data.

We offer the training scripts for 4 backbones: BERT-base, BERT-large, RoBERTa-base and RoBERTa-large.

For example, training IS-CSE-BERT-base:

bash run_unsup_bert_base.sh

Evaluation

Modify the model path in run_eval.sh

To evaluate a different task, simply modify the --task_set. ("sts" means STS tasks, "transfer" means transfer tasks and "full" means both STS tasks and transfer tasks.)

python evaluation.py \
    --model_name_or_path Path_to_model \
    --pooler cls_before_pooler \
	--task_set sts \
    --mode test

Then run the script:

bash run_eval.sh

Model List

Model Avg. STS
iscse-bert-base-constant-alpha-0.1 78.30
iscse-bert-large-cos-alpha-0.005-0.05 79.47
iscse-roberta-base-constant-alpha-0.1 77.73
iscse-roberta-large-cos-alpha-0.005-0.05 79.42

About

This repositoy contains the official implementation of AAAI 2023 paper: Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published