-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TSDAE + GPL and TAS-B + GPL #7
Comments
Hi @Yuan0320, thanks for your attention! Sorry that the description about this in the paper is kinda misleading. It is composed of three training stages: It can also be understood as the TSDAE baseline (which was also trained on MS MARCO) in Table 1 + GPL training. And actually, omitting step (2) leads to very little difference (cf. Table 4 in the paper). And from my observation, the major difference comes from Robust04 (around -1.0 nDCG@10 points). |
@kwang2049 Thanks for the detailed response! Is there no public code about (1) TSDAE on ${dataset} -> (2) MarginMSE on MSMARCO in this repo? Thx. |
Sorry that there is currently no one-step solution for this. To reproduce it, please run these one by one: wget https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/msmarco.zip
unzip msmarco.zip
export dataset="msmarco"
python -m gpl.train \
--path_to_generated_data "./$dataset" \
--base_ckpt "YOUR_TSDAE_CHECKPOINT" \
--gpl_score_function "dot" \
--batch_size_gpl 32 \
--gpl_steps 140000 \
--output_dir "output/$dataset" \
--evaluation_data "./$dataset" \
--evaluation_output "evaluation/$dataset" \
--generator "BeIR/query-gen-msmarco-t5-base-v1" \
--retrievers "msmarco-distilbert-base-v3" "msmarco-MiniLM-L-6-v3" \
--retriever_score_functions "cos_sim" "cos_sim" \
--cross_encoder "cross-encoder/ms-marco-MiniLM-L-6-v2" \
--use_train_qrels
# --use_amp # Use this for efficient training if the machine supports AMP
# One can run `python -m gpl.train --help` for the information of all the arguments
# Notice: Please do not set `qgen_prefix` and leave it as None by default. So bascially, it will skip the query generation step (by setting |
@kwang2049 Thanks for the detailed pipeline for TSDAE + GPL! BTW, if you have a backup of gpl-training-data.tsv file of MarginMSE on MSMARCO, could you share it with me if possible? Thanks. |
You can use this file https://sbert.net/datasets/msmarco-hard-negatives.jsonl.gz instead. The format is different from our |
@kwang2049 Thanks! Regarding the best method GPL/${dataset}-distilbert-tas-b-gpl-self_miner, is the following script correct? (Only change the base_ckpt and retrievers to 'msmarco-distilbert-base-tas-b', compared to the usage in your repo README.md) export dataset="MYDATASET"
python -m gpl.train \
--path_to_generated_data "./$dataset" \
--base_ckpt "msmarco-distilbert-base-tas-b" \
--gpl_score_function "dot" \
--batch_size_gpl 32 \
--gpl_steps 140000 \
--new_size -1 \
--queries_per_passage -1 \
--output_dir "output/$dataset" \
--evaluation_data "./$dataset" \
--evaluation_output "evaluation/$dataset" \
--generator "BeIR/query-gen-msmarco-t5-base-v1" \
--retrievers "msmarco-distilbert-base-tas-b" \
--retriever_score_functions "cos_sim" "cos_sim" \
--cross_encoder "cross-encoder/ms-marco-MiniLM-L-6-v2" \
--qgen_prefix "qgen" \
--do_evaluation \ |
I think you just need to change |
Hello, @kwang2049 ! |
Hi @ArtemisDicoTiar, your understanding is correct. "TSDAE"s in table 1 and 9 mean the same method, i.e. TSDAE (target → MS-MARCO). "Target" here means a certain dataset from the target domain (and we just trained with TSDAE on the corresponding unlabeled corpus). |
In the paper, I found that TSDAE is used to train the retrievers. (domain adaptation for dense retrieval) |
@kwang2049 Hi, thanks for your amazing work!
I wonder if the TSDAE + GPL mentioned in the paper refers to: fine-tuning the distilbert-base PLM with training order of (1)TSDAE on {dataset} -> (2) GPL on {dataset} ?
Thx.
The text was updated successfully, but these errors were encountered: