Question on the learning rate to fine tune lit5 v1 base on BioASQ dataset?

[Oussama-Ouarezki]

Hello Castorini team, I hope everyone is doing well. I’m really impressed by your work and I have tried many of your models. I’m currently using them for my thesis. I chose the BioASQ 13b dataset for fine-tuning LiT5.

I created the distillation dataset using DeepSeek and applied gold anchoring by pushing all relevant documents to the top, since DeepSeek was not a very strong teacher model for reranking. My question is whether this gold anchoring strategy is a good idea, and if it is still possible to improve LiT5 even though it already has very strong zero-shot results.

Also, what learning rate would you recommend for fine-tuning?

Thank you in advance.

Here are the retrieval metrics, i didn't use full corpus i used about 260k documents with bm25.

• nDCG@1: 0.9464
• nDCG@5: 0.9032
• nDCG@10: 0.8735
• nDCG@20: 0.8413
• MRR@10: 0.9651

[musaabhasan]

Gold anchoring can be useful, but I would treat it as a stabilizer rather than the only supervision signal. If every relevant document is always forced to the top, the model may learn a simplified ordering problem and lose sensitivity to hard negatives or partially relevant passages. For BioASQ, I would keep explicit hard negatives from the same query/topic and evaluate with MAP/nDCG on a held-out split before trusting the training loss.

For a strong zero-shot LiT5 baseline, I would start conservatively:

• learning rate: try 1e-5, 2e-5, and 5e-5
• warmup: around 5-10% of steps
• small number of epochs with early stopping on validation MAP/nDCG
• gradient clipping enabled
• compare against zero-shot and teacher-only distillation separately

The key experiment is an ablation:

1. zero-shot LiT5
2. DeepSeek distillation only
3. gold anchoring only
4. distillation + gold anchoring + hard negatives

If 4 improves validation nDCG without hurting recall for difficult biomedical queries, then the anchoring is helping rather than just making the training set easier.

[Oussama-Ouarezki]

First of all, thanks for answering. I did try these learning rates, but they always give diminishing returns. The cause may vary, but I think the model is in its optimal state, at least for the BioASQ testing set. What worked well for me and showed good performance was 5e-6 with LoRA.

My goal has changed now: I am only looking for a slight improvement in performance, just a small nudge in the parameters. That is why I used LoRA only on q and v with a low rank.

The ablation did not perform well; it gave worse results. The ordering became too artificial, and maybe BioASQ contains false positives. A better approach would probably be to only slightly push the gold documents upward with a few permutations, for example moving gold documents from top 16 to top 15, something like that just a small supervision.

About the hard negatives, I thought I could obtain them from DeepSeek distillation because I provided it with all top 50 documents.

[manveertamber]

Hello @Oussama-Ouarezki, thank you for appreciating our work and thanks @musaabhasan for your response.

I would recommend starting at a LR of 5e-5 with a batch-size of 64 and tuning from there. I do think it would be possible to improve LiT5 by tuning it on particular datasets like BioASQ.

As for gold anchoring, I agree with @musaabhasan that it is worth an ablation.

[Oussama-Ouarezki]

Hello @manveertamber the thanks goes to you guys , i will experiment more with gold anchoring maybe i am missing something, as for the batch size it's my bottle neck at the moment i am doing everything locally or in kaggle so best i can do is batch size of 4 and gradient accumulation of 2.