List view
We are integrating the RankLLaMA model: 🔗 https://github.com/texttron/tevatron/tree/main/examples/rankllama 📄 Paper: RepLLaMA: Fine-Tuning LLaMA for Multi-Stage Text Retrieval https://arxiv.org/abs/2310.08319 Tasks: Add RankLLaMA inference for reranking pipeline Validate input construction, prompt formatting, and output score parsing Ensure correct model weights and LoRA adapters are loaded Schedule evaluation on standard benchmarks Document and address any inconsistencies if they arise
No due dateWe are integrating the RepLLaMA model for inference-only use, based on the official implementation: 🔗 https://github.com/texttron/tevatron/tree/main/examples/repllama 📄 Paper: RepLLaMA: Fine-Tuning LLaMA for Multi-Stage Text Retrieval https://arxiv.org/abs/2310.08319 This milestone tracks the following tasks: 1. Add RepLLaMA inference module 2. Ensure input formatting and scoring follow the original setup 3. Confirm correct LLaMA weight usage (e.g., LoRA if applicable) 4. Plan to evaluate results later against reported benchmarks
No due dateThe results from the MonoT5 implementation in this repo don’t match the performance reported in benchmark datasets or the original paper. We need to investigate and verify the implementation details. Tasks: 1. Review the code and model loading (especially MonoT5 checkpoint usage) 2. Verify how inputs are tokenized and formatted 3. Check if scoring and ranking logic matches what the paper describes 4. Reproduce one or more benchmark runs to validate performance 5. Identify any mismatch or deviation from the expected setup 6. Suggest fixes or improvements if needed
No due dateThe results we’re getting from the current ListT5 implementation don’t match what’s reported in the original paper. This task is to take a closer look at the code and figure out why. Steps include: 1. Go through the implementation in detail (https://github.com/soyoung97/ListT5) 2. Try to reproduce the paper’s experiments 3. Identify anything that’s different from the paper
No due date