RankLLM: Merge LiT5 Models into RankLLM

[17Melissa]

The LiT5 model suite should be incorporated into RankLLM to centralize our models.

Relevant Repos:

• RankLLM
• LiT5

To work on this task, get started by doing the repro for RankZephyr here: https://github.com/castorini/rank_llm?tab=readme-ov-file#run-end-to-end-test

Then, attempt to repro LiT5 (just the LiT5-Distill.sh): https://github.com/castorini/LiT5. You may need to adjust the batch sizes according to GPU limitations; more details can be found in the script.

[17Melissa]

I've completed the repro for rank_llm and LiT5-Distill. Thanks @ronakice @manveertamber

[17Melissa]

I'm working on this

A potential solution for the integration (thanks @ronakice) :

in a rank_fid.py

• create a class RankFiDDistill(RankLLM)
• create a class RankFiDScore(RankLLM)

in a lit5_reranker.py

• create a LiT5DistillReranker
• create a LiT5ScoreReranker

[ronakice]

Perhaps also https://github.com/soyoung97/ListT5/blob/main/test.py, a contemporary very similar work. Not sure but maybe @soyoung97 can also verify their implementation! :)

[17Melissa]

I've pushed a draft version of LiT5-Distill integration into RankLLM at 17Melissa/rank_llm@eba8cc3. The data.py and model.py scripts are set up, but I'm still figuring out the way to integrate them.

Perhaps the evaluate, clean_response, and remove_duplicate functions should be integrated into run_llm, create_prompt, get_num_tokens, or get_num_tokens. Didn't add any new command line arguments for RankFiDDistill as they seem to be covered by RankZephyr or can be defaulted.

Any insights or suggestions will be appreciated!

[ronakice]

This is great stuff Melissa, we've been refactoring RankLLM so it will be great if we can try to align things with that, I'll give more comments tomorrow :)

[17Melissa]

Made a pull request castorini/rank_llm#116 with what has been accomplished so far and potential next steps