support training for multimodal reranker

[0russwest0]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @0russwest0, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the framework's capability to train multimodal reranker models. It introduces new training configurations, refactors model type definitions for better modularity, and implements a specialized preprocessor for multimodal datasets. The changes also include dynamic padding adjustments and robust handling of multimodal inputs, particularly integrating the JinaAI reranker model to ensure its compatibility and optimal performance within the system.

Highlights

• New Reranker Training Scripts: Two new shell scripts, train_reranker_auto_patch.sh and train_reranker_mm.sh, have been added to facilitate the training of multimodal reranker models, including one specifically for the JinaAI reranker.
• Reranker Model Type Refactoring: Reranker models (such as BGE, ModernBERT GTE, and Qwen3 rerankers) have been moved into a dedicated RERANKERModelType class for improved organization and clarity. The new jina_reranker_m0 model has also been integrated into this new type.
• Multimodal Reranker Dataset Preprocessing: A new TextCapsReRankPreprocessor has been introduced to handle the specific requirements of multimodal reranker training, including negative sampling, for the swift/TextCaps dataset.
• Dynamic Padding Side for Rerankers: The padding_side argument now defaults to None and is automatically configured to left for reranker tasks, ensuring correct token pooling behavior during training.
• Enhanced Multimodal Data Handling: The _reranker_encode and _reranker_data_collator methods have been updated to properly process and include image, audio, and video data, which is crucial for multimodal reranker models.
• JinaAI Reranker M0 Integration: Specific integration logic has been added for the JinaAI/jina-reranker-m0 model, including patching its forward method to align its outputs with standard sequence classification expectations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for training multimodal reranker models. It introduces new training scripts, a new preprocessor for reranking tasks on the TextCaps dataset, and refactors model type constants to better organize reranker models. It also adds specific support for the jina-reranker-m0 model by patching its forward pass. The changes are mostly well-structured, but I found a critical bug in the data collator for reranker inference that would cause a crash. I've also pointed out a minor improvement for exception handling.

[0russwest0]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for training multimodal reranker models, which is a significant and well-implemented feature. The changes include new example scripts, a new dataset preprocessor for reranking, refactoring of model types, and support for the jina-reranker-m0 model. The data collators and padding logic have also been updated to accommodate multimodal rerankers. The overall implementation is solid. I have a couple of minor suggestions in the new dataset preprocessor to improve robustness and performance.

[tastelikefeet]

是不是可用*args, **kwargs更适配一些，增加一个@wraps注解

[0russwest0]

不显式声明的话，pre_forward_hook会把position_ids pop掉，开padding_free会有bug