Add support for Keye-VL-1_5-8B

[hellopahe]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

• Add support for Keye-VL-1_5-8B

Experiment results

tested with

CUDA_VISIBLE_DEVICES=0 \
swift rlhf \
    --rlhf_type dpo \
    --model "Kwai-Keye/Keye-VL-1_5-8B" \
    --model_type keye_vl_1_5 \
    --train_type lora \
    --dataset 'ds_path' \
    --split_dataset_ratio 0.001 \
    --num_train_epochs 2 \
    --per_device_train_batch_size 4 \
    --learning_rate 1e-5 \
    --lora_rank 32 \
    --lora_alpha 128 \
    --target_modules all-linear \
    --gradient_accumulation_steps 16 \
    --eval_steps 200 \
    --save_steps 20 \
    --logging_steps 1 \
    --max_length 2048 \
    --output_dir 'output_dir' \
    --warmup_ratio 0.05 \
    --dataloader_num_workers 0 \
    --report_to 'swanlab' \
     --eval_use_evalscope \
    --per_device_eval_batch_size 64 \
    --eval_dataset "gsm8k" \
    --eval_generation_config '{"max_tokens": 2048, "temperature": 0}' \
    --eval_limit "200"

issue #5788

cc @hjh0119

[gemini-code-assist]

Summary of Changes

Hello @hellopahe, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant new feature by integrating the Keye-VL-1_5-8B model into the system. This allows users to leverage this advanced multimodal model for various tasks. The changes encompass updates across documentation, model registration, and utility functions to ensure a smooth and robust integration, enabling the platform to support a wider range of cutting-edge models.

Highlights

• New Model Support: Added comprehensive support for the Keye-VL-1_5-8B multimodal large language model, expanding the platform's capabilities.
• Documentation Updates: Updated both Chinese and English documentation to include the newly supported Keye-VL-1_5-8B model and its associated command-line parameters.
• Robust Utility Patching: Improved the patch_qwen_vl_utils function to be more resilient, gracefully handling cases where specific attributes or video reader backends might not be present in the vision_process module.
• Core System Integration: Integrated the new model by defining its type, registering its architecture, and associating it with the appropriate template within the system's core model and template management modules.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for the Keye-VL-1_5-8B model. The changes are mostly correct and well-structured, including updates to model constants, architecture registration, and documentation. However, I've identified a critical bug in patch_qwen_vl_utils where a dictionary is not being populated, and a couple of issues in mllm.py related to code duplication and an incorrect model architecture reference. Addressing these points will improve the correctness and maintainability of the code.

[gemini-code-assist]

The res dictionary is not being populated within the loop. This will cause patch_qwen_vl_utils to return an empty dictionary, and consequently, processor.global_vars will be empty. This appears to be a bug.

The previous implementation correctly populated the res dictionary. I suggest restoring that logic.

Suggested change

	setattr(vision_process, key.upper(), get_env_args(key, type_func, default_value))
	val = get_env_args(key, type_func, default_value)
	setattr(vision_process, key.upper(), val)
	res[key] = val

[Jintao-Huang]

If it's the same template, you can reuse/share it — there's no need to register another one.

[Jintao-Huang]

If it's the same model_arch, you can reuse/share it — there's no need to register another one.

[Jintao-Huang]

hello!

please run the following commands: 😊

pip install pre-commit
pre-commit run --all-files

[Jintao-Huang]

Please refer to the script here and write a test case.😊

https://github.com/modelscope/ms-swift/blob/main/tests/test_align/test_template/test_vision.py

[Jintao-Huang]

You can run this file to automatically update the documentation for supported models and datasets: https://github.com/modelscope/ms-swift/blob/main/scripts/utils/run_model_info.py

[Jintao-Huang]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for the Keye-VL-1_5-8B model. The changes include adding the model type, registering the model, updating documentation, and adding a test case. The changes are well-structured and correct. The refactoring of patch_qwen_vl_utils for improved robustness is a good enhancement. I have one suggestion to refactor the new test case to reduce code duplication and improve maintainability.

[gemini-code-assist]

This test function is very similar to test_keye_vl. To improve maintainability and reduce code duplication, you could use pytest.mark.parametrize to combine them into a single test function. This would also make it easier to add more keye_vl models in the future.

For example, you could replace both test_keye_vl and test_keye_vl_1_5 with a single parameterized test:

import pytest

@pytest.mark.parametrize('model_id', [
    'Kwai-Keye/Keye-VL-8B-Preview',
    'Kwai-Keye/Keye-VL-1_5-8B',
])
def test_keye_vl_models(model_id):
    pt_engine = PtEngine(model_id)
    messages = [{'role': 'user', 'content': '<image><image>What is the difference between the two images?'}]
    images = [
        'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png',
        'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png'
    ]
    pt_engine.default_template.template_backend = 'swift'
    response = _infer_model(pt_engine, messages=messages, images=images)
    pt_engine.default_template.template_backend = 'jinja'
    response2 = _infer_model(pt_engine, messages=messages, images=images)
    assert response == response2

You may need to add import pytest at the top of the file.