keye_vl1.5训练报错 AttributeError: 'list' object has no attribute 'prod'

[qiuhuiGithub]

报错如下，ms-swift版本是3.13.0.dev0，transformers==4.55.4，这个是什么原因呢？我在/ms-swift/swift/llm/template/template/kwai.py里面打印image_grid_thw是一个Tensor不是list呀。
File "/ms-swift/swift/llm/template/template/kwai.py", line 193, in _post_encode
[rank3]: vision_outputs = model.visual(
[rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 1095, in forward
[rank3]: return self.vision_model(
[rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 970, in forward
[rank3]: hidden_states = self.embeddings(
[rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 353, in forward
[rank3]: assert image_grid_thw.prod(1).sum().item() == embeddings.shape[1]
[rank3]: AttributeError: 'list' object has no attribute 'prod'

[slin000111]

报错如下，ms-swift版本是3.13.0.dev0，transformers==4.55.4，这个是什么原因呢？我在/ms-swift/swift/llm/template/template/kwai.py里面打印image_grid_thw是一个Tensor不是list呀。 File "/ms-swift/swift/llm/template/template/kwai.py", line 193, in _post_encode [rank3]: vision_outputs = model.visual( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 1095, in forward [rank3]: return self.vision_model( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 970, in forward [rank3]: hidden_states = self.embeddings( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 353, in forward [rank3]: assert image_grid_thw.prod(1).sum().item() == embeddings.shape[1] [rank3]: AttributeError: 'list' object has no attribute 'prod'

贴一下训练脚本

[qiuhuiGithub]

MODEL_PATH='/models/Kwai-Keye/Keye-VL-1_5-8B'
PORT=$(( RANDOM % (65535 - 1024 + 1) + 1024 ))
MASTER_PORT=${PORT}
NPROC_PER_NODE=4
CUDA_VISIBLE_DEVICES=0,1,2,3
nohup swift sft
--model_type keye_vl
--model ${MODEL_PATH}
--train_type lora
--torch_dtype bfloat16
--deepspeed zero2
--split_dataset_ratio 0.05
--dataset 1.json
--output_dir item_multimodal_keye_0106
--model_name CS
--model_author CS
--freeze_vit true
--freeze_aligner true
--vit_gradient_checkpointing false
--lora_rank 8
--lora_alpha 32
--num_train_epochs 2
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--eval_steps 500
--save_steps 500
--gradient_checkpointing true
--weight_decay 0.1
--learning_rate 1e-4
--gradient_accumulation_steps 8 > item_multimodal_keye_0106.log &

数据格式
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "描述一下这张图片「图片信息」:<image>"}], "images": ["/home/images/ITEM_DETAIL_IMAGE-4721842884-286231f6b7914199aec7a5ee543ecf4b.jpg"]}

[qiuhuiGithub]

报错如下，ms-swift版本是3.13.0.dev0，transformers==4.55.4，这个是什么原因呢？我在/ms-swift/swift/llm/template/template/kwai.py里面打印image_grid_thw是一个Tensor不是list呀。 File "/ms-swift/swift/llm/template/template/kwai.py", line 193, in _post_encode [rank3]: vision_outputs = model.visual( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 1095, in forward [rank3]: return self.vision_model( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 970, in forward [rank3]: hidden_states = self.embeddings( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 353, in forward [rank3]: assert image_grid_thw.prod(1).sum().item() == embeddings.shape[1] [rank3]: AttributeError: 'list' object has no attribute 'prod'

贴一下训练脚本

想问下是脚本有问题或者版本用的不对吗？

[slin000111]

报错如下，ms-swift版本是3.13.0.dev0，transformers==4.55.4，这个是什么原因呢？我在/ms-swift/swift/llm/template/template/kwai.py里面打印image_grid_thw是一个Tensor不是list呀。 File "/ms-swift/swift/llm/template/template/kwai.py", line 193, in _post_encode [rank3]: vision_outputs = model.visual( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 1095, in forward [rank3]: return self.vision_model( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 970, in forward [rank3]: hidden_states = self.embeddings( [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank3]: return self._call_impl(*args, **kwargs) [rank3]: File "/tools/env/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank3]: return forward_call(*args, **kwargs) [rank3]: File "/root/.cache/huggingface/modules/transformers_modules/Keye-VL-1_5-8B/modeling_keye_vl_1_5.py", line 353, in forward [rank3]: assert image_grid_thw.prod(1).sum().item() == embeddings.shape[1] [rank3]: AttributeError: 'list' object has no attribute 'prod'

贴一下训练脚本

想问下是脚本有问题或者版本用的不对吗？

检查下是不是数据有问题。试了一下没问题。

CUDA_VISIBLE_DEVICES=1 \
swift sft \
    --model Kwai-Keye/Keye-VL-1_5-8B \
    --dataset 'modelscope/coco_2014_caption:validation#200' \
    --load_from_cache_file true \
    --split_dataset_ratio 0.1 \
    --tuner_type lora \
    --torch_dtype bfloat16 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --learning_rate 1e-4 \
    --lora_rank 8 \
    --lora_alpha 32 \
    --target_modules all-linear \
    --freeze_vit true \
    --gradient_accumulation_steps 8 \
    --eval_steps 100 \
    --save_steps 100 \
    --save_total_limit 2 \
    --logging_steps 5 \
    --max_length 2048 \
    --output_dir output \
    --warmup_ratio 0.05 \
    --dataloader_num_workers 4

transformers                             4.53.2
peft                                     0.16.0
ms_swift                                 4.0.0.dev0