Kandinsky2.2 训练支持 #268
Labels
HappyOpenSource Pro
快乐开源issue与PR，更具挑战的任务
Comments
Closed
JunnYu
pushed a commit
that referenced
this issue
Jan 15, 2024
## [[Kandinsky2.2 训练支持 · Issue #268 · PaddlePaddle/PaddleMIX](https://github.com/PaddlePaddle/PaddleMIX/issues/268)](https://github.com/PaddlePaddle/PaddleMIX/issues/268) ### 1 前200steps loss对齐结果： - decoder w/o LoRA:  - prior w/o LoRA:  - decoder with LoRA:  - prior with LoRA:  - decoder finue-tune 3k steps results(prompts: A robot pokemon, 4k photo):  ### 2 其他修改 [[ppdiffusers/models/attention_processor.py/LoRAAttnAddedKVProcessor.call](https://github.com/PaddlePaddle/PaddleMIX/blob/ff0d2f25c79cc6e34e7d9c071328a7ed8bea4bc3/ppdiffusers/ppdiffusers/models/attention_processor.py#L789C57-L789C79)] : axis = 1 -> axis = 2 修改原因：运行python train_text_to_image_decoder_lora.py使用LoRAAttnAddedKVProcessor出现concat拼接维度错误。 ### 3 对齐说明 - 关闭diffusers和ppdiffusers中dataloader中的shuffle，保证数据顺序一致; - 设置同一随机种子，并将在trainning loop中造成随机性的noise和timesteps改为由numpy生成统一随机结果(提交代码已删除该逻辑)。 ### 4 存在问题 - 在ppdiffusers中使用AutoPipelineForText2Image(args.pretrained_decoder_model_name_or_path)出现组件缺失： ```bash ValueError: Pipeline <class 'ppdiffusers.pipelines.kandinsky2_2.pipeline_kandinsky2_2_combined.KandinskyV22CombinedPipeline'> expected {'unet', 'prior_image_processor', 'prior_text_encoder', 'prior_image_encoder', 'movq', 'prior_prior', 'prior_scheduler', 'prior_tokenizer', 'scheduler'}, but only {'unet', 'movq', 'scheduler'} were passed. ``` 只能识别部分组件，无法像diffusers自动识别所有组件。故在提交代码中采取下策：在AutoPipelineForText2Image前逐个定义好后传入，不够简洁。目前原因未定，看到一个[[diffusers的issue](https://github.com/PaddlePaddle/PaddleMIX/compare/%5Bhttps://github.com/huggingface/diffusers/issues/5044)]([https://github.com/huggingface/diffusers/issues/5044)与该问题类似。 - 使用pip install ppdiffusers=0.19.4 在下载prior的LoRA权重时会出现PriorTransformer找不到load_attn_procs, 无法使用pipeline.prior_prior.load_attn_procs(args.output_dir)，但使用最新develop分支构建ppdiffusers安装包则不会出现这个问题。 ----------期待回复与关于合入的建议, Thx :)------------------ --------- Co-authored-by: Tsaiyue <tsaiyue01@gamil.com>
|
close due to #378. |
Closed
westfish
pushed a commit
to westfish/PaddleMIX
that referenced
this issue
Sep 25, 2024
## [[Kandinsky2.2 训练支持 · Issue PaddlePaddle#268 · PaddlePaddle/PaddleMIX](https://github.com/PaddlePaddle/PaddleMIX/issues/268)](https://github.com/PaddlePaddle/PaddleMIX/issues/268) ### 1 前200steps loss对齐结果： - decoder w/o LoRA:  - prior w/o LoRA:  - decoder with LoRA:  - prior with LoRA:  - decoder finue-tune 3k steps results(prompts: A robot pokemon, 4k photo):  ### 2 其他修改 [[ppdiffusers/models/attention_processor.py/LoRAAttnAddedKVProcessor.call](https://github.com/PaddlePaddle/PaddleMIX/blob/ff0d2f25c79cc6e34e7d9c071328a7ed8bea4bc3/ppdiffusers/ppdiffusers/models/attention_processor.py#L789C57-L789C79)] : axis = 1 -> axis = 2 修改原因：运行python train_text_to_image_decoder_lora.py使用LoRAAttnAddedKVProcessor出现concat拼接维度错误。 ### 3 对齐说明 - 关闭diffusers和ppdiffusers中dataloader中的shuffle，保证数据顺序一致; - 设置同一随机种子，并将在trainning loop中造成随机性的noise和timesteps改为由numpy生成统一随机结果(提交代码已删除该逻辑)。 ### 4 存在问题 - 在ppdiffusers中使用AutoPipelineForText2Image(args.pretrained_decoder_model_name_or_path)出现组件缺失： ```bash ValueError: Pipeline <class 'ppdiffusers.pipelines.kandinsky2_2.pipeline_kandinsky2_2_combined.KandinskyV22CombinedPipeline'> expected {'unet', 'prior_image_processor', 'prior_text_encoder', 'prior_image_encoder', 'movq', 'prior_prior', 'prior_scheduler', 'prior_tokenizer', 'scheduler'}, but only {'unet', 'movq', 'scheduler'} were passed. ``` 只能识别部分组件，无法像diffusers自动识别所有组件。故在提交代码中采取下策：在AutoPipelineForText2Image前逐个定义好后传入，不够简洁。目前原因未定，看到一个[[diffusers的issue](https://github.com/PaddlePaddle/PaddleMIX/compare/%5Bhttps://github.com/huggingface/diffusers/issues/5044)]([https://github.com/huggingface/diffusers/issues/5044)与该问题类似。 - 使用pip install ppdiffusers=0.19.4 在下载prior的LoRA权重时会出现PriorTransformer找不到load_attn_procs, 无法使用pipeline.prior_prior.load_attn_procs(args.output_dir)，但使用最新develop分支构建ppdiffusers安装包则不会出现这个问题。 ----------期待回复与关于合入的建议, Thx :)------------------ --------- Co-authored-by: Tsaiyue <tsaiyue01@gamil.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Kandinsky2.2 训练支持
任务描述
任务背景
完成步骤
提交内容：
The text was updated successfully, but these errors were encountered: