Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add drop_keys to drop certain keys from the LoRA network #964

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rockerBOO
Copy link
Contributor

I was experimenting with dropping certain keys but thought it would be a good experiment to have in the training process. Ideally drop_keys would support regex .*(attentions).*(attn_v).* (one single regex) or a list of keys drop_keys=proj,to_v

I am proposing it as it is but willing to change or extend fully. This implementation does work as one may expect and may be enough to work off from.

network_args=["drop_keys=proj"]

Note: the skipped message should be updated.

because block_lr_weight is 0 or dim (rank) is 0, 140 LoRA modules are skipped / block_lr_weightまた
    lora_te_text_model_encoder_layers_0_self_attn_k_proj
    lora_te_text_model_encoder_layers_0_self_attn_v_proj
    lora_te_text_model_encoder_layers_0_self_attn_q_proj
    lora_te_text_model_encoder_layers_0_self_attn_out_proj
...
    lora_unet_down_blocks_0_attentions_0_proj_in
    lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_ff_net_0_proj
    lora_unet_down_blocks_0_attentions_0_proj_out
    lora_unet_down_blocks_0_attentions_1_proj_in
    lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_ff_net_0_proj
    lora_unet_down_blocks_0_attentions_1_proj_out
    lora_unet_down_blocks_1_attentions_0_proj_in
    lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_ff_net_0_proj
    lora_unet_down_blocks_1_attentions_0_proj_out
    lora_unet_down_blocks_1_attentions_1_proj_in
...

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant